Upgrading to Bullseye (troubleshooting Armbian 21.08.1)

[ebin-dev]

Upgrading Helios64 from Armbian Buster to Bullseye (see below) works as expected on my system.

However, I am using systemd-networkd and just a few services (nextcloud, netatalk, etc. and not ZFS)

 

EDIT: Upgrading Buster installations to Bullseye also works fine if you use network-manager, even if you have a bridge configured (using bridge-slave; binutils-bridge).

 

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

 

 _   _      _ _            __   _  _   
| | | | ___| (_) ___  ___ / /_ | || |  
| |_| |/ _ \ | |/ _ \/ __| '_ \| || |_ 
|  _  |  __/ | | (_) \__ \ (_) |__   _|
|_| |_|\___|_|_|\___/|___/\___/   |_|  
                                       
Welcome to Armbian 21.08.1 Bullseye with Linux 5.10.43-rockchip64

System load:   2%           	Up time:       12:29	
Memory usage:  19% of 3.77G  	IP:	       xx.xx.xx.xx
CPU temp:      42°C           	Usage of /:    41% of 15G    	
storage/:      57% of 3.6T   	

 

Edit: Attention - if you upgrade your Buster or Bullseye installation on emmc to Armbian 21.08.1 it will not be writable anymore.

You will then have to downgrade linux on emmc from 5.10.60 to 5.10.43 as described in this thread.

 

Edit: There is a temporary fix for the problem. See this message from @piter75

 

To upgrade Armbian Buster to Bullseye, first disable Armbian updates in /etc/apt/sources.list.d/Armbian.list for the time being.

Then fully upgrade Buster (sudo apt update && sudo apt upgrade -y) , then change the apt sources (see below) followed by 'sudo apt update && sudo apt full-upgrade'.

I kept all the configuration files by confirming 'N' in the following dialogue.

 

# cat /etc/apt/sources.list
deb http://deb.debian.org/debian/ bullseye main
deb-src http://deb.debian.org/debian/ bullseye-updates main
deb http://security.debian.org/debian-security bullseye-security main
deb-src http://security.debian.org/debian-security bullseye-security main
deb http://ftp.debian.org/debian bullseye-backports main 

 

[barnumbirr]

Seems risky to me to upgrade now: Bullseye support is coming but might take a few months.
I'll stick with Buster for now as I already have enough stability issues.

[Werner]

Userspace upgrades (like Buster to Bullseye or Bionic to Focal) were never actively supported by Armbian.

[ebin-dev]

On 8/27/2021 at 2:57 PM, barnumbirr said:

I'll stick with Buster for now as I already have enough stability issues.

 

There were no stability issues with Armbian Buster on my system - and everything seems fine so far with Bullseye too. I am using Bullseye on emmc as of today.

If any issues should arise, I will post them here.

 

The bonding issue occurs only if 'ifenslave' is installed.

 

On 8/27/2021 at 3:41 PM, Werner said:

Userspace upgrades (like Buster to Bullseye or Bionic to Focal) were never actively supported by Armbian.

 

That may be true - but the message is that Bullseye is working with the current firmware and kernel - therefore an Armbian Bullseye image might be prepared for testing purposes, by just switching to the Bullseye sources.

EDIT: Bullseye images are already available on the Helios64 download page but marked as 'unstable'.

 

P.S.:

# networkctl
IDX LINK       TYPE     OPERATIONAL SETUP
  1 lo         loopback carrier     unmanaged
  2 eth0       ether    enslaved    configured
  3 br0        bridge   routable    configured
  4 eth1       ether    off         unmanaged

 

[jotapesse]

This one is interesting, I would like to upgrade as well! I have a stable Helios64 Armbian 21.05.8 Buster with ZFS from buster-backports and no OMV so I would expect to work well as well. So I guess for the update to Bullseye you have just replaced "buster" for "bullseye" on the several source.list files and then APT updated and full-upgraded right? Or you did only Debian repository packages?

I didn't know that there was an Armbian "bullseye" repository already. Is it being updated and developed from "buster" already? Or still on a limbo early stages not ready to use state?

[ebin-dev]

On 8/28/2021 at 10:51 AM, jotapesse said:

This one is interesting, I would like to upgrade as well! I have a stable Helios64 Armbian 21.05.8 Buster with ZFS from buster-backports and no OMV so I would expect to work well as well. So I guess for the update to Bullseye you have just replaced "buster" for "bullseye" on the several source.list files and then APT updated and full-upgraded right? Or you did only Debian repository packages?

I didn't know that there was an Armbian "bullseye" repository already. Is it being updated and developed from "buster" already? Or still on a limbo early stages not ready to use state?

 

To upgrade Armbian Buster to Bullseye, I  fully upgraded Buster first (sudo apt update && sudo apt upgrade -y) , then changed the apt sources (see below) followed by 'sudo apt update && sudo apt full-upgrade'. I kept all the configuration files by confirming 'N' in the following dialogue. Finally I disabled Armbian updates in /etc/apt/sources.list.d/Armbian.list for the time being.

This process works for me - but is not supported by Armbian - please be careful. I tried this successfully on the system on microSD and then transferred the installation to emmc.

Debian 11 would appear to boot faster - and the services delivered i.e by nextcloud are more responsive now.

 

# cat /etc/apt/sources.list
deb http://deb.debian.org/debian/ bullseye main
deb-src http://deb.debian.org/debian/ bullseye-updates main
deb http://security.debian.org/debian-security bullseye-security main
deb-src http://security.debian.org/debian-security bullseye-security main
deb http://ftp.debian.org/debian bullseye-backports main 

 

[jotapesse]

Thanks for confirming. There is also an Armbian bullseye repository available. Would that work?

 

Update: I guess yes from the linked post "Armbian (Debian Bulleye) - supported or not?" above.

Edited August 28, 2021 by jotapesse

[ebin-dev]

6 hours ago, jotapesse said:

Thanks for confirming. There is also an Armbian bullseye repository available. Would that work?

 

Update: I guess yes from the linked post "Armbian (Debian Bulleye) - supported or not?" above.

 

Please note that the Debian security URLs have changed their structure.

 

The Armbian Bullseye repository currently offers updates to Armbian 21.08.1 - including a new kernel. Will test them on SD ...

 

# apt list --upgradable
Listing... Done
armbian-bsp-cli-helios64/bullseye 21.08.1 arm64 [upgradable from: 21.05.8]
armbian-config/bullseye,bullseye 21.08 all [upgradable from: 21.05.8]
armbian-firmware/bullseye,bullseye 21.08 all [upgradable from: 21.05.8]
hostapd/bullseye 3:2.9-102~armbian20.05.2+1 arm64 [upgradable from: 3:2.9-102~armbian20.05.2+1]
htop/bullseye 3.1.0-0~armbian20.08.2+1 arm64 [upgradable from: 3.1.0-0~armbian20.08.2+1]
linux-dtb-current-rockchip64/bullseye 21.08.1 arm64 [upgradable from: 21.05.4]
linux-headers-current-rockchip64/bullseye 21.08.1 arm64 [upgradable from: 21.05.4]
linux-image-current-rockchip64/bullseye 21.08.1 arm64 [upgradable from: 21.05.4]
linux-libc-dev/bullseye 21.08.1 arm64 [upgradable from: 21.05.8]

 

[ebin-dev]

All updates from the Armbian Bullseye repository were installed without any issues (just needed to wait until the updates have propagated to the Armbian mirrors :-)) ).

 

Now the system is migrated to Bullseye with adapted firmware, new armbian-config and a new kernel 5.10.60.

 

All my services are up and running from SD and the welcome screen is updated to "Bullseye" too ...

 

Unfortunately accessing emmc does not work correctly with this kernel : emmc mounted readonly, mmc2: running CQE recovery ...

 

 
 _   _      _ _            __   _  _   
| | | | ___| (_) ___  ___ / /_ | || |  
| |_| |/ _ \ | |/ _ \/ __| '_ \| || |_ 
|  _  |  __/ | | (_) \__ \ (_) |__   _|
|_| |_|\___|_|_|\___/|___/\___/   |_|  
                                       
Welcome to Armbian 21.08.1 Bullseye with Linux 5.10.60-rockchip64

No end-user support: work in progress

System load:   23%           	Up time:       0 min	
Memory usage:  6% of 3.77G  	IP:	       xxx.xxx.xxx.xx
CPU temp:      51°C           	Usage of /:    34% of 15G    	
storage/:      57% of 3.6T   

# cat /proc/version
Linux version 5.10.60-rockchip64 (root@runner7) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #21.08.1 SMP PREEMPT Wed Aug 25 18:56:55 UTC 2021

 

[BipBip1981]

Hello,

 

Same issu: emmc mounted readonly, mmc2: running CQE recovery ... with Buster or Bulleye and kernel 5.10.60-rockchip64.

 

ext4 on emmc pass to read-only after lot of I/O error and Helios become unusable.

 

Downgrade to Armbian 21.05.8 Buster with Linux 5.10.43-rockchip64 and all is Okok, no I/O error on emmc.

 

 

Edited August 29, 2021 by BipBip1981

[ebin-dev]

4 hours ago, BipBip1981 said:

Downgrade to Armbian 21.05.8 Buster with Linux 5.10.43-rockchip64 and all is Okok, no I/O error on emmc.

 

Where did you find the required 21.05 kernel packages ? 

For the downgrade you would need those:

armbian-bsp-cli-helios64/buster 21.05.8  
armbian-config/buster,buster 21.05  
armbian-firmware/buster,buster 21.05 
linux-dtb-current-rockchip64/buster 21.05.4 
linux-headers-current-rockchip64/buster 21.05.4 
linux-image-current-rockchip64/buster 21.05.4 

 

EDIT: I found them here: http://armbian.hosthatch.com/apt/pool/main/l/linux-5.10.43-rockchip64/

[ebin-dev]

Back to Armbian Bullseye with linux-5.10.43 - emmc I/O errors are gone. I will run Debian Bullseye from emmc now with Armbian updates disabled for the time being.

 

 _   _      _ _            __   _  _   
| | | | ___| (_) ___  ___ / /_ | || |  
| |_| |/ _ \ | |/ _ \/ __| '_ \| || |_ 
|  _  |  __/ | | (_) \__ \ (_) |__   _|
|_| |_|\___|_|_|\___/|___/\___/   |_|  
                                       
Welcome to Armbian 21.08.1 Bullseye with Linux 5.10.43-rockchip64

System load:   2%           	Up time:       11 min	
Memory usage:  16% of 3.77G  	IP:	       xxx.xxx.xxx.xx
CPU temp:      45°C           	Usage of /:    36% of 15G    	
storage/:      57% of 3.6T   	

~# cat /proc/version
Linux version 5.10.43-rockchip64 (root@runner4) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #21.05.4 SMP PREEMPT Wed Jun 16 08:02:12 UTC 2021

 

@jotapesse The answer to your question is clearly: Do not yet use Armbian Bullseye repositories for updating your Debian Bullseye installation !

 

[ebin-dev]

@Werner@piter75 Is there a way to support Helios64 software development by testing images before they are released ?

[Werner]

Since released images are built from master with specific version number simply checkout master and build your own image and test it.

[Igor]

4 hours ago, ebin-dev said:

Is there a way to support Helios64 software development by testing images before they are released ?

 

Only best long term way is to have one such device in auto testing lab and some help to improve testing routines. With no resources, I can only move super slow into this direction.

[ebin-dev]

13 hours ago, Igor said:

Only best long term way is to have one such device in auto testing lab and some help to improve testing routines. With no resources, I can only move super slow into this direction.

 

There is a severe issue in Armbian 21.08.1 with kernel 5.10.60 on Helios64 - both in Buster (according to @BipBip1981) and in Bullseye: emmc is not accessible anymore - i/o errors occur, and all those installations on emmc are therefore broken after the update.  The same bug was observed weeks ago by @alchemist - the reason was that some Armbian patches do not compile anymore with recent kernels.

 

If linux-5.10.43 is kept, Bullseye is perfectly stable on Helios64. Would you therefore roll back the current kernel versions to 5.10.43 in Buster and Bullseye images ?

 

In the meantime I had a look at the bootloader images in the file system after the update to 21.08.1. They are still dated July 24th and can be used to update the bootloader on emmc:

 

# cd /usr/lib/linux-u-boot-helios64-current_21.08.1_arm64
# ls -la
total 4192
drwxrwxr-x  2 root root    4096 Jul 25 19:59 .
drwxr-xr-x 80 root root    4096 Aug 27 12:50 ..
-rw-rw-r--  1 root root  206844 Jul 24 14:11 idbloader.bin
-rw-rw-r--  1 root root 4194304 Jul 24 14:11 trust.bin
-rw-rw-r--  1 root root 4194304 Jul 24 14:11 uboot.img

# dd if=idbloader.bin of=/dev/mmcblk2 seek=64 conv=notrunc
# dd if=uboot.img of=/dev/mmcblk2 seek=16384 conv=notrunc
# dd if=trust.bin of=/dev/mmcblk2 seek=24576 conv=notrunc

# reboot now

 

P.S.: I am happy to test any release candidates (in particular new kernel packages and Armbian Bullseye images).

[Igor]

2 hours ago, ebin-dev said:

If linux-5.10.43 is kept, Bullseye is perfectly stable on Helios64. Would you therefore roll back the current kernel versions to 5.10.43 in Buster and Bullseye images ?

We can only hold / remove upgrades for entire rockchip64 current family (5.10.y) but this bug has to be solved asap since this is temporally solution / hack. 

[IcerJo]

8 hours ago, Igor said:

We can only hold / remove upgrades for entire rockchip64 current family (5.10.y) but this bug has to be solved asap since this is temporally solution / hack. 

Howdy Igor! A workaround I found that seems to work for me, was to boot from the latest release from an SD card, and update the bootloader. It seemed to let me back in and stopped giving me the login loop i was experiencing. I am running Buster.

[bunducafe]

Am 30.8.2021 um 11:31 schrieb ebin-dev:

 

There is a severe issue in Armbian 21.08.1 with kernel 5.10.60 on Helios64 - both in Buster (according to @BipBip1981) and in Bullseye: emmc is not accessible anymore - i/o errors occur, and all those installations on emmc are therefore broken after the update.  The same bug was observed weeks ago by @alchemist - the reason was that some Armbian patches do not compile anymore with recent kernels.

Uh, I should have followed this thread a little bit earlier as I ran into the emmc issue (no write permission) and the system does not come up and I cannot manipulate anything.

For now I am back with sd cards and will probably stick to it as I can make clones from the sdcard a bit easier with my setup here.

 

Is there an easy way to roll back the actual kernel?

[IcerJo]

20 minutes ago, bunducafe said:

Uh, I should have followed this thread a little bit earlier as I ran into the emmc issue (no write permission) and the system does not come up and I cannot manipulate anything.

For now I am back with sd cards and will probably stick to it as I can make clones from the sdcard a bit easier with my setup here.

 

Is there an easy way to roll back the actual kernel?

I don't know much when it comes to armbian, but from the temp fix I did, it seems like the bootloader was the culprit/issue, I installed/pushed a new bootloader via the latest image on an sd card and my system is back up and running with the latest kernel with no problems. Or rather i think it might be a combination of issues, I am not sure how the bootloader being pushed would work if the kernel is the culprit but at least my system is back up and stable till they have a proper fix implemented.

 

I read another Post where its a Kernel Panic that Occurs at 150372 seconds = 41.7 hours..

[ebin-dev]

33 minutes ago, bunducafe said:

Is there an easy way to roll back the actual kernel?

 

In order to roll back the kernel you just need to install those packages (on sd): http://armbian.hosthatch.com/apt/pool/main/l/linux-5.10.43-rockchip64/

 

[bunducafe]

vor 46 Minuten schrieb ebin-dev:

 

In order to roll back the kernel you just need to install those packages (on sd): http://armbian.hosthatch.com/apt/pool/main/l/linux-5.10.43-rockchip64/

 

 

Thanks. As the sd card is working flawlessly I am not in need to do so right now. And with the emmc issue I cannot write on that thing anyway. But just in case: It's done with "dpkg -i" I suppose. No specific order, right?

[ebin-dev]

@bunducafe dpkg -i *.deb did the job;  @IcerJo I would not trust a "temporary" fix I don't really understand. Helios64 saves your data...

 

[IcerJo]

31 minutes ago, ebin-dev said:

@bunducafe dpkg -i *.deb did the job;  @IcerJo I would not trust a "temporary" fix I do not understand.

 

In Essence I did what you stated but by using armbian-config to re-push the bootloader. How would one go about making the emmc Write-able in order to install the 3 packages to downgrade the kernel? Or would I need to re-setup OpenMediavault and everything, and run it from the SDcard or copy it over to the emmc from the SD Card with the old kernel, and then do a kernel freeze for time being till issue is resolved?

[ebin-dev]

35 minutes ago, IcerJo said:

How would one go about making the emmc Write-able in order to install the 3 packages to downgrade the kernel?

 

First I would downgrade the system on SD to 21.05.4 (you can access emmc again without i/o errors), then format the partition on emmc 'mkfs.ext4 /dev/mmcblk2p1' (to get rid of potentially corrupt content) and rsync the content from sd to emmc (assuming that the sd contains a valid copy of your system on emmc).

For the last step you can adapt the script I use (it is a modified Armbian script). I did no use any armbian-config routines this time.

You may need to update the bootloader on emmc too.

 

# cat copytoemmc.sh
#!/bin/bash

# Check if user is root
if [ $(id -u) != "0" ]; then
    echo "Error: You must be root to run this script."
    exit 1
fi

cat > install-exclude <<EOF
/dev/*
/proc/*
/sys/*
/media/data1/*
/media/data2/*
/media/data3/*
/media/data4/*
/media/data5/*
/mnt/sd/*
/mnt/emmc/*
/mnt/ssd/*
/mnt/usb/*
/mnt/hd/*
/run/*
# /tmp/*
# /root/*
EOF

exec 2>/dev/null
umount /mnt/emmc
exec 2>&1

mount /dev/mmcblk2p1 /mnt/emmc
rsync -avxSE --delete --exclude-from="install-exclude"  /  /mnt/emmc

# change fstab
sed -e 's/UUID=< insert uuid of sd >/UUID=< insert uuid of emmc >/g' -i /mnt/emmc/etc/fstab
sed -e 's/UUID=< insert uuid of sd >/UUID=< insert uuid of emmc >/g' -i /mnt/emmc/boot/armbianEnv.txt
umount /mnt/emmc
rm install-exclude

echo "All done."

 

[bunducafe]

vor 3 Stunden schrieb ebin-dev:

 

First I would downgrade the system on SD to 21.05.4 (you can access emmc again without i/o errors), then format the partition on emmc 'mkfs.ext4 /dev/mmcblk2p1' (to get rid of potentially corrupt content) and rsync the content from sd to emmc (assuming that the sd contains a valid copy of your system on emmc).

For the last step you can adapt the script I use (it is a modified Armbian script). I did no use any armbian-config routines this time.

You may need to update the bootloader on emmc too.

 

Is there any need to format the emmc and do a rsync? I mean, if I roll back and the i/o errors on the emmc are gone shouldn't it boot flawlessly again and preserves the exact same state as before the update?

[ebin-dev]

@bunducafeYou could try to update the bootloader on emmc first. @IcerJo claims that this would make it writable again. Then you could downgrade linux on emmc to 21.05.4. There is no need to format emmc - alternatively you could also repair errors in the filesystem with 'fsck -f /dev/mmcblk2p1'. You will see that something needs to be corrected. If you trust that everything is fine after that you are done.

EDIT: This does not work - since rewriting the bootloader does not solve the issue.

 

The intention of this thread was to show that Buster can be upgraded to Bullseye and that it is stable with Armbian 21.05.4 with linux-5.10.43.

 

Issues arise if Buster or Bullseye installations are updated from 21.05.4 to Armbian 21.08.1 with linux-5.10.60 against the recommendations earlier in this thread, because emmc will not be accessible anymore without i/o errors.

 

Edit: You also could boot a fresh Armbian 21.05.4 off  SD and rsync with it the content from emmc to another bootable SD. Then you continue to downgrade linux on that second SD (booted)  ... and rsync the result back to emmc.

Maybe somebody else could explain how to downgrade the kernel on emmc using a chrooted environment.

[IcerJo]

[mention=17374]bunducafe[/mention]You could try to update the bootloader on emmc first. [mention=16373]IcerJo[/mention] claims that this would make it writable again. Then you could downgrade linux on emmc to 21.05.4. There is no need to format emmc - alternatively you could also repair errors in the filesystem with 'fsck -f /dev/mmcblk2p1'. You will see that something needs to be corrected. If you trust that everything is fine after that you are done.

Seems like it doesn't make it fully writeable, but it does let the system boot and I'm able to access all of my files and save some changes in the system, like turn some services on and off. But I couldnt write the 3 files to the emmc to do a downgrade so unless I messed up I am at a loss but would gladly try the above mentioned.

Sent from my Pixel 4a (5G) using Tapatalk

[bunducafe]

vor 19 Minuten schrieb IcerJo:

Seems like it doesn't make it fully writeable, but it does let the system boot and I'm able to access all of my files and save some changes in the system, like turn some services on and off. But I couldnt write the 3 files to the emmc to do a downgrade so unless I messed up I am at a loss but would gladly try the above mentioned.

 

If you do fiddle around keep us posted, would be great to know if it works... I am somehow reluctant to try it at the moment as I need a running system. Because the mentioned workaround (updating bootloader) does not make the emmc fully writeable I would rather stick to the sd card at the moment.

[Igor]

eMMC troubles has been reported on other RK3399 boards such as Nanopi T4. Since we didn't do anything in this area I suspect someone pushed bad code upstream. Need analysis, while I don't see possibility to check and fix in the following weeks. 

[TDCroPower]

vor 16 Stunden schrieb ebin-dev:

@bunducafeYou could try to update the bootloader on emmc first. @IcerJo claims that this would make it writable again. Then you could downgrade linux on emmc to 21.05.4. There is no need to format emmc - alternatively you could also repair errors in the filesystem with 'fsck -f /dev/mmcblk2p1'. You will see that something needs to be corrected. If you trust that everything is fine after that you are done.

 

The intention of this thread was to show that Buster can be upgraded to Bullseye and that it is stable with Armbian 21.05.4 with linux-5.10.43.

 

Issues arise if Buster or Bullseye installations are updated from 21.05.4 to Armbian 21.08.1 with linux-5.10.60 against the recommendations earlier in this thread, because emmc will not be accessible anymore without i/o errors.

 

Edit: You also could boot a fresh Armbian 21.05.4 off  SD and rsync with it the content from emmc to another bootable SD. Then you continue to downgrade linux on that second SD (booted)  ... and rsync the result back to emmc.

Maybe somebody else could explain how to downgrade the kernel on emmc using a chrooted environment.

 

 

I unfortunately ran fsck last night as mentioned by you, which probably does not boot cleanly or I no longer have ssh access.
Maybe I have to connect it to the monitor and check what I see in the bootlog.

If someone could offer a step by step guide on how to recover I would be very grateful!