Stability with kernel 5.15?

[axeleroy]

Hello,

 

I have been holding onto kernel 5.10 (5.10.21-rockchip64 to be exact) for a while as it has been pretty stable for me and also because I heard bad feedback from newer kernels on the Helios 64.

But with the recent discovery of the DirtyPipe exploit (which has been introduced in Linux 5.8), I might be looking to upgrade to the latest Armbian kernel (which contains a fix to the vulnerability).

 

So, my question to fellow Helios64 users is: have you encountered any issue with kernel 5.15, especially when using OpenMediaVault 5, Docker, MergerFS and NFS?

[Zac]

Hey there!

 

I've was not sure about upgrading either, and in the end, for security reasons as well, I did upgrade. 

I'm using OMV 5, and docker (although right now I'm not running anything on it).  I'm switching governors between ondemand and powersave to reduce power consumption, as I'm currently not running anything too cpu hungry. I've been running for about 10 days now, with no stability issue.  But as I said, my device is mostly idle.

[rjgould]

On 3/16/2022 at 11:00 AM, axeleroy said:

Hello,

 

I have been holding onto kernel 5.10 (5.10.21-rockchip64 to be exact) for a while as it has been pretty stable for me and also because I heard bad feedback from newer kernels on the Helios 64.

But with the recent discovery of the DirtyPipe exploit (which has been introduced in Linux 5.8), I might be looking to upgrade to the latest Armbian kernel (which contains a fix to the vulnerability).

 

So, my question to fellow Helios64 users is: have you encountered any issue with kernel 5.15, especially when using OpenMediaVault 5, Docker, MergerFS and NFS?

I've actually just re-installed my Helios64 from scratch in an attempt to make it useful.

 

I nabbed the latest Bullseye image from the archives and ran the updates.  it's now running Armbian 22.02.1 with Linux 5.15.25-rockchip64.  I've not noticed any stability issues so far 🤞

 

I also installed OMV6 on top without any errors.  None of this was an upgrade though so your mileage may vary.

[ebin-dev]

Hi,

 

updating Helios64 (Armbian Bullseye) to the latest kernel (Armbian 22.02.1 with Linux 5.15.25-rockchip64) went flawlessly - it seems to be stable so far (Thanks!).

However, emmc read/write speed is still reduced by a factor of 2 compared to linux 5.10.43.

 

@piter75 Do you know if there is anybody trying to re-enable high emmc read/write speeds in newer kernels  ?

[freed00m]

So far so good on my side.

 

Though I use only plain setup, mdadm 2 drives mirror raid + extra 3 satas for transmission downloading.

 

And just transferred 1TB via Rclone on another over the internet NAS that took 8h and have not observed a fail.

 

Running OS from emmc and I don't write often as apt cache moved to SATA drive. Didn't test speeds prior upgrade. I have helios64 from the last latest batch, those with 2.5gbit fixed.

 

[bunducafe]

Since the emmc bug I am kind of reluctant to put the OS again on that device. I have gone back with SD Cards that I can easily backup once in a while. That said I always have a running backup.

[crosser]

Kernel 5.15 is unusable for me (helios64), 5.15.25 and 5.15.35 do not see internal mmc:

[    2.404726] mmc1: SDHCI controller on fe330000.mmc [fe330000.mmc] using ADMA
[    2.474955] mmc1: mmc_select_hs200 failed, error -110
[    2.475458] mmc1: error -110 whilst initialising MMC card
[    2.592202] mmc1: mmc_select_hs200 failed, error -110
[    2.592665] mmc1: error -110 whilst initialising MMC card
[    2.724392] mmc1: mmc_select_hs200 failed, error -110
[    2.724853] mmc1: error -110 whilst initialising MMC card

It works with all previous kernels (up to 5.10.63), u-boot boots from it just fine.

Is it a known problem?

[prahal]

On 6/3/2022 at 10:21 PM, crosser said:

Kernel 5.15 is unusable for me (helios64), 5.15.25 and 5.15.35 do not see internal mmc:

[    2.404726] mmc1: SDHCI controller on fe330000.mmc [fe330000.mmc] using ADMA
[    2.474955] mmc1: mmc_select_hs200 failed, error -110
[    2.475458] mmc1: error -110 whilst initialising MMC card
[    2.592202] mmc1: mmc_select_hs200 failed, error -110
[    2.592665] mmc1: error -110 whilst initialising MMC card
[    2.724392] mmc1: mmc_select_hs200 failed, error -110
[    2.724853] mmc1: error -110 whilst initialising MMC card

It works with all previous kernels (up to 5.10.63), u-boot boots from it just fine.

Is it a known problem?

 

Do you mean all kernel from below 5.10.63 up to 5.10.63 ?

If so could you report your serial console u-boot boot output ?

5.10.60 in https://mirrors.xtom.de/armbian/pool/main/l/linux-5.10.60-rockchip64/ are broken for most (all) of us.

It would be really helpfull if an u-boot version could be related to working vs not working

I found a bad commit in the kernel between 5.10.43 and 5.10.60 bu it makes no sense to me that it breaks emmc (though I don't know much about emmc)

 

But if you manage to run 5.10.60 on emmc that would help us both.

-110 means timeout attempting to contact the emmc,

[crosser]

On 6/8/2022 at 12:51 AM, prahal said:

 

Do you mean all kernel from below 5.10.63 up to 5.10.63 ?

If so could you report your serial console u-boot boot output ?

5.10.60 in https://mirrors.xtom.de/armbian/pool/main/l/linux-5.10.60-rockchip64/ are broken for most (all) of us.

It would be really helpfull if an u-boot version could be related to working vs not working

I found a bad commit in the kernel between 5.10.43 and 5.10.60 bu it makes no sense to me that it breaks emmc (though I don't know much about emmc)

 

But if you manage to run 5.10.60 on emmc that would help us both.

-110 means timeout attempting to contact the emmc,

I remember _one_ version of the kernel (released as a package in the focal channel) about a year ago was broken, but I don't remember details. I do remember that I booted from the (stock) SD, ran dist-upgrade, and things returned to norm. It is possible that 5.10.60 was the one that was broken for me too.

 

I am attaching a log from reset to "Welcome" when the system is booted, with kernel 5.15.35. At this point, internal mmc is not visible in the system (and obviously not mountable).

 

Eugene

kobolboot-5.15.35-fail-to-mount

[prahal]

Sorry it took me so long (investigating the hard freezes on my side took too much time, I hope to have a simple reproducer of the freeze one day).

 

So with your u-boot instead of the current armbian one I noticed a new error at the point you get emmc failures

[    2.556749] mmc1: SDHCI controller on fe330000.mmc [fe330000.mmc] using ADMA
[    2.627809] mmc1: Command Queue Engine enabled
[    2.628301] mmc1: new HS200 MMC card at address 0001
[    2.629765] mmcblk1: mmc1:0001 AJTD4R 14.6 GiB 
[    2.633459]  mmcblk1: p1
[    2.634455] mmcblk1boot0: mmc1:0001 AJTD4R 4.00 MiB 
[    2.637466] mmcblk1boot1: mmc1:0001 AJTD4R 4.00 MiB 
[    2.640380] mmcblk1rpmb: mmc1:0001 AJTD4R 4.00 MiB, chardev (241:0)
[    2.720113] usb usb1-port1: Cannot enable. Maybe the USB cable is bad?

that is the usb cable is bad (which could be power related). I do not have this error with current armbian helios64 u-boot.

 

I noticed you boot from NVME and from Helios64_FCC_CE_Test_Manual_v0.2.pdf the M.2 Slot is for SATA SSD
or USB 2.0 Device.

Thus I suspect the M.2 port plus the  power issue

 

I suggest you to try to upgraded your u-boot and retry. With linux-u-boot-helios64-current at 21.08.9, do armbian-config > System > Install > and there from your logs it seems you should

chose "5  Install/Update the bootloader on SD/eMMC" .

 

If you want to restore to you current u-boot install:

apt install linux-u-boot-helios64-current=21.08.2

 

EDIT: sorry I told you to chose ""3  Boot from eMMC - system on SATA, USB or NVMe", I copy pasted too fast (this would show a warning that tell you that the system partition will be rewritten so you might have guessed that it was a bad idea). "Install/Update the bootloader on SD/eMMC" is  the way to update u-boot.

 

Alban

Edited June 18, 2022 by prahal

[prahal]

I found a similar issue in the forum:

The user https://forum.armbian.com/profile/16806-mortomanos/

was provided an armbian overlay that fixed his emmc -110 issue. But this overlay was not disclosed. Maybe you could ask him (and share the overlay or ask him to share the overlay).

 

Alban

[bunducafe]

Just out of curiosity: On which kernel are you guys on at the moment?

 

With the latest updates I have several hickups, meanining: random reboots... they occur after some days, sometimes 4, sometimes up to 10. As I have not written down the kernel I was working flawless for quite some time. I will do my homework better now but as I am running my machine headless once it reboots all the logs are gone what went wrong... 

[crosser]

There's been a problem when 5.15 was first rolled out (mmc access, fan control, ...) and I pinned 5.10 for a while.

 

A couple of months ago I tried Lunar with edge kernel (6.1) and it worked fine. So I switched and run it now, without any problems(*). I tried 5.15 as well, and it worked fine too. Things look quite good now, great thanks to the unsung heroes who are keeping the distro for this hardware alive!

 

(*) I use NFS server, and due to a problem not specific to the kernel, it fails to start at boot. That is because `nfsd` module is not included in initramfs, and `proc-fs-nfsd.mount` unit tries to load before the "real" root fs is mounted. And fails, and prevents all other nfs services from starting. That is solved by adding `After=local-fs.target` to the unit.

[bunducafe]

I have never encountered fan issues with 5.15.x but the mentioned reboots. Nice insight the kernel 6 is also running smoothly. I will give it a try later this week and see what happens. I will report back.

 

@crosser Did you update the kernel via armbian-config?