Latest Armbian - 25.5.0-trunk.185 no longer booting on Rock 4b

[Gwolf2u]

Hello guys

Been using Armbian on my RockPi 4b as a media server for few months now and took every update up until the latest 25.5.0-trunk.185

Now this was made with apt upgrade command and after I reboot, system no longer boots at all, freezing on initial boot stage.

No HDMI sign at all also.

If I plug an old armbian that I have on sdcard, I see U-Boot screen for few seconds, then reboots (basically bootloop).

OS was installed to internal eMMC.

Downgraded back to Armbian_community_25.5.0-trunk.131_Rockpi-4b_bookworm_current_6.12.16_minimal from github and all is running fine, but again after upgrade command, same thing happens.

Not that skilled to debug this further.

What I remember is that I saw in the upgrade log that kernel was updated, so thinking it might be that.

[Torgar]

The same thing happened with rockpi-4cplus. This happened after upgrading from kernel 6.12.17 to 6.12.18. After reboot no response and fail to boot. Only uses the eMMC chip.

[Gwolf2u]

@Torgar had the chance to check 6.12.19 ?

 

possible change fix ?

https://web.git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.12.y&id=39e4a0b613bd4d577321b207bb333f6213e8af6b

 

sudo apt show linux-image-current-rockchip64
Package: linux-image-current-rockchip64
Version: 25.5.0-trunk.230
Priority: optional
Section: kernel
Source: linux-6.12.19
Maintainer: Armbian Linux <info@armbian.com>
Installed-Size: 277 MB
Provides: linux-image, linux-image-armbian, armbian-current
Armbian-Kernel-Version: 6.12.19
Armbian-Kernel-Version-Family: 6.12.19-current-rockchip64
Armbian-Original-Hash: 6.12.19-Se9cc-Da873-Pfa20-C1f18H02eb-HK01ba-Vc222-B9bbb-R448a
Download-Size: 48.6 MB
APT-Sources: https://beta.armbian.com bookworm/main arm64 Packages
Description: Armbian Linux current kernel image 6.12.19-current-rockchip64
 This package contains the Linux kernel, modules and corresponding other files.
 version "6.12.19" git revision "e9cc806c0152fa9993f817cebf42989a3e2530bb" codename "Baby Opossum Posse" drivers hash "a873450a_3865dc87" patches hash "fa2093d7b29c5de9" .config hash "1f18a394d4cddb05" .config hook hash "02eb6b2bfb4dca2a" variables hash "c22207b66dc5dd57157df2cd7b9c20559b2ed8ac44b9c3c3b9704002c06b8921" framework bash hash "9bbbd4b74c282bc5"

 

Edited March 15 by Gabriel Nicolae Lup

[陈少举]

Sorry for my English.

This issue is same happen on NanoPi R2S,After write Armbian_community_25.5.0-trunk.185_Nanopi-r2s_bookworm_current_6.12.17_minimal.img to SD card, The R2S can't boot.

Rollback to Armbian_community_24.11.0-trunk.202_Nanopi-r2s_bookworm_current_6.6.53_minimal.img , Everything is performing well.

[Gwolf2u]

for now I've stopped kernel upgrades

sudo apt-mark hold linux-image-current-rockchip64 linux-dtb-current-rockchip64 linux-u-boot-rockpi-4b-current

can resume with

sudo apt-mark unhold linux-image-current-rockchip64 linux-dtb-current-rockchip64 linux-u-boot-rockpi-4b-current

 

did this since the option to disable armbian kernel upgrades in armbian-config is not working in my case for some reason

Edited March 16 by Gabriel Nicolae Lup

[陈少举]

This issue has been fixed in 25.5.0-trunk.256 or after version.

[Torte]

This could be a similar problem that I have had with the mks-klipad50 board (also a rockchip64).

In case of the bootloop, do the u-boot messages show "efi_free_pool" messages, directly followed by a "Synchronous Abort", like this? (The hex numbers can vary and there are a lot more registers dumped, this is just the start):

Starting kernel ...

efi_free_pool: illegal free 0x000000003cf21040
"Synchronous Abort" handler, esr 0x96000004

In that case, it is rather caused by the bootloader (u-boot) and the kernel version change had just a very indirect effect which triggered that problem.

[Gwolf2u]

In my case, armbian 25.5.0-trunk.256 still no go

same thing happens

not able to boot - rock pi 4b

Edited March 20 by Gwolf2u

[Torte]

@Gwolf2u (or anyone else): Are you able to provide a log from the bootloader when it goes into bootloop?

[Gwolf2u]

Personally not skilled enough to do it, but if no additional hardware needed for it, and let me know how to do it, I'll do it

[Torte]

With a usb-serial-adapter it would be easier, but well, it is how it is

 

Can you try pressing the spacebar when the device is bootlooping (keeping it pressed for a while)?
This should interrupt the bootloader - and with a bit of luck, the interesting lines are still visible on the screen, then you could simply take a photo and post it here (if the lines are readable).

[Gwolf2u]

Nothing happens on holding or taping space bar while booting

At least nothing on screen

Unfortunately don't have a adapter to check thing on serial

[imnlfn]

On 3/16/2025 at 2:49 AM, Gwolf2u said:

did this since the option to disable armbian kernel upgrades in armbian-config is not working in my case for some reason

 

I can't tell you how aggravating it is to read this, since this is the second time in three months that my Rock 4A has stopped booting after a kernel upgrade. I disabled kernel upgrades in armbian-config the last time it happened, but clearly that wasn't sufficient.

 

What's especially annoying is that in order to fix this, I have to take apart my whole device and disconnect my NVMe connection in order to boot from an SD card instead of the onboard eMMC.

 

I appreciate you pointing out how to stay on a specific kernel version using apt instead of relying on Armbian, since the latter is obviously unreliable. I'm likely to move to an x86 device in the not-too-distant future though, given occurrences like this, plus the lackluster support from the board vendor.

 

[imnlfn]

Fortunately, this time I didn't have to take apart my device in order to get it working again. It eventually allowed me to log in, and I was able to switch all of the links back to their original locations without too much trouble. This allowed me to regain access to my SSDs, and normal operation.

 

This time, it was the 6.14.0-rc7-edge version of the kernel that left out NVMe support again. How the absolute Hell did this happen twice in less than three months???

 

Does anyone know what the most recent kernel version is that is safe to use as far as NVMe/PCIe support is concerned?

 

This is frankly ridiculous. 

[Gwolf2u]

in my case last bootable is 6.12.17

[Igor]

12 hours ago, imnlfn said:

How the absolute Hell did this happen twice in less than three months???

 

This is expected and totally normal behavior we are seeing constantly. Without maintenance, support breaks apart. This is just the way it is. You can believe or understand.

 

All the time. And we can't afford to invest more if we already generate huge loss. Not a single vendor ever helped around old devices. While we don't have the millions needed for keeping devices in perfect state, we still invest hundreds of thousands each year to keep these devices running on at  'best effort' basis. Understandably, end users often - always expect military-grade software quality, even though they contribute nothing financially and refer back to hardware dealers for support.

 

On 3/26/2025 at 2:10 AM, imnlfn said:

plus the lackluster support from the board vendor.

 

Their support is cosmetic, some even fake that support (there are few companies stealing from us, while bragging how they invested into open source). It is their word against ours and it seems users believe dealers, which are always sweet and nice, more ...

 

Loss of the time is always a lot bigger what people can afford, so many people got broke, suffer mental breakdown when helping you (and HW dealers). 

 

 

Lack of your support / compensation is a fundamental problem.

 

12 hours ago, imnlfn said:

6.14.0-rc7-edge

 

Numbers and name suggests lack of polishing and stability, but you expect that things works perfectly?  I don't know how to describe you this in tl;dr; - this doesn't work this way. We have Linux kernel, general stuff, which is maintained by general Linux community. Then there is this device family, this device itself (here sadly even several badly compatible revisions). If nobody is dealing with it in general Linux community, which is usually the case, it often breaks at this level before anyone at Armbian even touches it. Then we apply family patch-set on top of this source. Patches often needs adjustments, but luckily they are usually trivial. Still, this takes time. A lot of. We usually don't fix devices at EDGE kernels. This mainly happens when its time to switch kernel at CURRENT (stable / LTS) branch. In 25.2 we switched from 6.6 -> 6.12. For that, our focus for half a year (!) were to 6.12 which was EDGE at that time. Stabilization of mainline kernel with functional addon is our major loss of time. And when it gets down to devices specifics, we are just more limited - we can't test all devices (+ all revisions = impossible) and we can't fix all of them. Recording a bug / known issue is already a luxury. And there is 99% our private time that is lost. Most of end users demands feels very insulting as you have no idea who pays the bills. You don't as also most of vendors don't want to hear about those troubles.

Since you don't support developers, people who you think to support you but they don't support you, support at least people which job is to listen to you and collect info for developers. From the loss those people generate, they can't support them better. But you can.

 

 

On 3/26/2025 at 2:10 AM, imnlfn said:

I appreciate you pointing out how to stay on a specific kernel version using apt instead of relying on Armbian, since the latter is obviously unreliable.

 

1. Switch to selected kernel

2. Freezing kernel upgrade

 

But bugs that can't be reproduce in virtual environment, are not related to Armbian.

 

[Gwolf2u]

So am trying here to provide more info.

Friend of mine borrowed me a usb to serial adapter, but clueless on what to do.

On Windows I know about Putty, but would like to do it on Ubuntu.

Below is my hardware, so please let me know how I can provide more info (what wire goes where) !?

[Torte]

Great!

See https://wiki.radxa.com/Rockpi4/dev/serial-console for a guide on how to connect the serial lines to the board and setting up a terminal program.

The wire colours are different, but take a look at the picture on that link:

• Do not connect the +5V or 3V3 pins
• GND cable needs to be connected to the 3rd GPIO-pin (GND) on the top/outer side of the connector (the black wire in the picture)
• RX cable needs to be connected to TX, that's the 4th GPIO-pin, just below the GND wire (it's white in the picture)
• TX cable needs to be connected to RX, that's the 5th GPIO-pin, just below the RX wire (it's green in the picture)

(If you can't connect, then RX and TX needs to be swapped. Don't worry, having these lines swapped won't break anything, it just doesn't allow a connection when wrong.)

 

Then start the terminal program (e.g. putty on Windows or minicom on Linux, both described in that link).

When booting the RockPi, you should see the boot messages in the terminal window (otherwise try swapping RX/TX).

 

Keeping the spacebar pressed (in the terminal program, make sure it is selected) should then interrupt the bootloop and you should be able to copy&paste the output.

If interrupting still does not work: Putty has the opption to copy the whole content to clipboard: Right-click the title bar and select "Copy All to Clipboard". Then paste it into an editor.

 

Once you have pasted that log into an editor, you can try to clean it up a little bit, it may contain several boot attempts. It would be sufficient to include just one complete boot attempt and the start of the next. But if in doubt, use the complete log.

Finally post that log here.

[MaxT]

@Gwolf2u

Hate to disappoint, but if your YP-01 serial converter is based on PL2303 chip, it will not work with Rockchip CPUs based boards as pl2303 doesn't support baud rate of 1 500 000 required for Rockchip CPUs (table 6-2 in the attached datasheet).

DOC001036118.pdf

[Gwolf2u]

OK so after I've connected successfully and ran the upgrade for kernel, what do you know, the OS booted :))

Anyway, managed to log it and here's a diff of boot from before and after the upgrade

https://www.diffchecker.com/ER6op67Q/

What I can tell is that some addresses for memory I think, seem to have changed

Anyway, thanks a lot guys for helping and eventually fixing the issue, even if I don't know what changed

Hopefully will continue to work fine for the foreseeable future

[Torte]

Thanks for publishing these logs.

Now we have to wait for the bootloop to happen, when the serial-adapter is still in reach .

If the loop happens directly at the "illegal free" messages, then switching to a more recent bootloader (u-boot 2025) could fix this.

[Gwolf2u]

 Did a fast search on Google for u-boot

Does u-boot need to be compiled or are binaries available for armbian?

[Torte]

u-boot has to be compiled specifically for the board that it runs on.
That means the Armbian project for the board has to be changed and it may require some additional source code changes - these additional requirements show up, when trying to boot with that new bootloader and something does not work.
I have managed to switch to a recent u-boot on a RK3328 board that I own (required changes can be seen in these commits) and I could try adopting that for the RockPi4b.
But as that change would affect all RockPi4b boards and I don't have a direct way to test it before publishing (it took me dozens of image builds and boot attempts), I am reluctant to give it a go without more indications that the bootloop is really caused by that specific bootloader problem: It may have a completely different cause and the u-boot switch may also introduce new problems.
So I'd rather wait until that loop can be reproduced and the log proves u-boot being the cause before opening that can of worms.