Jump to content

Orangepi3b kernel panic when trying to boot with the new vendor kernel(6.1.43)


Go to solution Solved by Protx,

Recommended Posts

Posted (edited)

Hi.

I testing out different images for my DIY NAS project.  My plan is to use the Orangepi3b board with armbian bookworm and OMV.

I have built images from armbian github, (main,trunk) the edge 6.7.10 kernel is working great. But I do get some Iperf3 retries when sending from Opi3b.

From Opi3b 60-70MB/s To Opi3b 110MB/s. With Orangepi own bookworm image I have 110/110 and no retries. The same with Joshua Riek's Ubuntu server image, no retries and 110/110.

 

Orangepi3b armbian with 6.7.10 kernel

orangepi@orangepi3b:~$ iperf3 -c 192.168.10.197 -f M
Connecting to host 192.168.10.197, port 5201
[  5] local 192.168.10.155 port 56514 connected to 192.168.10.197 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  74.5 MBytes  74.5 MBytes/sec  217   19.8 KBytes       
[  5]   1.00-2.00   sec  75.9 MBytes  75.9 MBytes/sec  218   11.3 KBytes       
[  5]   2.00-3.00   sec  79.1 MBytes  79.1 MBytes/sec  200   25.5 KBytes       
[  5]   3.00-4.00   sec  73.3 MBytes  73.3 MBytes/sec  225   29.7 KBytes       
[  5]   4.00-5.00   sec  79.1 MBytes  79.1 MBytes/sec  210   33.9 KBytes       
[  5]   5.00-6.00   sec  77.8 MBytes  77.8 MBytes/sec  211   86.3 KBytes       
[  5]   6.00-7.00   sec  77.4 MBytes  77.4 MBytes/sec  202   36.8 KBytes       
[  5]   7.00-8.00   sec  76.1 MBytes  76.1 MBytes/sec  207   73.5 KBytes       
[  5]   8.00-9.00   sec  75.0 MBytes  75.0 MBytes/sec  222   12.7 KBytes       
[  5]   9.00-10.00  sec  73.3 MBytes  73.3 MBytes/sec  239   11.3 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   761 MBytes  76.1 MBytes/sec  2151             sender
[  5]   0.00-10.04  sec   760 MBytes  75.7 MBytes/sec                  receiver

 

Orangepi3b Ubuntu with 6.1 kernel

ubuntu@ubuntu:~$ iperf3 -R -c 192.168.10.197 -f M
Connecting to host 192.168.10.197, port 5201
Reverse mode, remote host 192.168.10.197 is sending
[  5] local 192.168.10.154 port 42888 connected to 192.168.10.197 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   1.00-2.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   2.00-3.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   3.00-4.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   4.00-5.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   5.00-6.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   6.00-7.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   7.00-8.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   8.00-9.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   9.00-10.00  sec   112 MBytes   112 MBytes/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  1.10 GBytes   112 MBytes/sec    1             sender
[  5]   0.00-10.00  sec  1.10 GBytes   112 MBytes/sec                  receiver

 

So to see if the vendor kernel was different I built with the new 6.1.43-vendor-rk35xx kernel, but It will not boot.

build command:  ./compile.sh build BOARD=orangepi3b BRANCH=edge BUILD_DESKTOP=no BUILD_MINIMAL=yes KERNEL_CONFIGURE=no RELEASE=bookworm J=4

 

It seems to be something with power to the NPU. ?

"rockchip-pm-domain fdd90000.power-management:power-controller: failed to get ack on domain 'npu', target_idle = 0, target_ack = 0, val=0x6"

 

uart log attachment



 

kernelpanic.log

Edited by Protx
To long log file
  • Solution
Posted (edited)

I now have the Armbian-unofficial_24.5.0-trunk_Orangepi3b_bookworm_vendor_6.1.43_minimal image booting on my orangpi3b. This is an unsupported board, and the images is built from trunk and bleeding edge, so I guess it is expected to have some issues. Anyway I post my findings in case other have the same problem.

Iper3 works without any retries. So next is to install OMV7 and see the file transfer speed.

 

 

orangepi@orangepi3b:~$ iperf3 -c 192.168.10.10 -f M
Connecting to host 192.168.10.10, port 5201
[  5] local 192.168.10.227 port 41766 connected to 192.168.10.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   114 MBytes/sec    0    407 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   112 MBytes/sec    0    407 KBytes       
[  5]   2.00-3.00   sec   113 MBytes   113 MBytes/sec    0    428 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   112 MBytes/sec    0    428 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   112 MBytes/sec    0    451 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   112 MBytes/sec    0    451 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   112 MBytes/sec    0    451 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   112 MBytes/sec    0    451 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   112 MBytes/sec    0    451 KBytes       
[  5]   9.00-10.00  sec   113 MBytes   113 MBytes/sec    0    451 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   112 MBytes/sec    0             sender
[  5]   0.00-10.04  sec  1.10 GBytes   112 MBytes/sec                  receiver

iperf Done.

 

I did not know that the bootloader/u-boot on the SPI and EMMC could have impact on booting of SD card, because all images with edge kernel booted just fine from SD card. I have an EMMC with armbian-bookworm-edge-minimal installed, and have flashed the SPI with the u-boot version installed on the EMMC.

Error when trying to boot a vendor image from SD card:

 

[    9.413438] rockchip-pm-domain fdd90000.power-management:power-controller: failed to get ack on domain 'npu', target_idle = 0, target_ack = 0, val=0x6
[    9.413543] Kernel panic - not syncing: panic_on_set_idle set ...

 

Then I tried other images, armbian with legacy kernel and ubuntu24.04beta-6.1 kernel. Both with the same error.
I then flashed the ubuntu's u-boot on the SPI and removed the EMMC. Then the both images booted.
But the armbian-bookworm-vendor-mininal still would not boot. But this time it stopped with another error:

 

[   23.575850] mtty_probe init device addr: 0x000000004f52d231
[   23.576178] -->rfkill_bluetooth_init
[   23.576456] <--rfkill_bluetooth_init
[   23.577060] sysfs: cannot create duplicate filename '/devices/virtual/tty/ttyBT0'
[   23.577107] CPU: 1 PID: 381 Comm: systemd-modules Not tainted 6.1.43-vendor-rk35xx #1
[   23.577127] Hardware name: Rockchip RK3566 OPi 3B (DT)
[   23.577141] Call trace:
[   23.577153]  dump_backtrace+0xe4/0x108

 

Next I erased the SPI, and with no EMMC the only bootloader on the SBC was on the SD card.
Then the armbian-bookworm-vendor-mininal image stopped with the first kernel panic again:

 

[    9.413438] rockchip-pm-domain fdd90000.power-management:power-controller: failed to get ack on domain 'npu', target_idle = 0, target_ack = 0, val=0x6
[    9.413543] Kernel panic - not syncing: panic_on_set_idle set ...

 

So I made changes and reverted to an older version of uboot:

	BOOTBRANCH="commit:63073b4af636146d26a7f0f258610eed060c8f34" # specific commit, from "branch:rk3568-2023.10"
	BOOTDIR="u-boot-${BOARD}"                                    # do not share u-boot directory
	BOOTPATCHDIR="v2023.10-orangepi3b"                           # empty, patches are already in Kwiboo's branch:rk3568-2023.10

revert-u-boot-orangepi3b.patch

Again booting from SD card I got the "duplicate filename" bluetooth error again.
So after scratching my head and poking around I noticed that not all the UW5622 patches was applied?:

 

# Apply patches that adjust the driver only for rockchip platforms
		if [[ "$LINUXFAMILY" == rockchip* ]]; then
			if linux-version compare "${version}" le 6.1; then
				process_patch_file "${SRC}/patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-pre-6.1.patch"
			else
				process_patch_file "${SRC}/patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-post-6.1.patch"
			fi
		fi

 

Since the vendor kernel is defined as "$LINUXFAMILY" == rk35xx ? this part of the patch does not get applied.
And since the vendor kernel is 6.1.43, it should use the post version of the patch?, aka the "le" won't work.?
I changed the script to:

		# Apply patches that adjust the driver only for rockchip platforms
		if [[ "$LINUXFAMILY" == rockchip* || "$LINUXFAMILY" == "rk35xx" ]]; then
			if linux-version compare "${version}" ge 6.1; then
 				process_patch_file "${SRC}/patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-post-6.1.patch"
			else
				process_patch_file "${SRC}/patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-pre-6.1.patch"
		fi

uw5622-fix-adjust-for-rockchip-post-6.1.patch

Compiled, now it boots from SD and seems to work just fine.

Edited by Protx
Posted

hi Protx:

Hello, I have just started learning the compilation of Armbian. I am also using the OrangePI3b with the RK3566. I need to study the startup of rknpu in version 6.1.43. I downloaded the latest repository from https://github.com/armbian/build, specifically the main branch. I compiled it according to your compilation configuration. There was an error message upon booting.

** Booting bootflow 'mmc@fe2b0000.bootdev.part_1' with script
Boot script loaded from mmc 1:1
155 bytes read in 3 ms (49.8 KiB/s)
8598033 bytes read in 724 ms (11.3 MiB/s)
37388800 bytes read in 3133 ms (11.4 MiB/s)
167999 bytes read in 47 ms (3.4 MiB/s)
Working FDT set to a100000
Trying kaslrseed command... Info: Unknown command can be safely ignored since kaslrseed does not apply to all boards.
Unknown command 'kaslrseed' - try 'help'
Moving Image from 0x2080000 to 0x2200000, end=4660000
## Loading init Ramdisk from Legacy Image at 0a200000 ...
   Image Name:   uInitrd
   Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
   Data Size:    8597969 Bytes = 8.2 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 0a100000
   Booting using the fdt blob at 0xa100000
Working FDT set to a100000
   Loading Ramdisk to ec68e000, end ecec11d1 ... OK
ERROR: reserving fdt memory region failed (addr=0 size=0 flags=0)
ERROR: reserving fdt memory region failed (addr=0 size=0 flags=0)
   Loading Device Tree to 00000000ec5fc000, end 00000000ec68dfff ... OK
Working FDT set to ec5fc000

Starting kernel ...

[   14.607208] Internal error: Oops: 0000000096000044 [#1] SMP
[   14.607281] pwm-backlight backlight: Looking up power-supply from device tree
[   14.607780] Modules linked in:
[   14.607848] pwm-backlight backlight: Looking up power-supply property in node /backlight failed
[   14.608469]  sprdbt_tty(+)
[   14.608536] pwm-backlight backlight: supply power not found, using dummy regulator
[   14.608569]  fuse dm_mod ip_tables ipv6 panel_simple pwm_bl
[   14.611186] CPU: 1 PID: 312 Comm: systemd-modules Tainted: G        W          6.1.43-vendor-rk35xx #1
[   14.612092] Hardware name: Rockchip RK3566 OPi 3B (DT)
[   14.612601] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   14.613282] pc : tty_unregister_driver+0x44/0x78
[   14.613752] lr : tty_unregister_driver+0x40/0x78
[   14.614212] sp : ffff80000d02b7c0
[   14.614544] x29: ffff80000d02b7c0 x28: ffff80000a10f0c8 x27: 0000000000000000
[   14.615242] x26: 0000000000000000 x25: ffff80000a23f000 x24: ffff0000092f8400
[   14.615937] x23: ffff80000122d000 x22: ffff80000122e178 x21: 00000000ffffffef
[   14.616632] x20: ffff80000a2272e0 x19: ffff000008c7fb00 x18: 0000000000000000
[   14.617328] x17: 7420797274207427 x16: 6e6f64202c545349 x15: 5845452d20687469
[   14.618028] x14: 7720305442797474 x13: 2e79726f74636572 x12: 696420656d617320
[   14.618724] x11: 656874206e692065 x10: 6d616e20656d6173 x9 : ffff80000915ee50
[   14.619420] x8 : 696166206c616e72 x7 : 65746e695f646461 x6 : 0000000000000000
[   14.620115] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[   14.624209] x2 : ffff000001731c80 x1 : dead000000000100 x0 : dead000000000122
[   14.628287] Call trace:
[   14.631921]  tty_unregister_driver+0x44/0x78
[   14.635732]  mtty_probe+0x1e8/0x2b0 [sprdbt_tty]
[   14.639572]  platform_probe+0x70/0xc0
[   14.643282]  really_probe+0x1cc/0x390
[   14.646981]  __driver_probe_device+0x140/0x158
[   14.650736]  driver_probe_device+0x48/0xd0
[   14.654452]  __device_attach_driver+0xd8/0x128
[   14.658174]  bus_for_each_drv+0xa0/0xc8
[   14.661870]  __device_attach+0xf8/0x180
[   14.665557]  device_initial_probe+0x1c/0x28
[   14.669266]  bus_probe_device+0x38/0x9c
[   14.672951]  device_add+0x550/0x688
[   14.676590]  platform_device_add+0xec/0x224
[   14.680286]  mtty_pdev_init+0x50/0x1000 [sprdbt_tty]
[   14.684061]  do_one_initcall+0x94/0x1e4
[   14.687709]  do_init_module+0x58/0x1e0
[   14.691329]  load_module+0x1818/0x18e0
[   14.694906]  __do_sys_finit_module+0xf8/0x118
[   14.698524]  __arm64_sys_finit_module+0x24/0x30
[   14.702132]  invoke_syscall+0x8c/0x128
[   14.705606]  el0_svc_common.constprop.0+0xd8/0x128
[   14.709161]  do_el0_svc+0xac/0xbc
[   14.712551]  el0_svc+0x2c/0x54
[   14.715888]  el0t_64_sync_handler+0xac/0x13c
[   14.719319]  el0t_64_sync+0x19c/0x1a0
[   14.722684] 
[   14.722684] PC: 0xffff80000875f79c:
[   14.723515] rk_pcie_establish_link: 132 callbacks suppressed


I found that the latest repository does not match your operating instructions. For example, the commit 63073b4af636146d26a7f0f258610eed060c8f34 in the Kwiboog's uboot repository branch rk3568-2023.10 is no longer found. However, the compilation does not report any errors. Could you please share your compilation repository and configuration?
Thank you.

@Protx

kernel.log

Posted (edited)

@darcyg

 

First let me say that I am no expert on this. I have not tried to compile it lately, but I see that that commit does not longer exist on kwiboo's github. So I don't now if you need to revert to an older version of u-boot. And if you have an EmmC og SPI with u-boot, then it should work to boot from SC card with the new uboot anyway.

 

My log was with debug enabled (and from uart), and I dont have that log anymore.

mtty_probe+0x1e8/0x2b0 [sprdbt_tty]

So I am not sure the error is the same wireless / bluetooth related error that I got . But I think it is.

Make sure that the patch uw5622-fix-adjust-for-rockchip-post-6.1.patch got applied.

 

Take a look in the /output/logs, If you have build the kernel several times it could be using a cached kernel and not included in the log!?, (look in the largest log files). And verify that uw5622-fix-adjust-for-rockchip-post-6.1.patch  was included in your compilation:

--> (15) INFO: Adding [ Drivers for Unisoc uwe5622 found on some Allwinner and Rockchip boards ]
--> (15) INFO: * applying patch/misc/wireless-uwe5622/uwe5622-allwinner.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/uwe5622-allwinner-bugfix.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/uwe5622-warnings.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/uwe5622-park-link-v6.1-post.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/uwe5622-v6.1.patch
--> (15) INFO: *  patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-post-6.1.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/wireless-uwe5622-Fix-compilation-with-6.7-kernel.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/wireless-uwe5622-reduce-system-load.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/w-wireless-fix-out-of-tree-build.patch
--> (15) INFO: Preparing driver [ driver_rtl8723cs ]

 

Edited by Protx
Posted (edited)

@darcyg

 

What i meant was that you manually apply the uw5622-fix-adjust-for-rockchip-post-6.1.patch before ./compile command.

Then after compilation check that :

--> (15) INFO: *  patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-post-6.1.patch

is included in the log file.

 

BTW: Do not mind the

* applying patch/misc/wireless-uwe5622/w-wireless-fix-out-of-tree-build.patch

That was a patch I saw was included in the ubuntu rockchip kernel. I do not think that is needed, does not seem to make a difference.

Edited by Protx
Posted

@Protx 
Thank you very much. I am currently compiling other systems, and I will test your solution later. Additionally, I have found that the latest official trunk403 build, as well as the previous trunk250, both exhibit this error. I am testing using an SD card.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines