Jump to content

Orangepi3b kernel panic when trying to boot with the new vendor kernel(6.1.43)

Protx

By Protx
in Rockchip

 Share
Go to solution Solved by Protx,

Recommended Posts

Protx

Protx

Posted
Posted (edited)

Hi.

I testing out different images for my DIY NAS project.  My plan is to use the Orangepi3b board with armbian bookworm and OMV.

I have built images from armbian github, (main,trunk) the edge 6.7.10 kernel is working great. But I do get some Iperf3 retries when sending from Opi3b.

From Opi3b 60-70MB/s To Opi3b 110MB/s. With Orangepi own bookworm image I have 110/110 and no retries. The same with Joshua Riek's Ubuntu server image, no retries and 110/110.

 

Orangepi3b armbian with 6.7.10 kernel 

orangepi@orangepi3b:~$ iperf3 -c 192.168.10.197 -f M
Connecting to host 192.168.10.197, port 5201
[  5] local 192.168.10.155 port 56514 connected to 192.168.10.197 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  74.5 MBytes  74.5 MBytes/sec  217   19.8 KBytes       
[  5]   1.00-2.00   sec  75.9 MBytes  75.9 MBytes/sec  218   11.3 KBytes       
[  5]   2.00-3.00   sec  79.1 MBytes  79.1 MBytes/sec  200   25.5 KBytes       
[  5]   3.00-4.00   sec  73.3 MBytes  73.3 MBytes/sec  225   29.7 KBytes       
[  5]   4.00-5.00   sec  79.1 MBytes  79.1 MBytes/sec  210   33.9 KBytes       
[  5]   5.00-6.00   sec  77.8 MBytes  77.8 MBytes/sec  211   86.3 KBytes       
[  5]   6.00-7.00   sec  77.4 MBytes  77.4 MBytes/sec  202   36.8 KBytes       
[  5]   7.00-8.00   sec  76.1 MBytes  76.1 MBytes/sec  207   73.5 KBytes       
[  5]   8.00-9.00   sec  75.0 MBytes  75.0 MBytes/sec  222   12.7 KBytes       
[  5]   9.00-10.00  sec  73.3 MBytes  73.3 MBytes/sec  239   11.3 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   761 MBytes  76.1 MBytes/sec  2151             sender
[  5]   0.00-10.04  sec   760 MBytes  75.7 MBytes/sec                  receiver

 

Orangepi3b Ubuntu with 6.1 kernel 

ubuntu@ubuntu:~$ iperf3 -R -c 192.168.10.197 -f M
Connecting to host 192.168.10.197, port 5201
Reverse mode, remote host 192.168.10.197 is sending
[  5] local 192.168.10.154 port 42888 connected to 192.168.10.197 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   1.00-2.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   2.00-3.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   3.00-4.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   4.00-5.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   5.00-6.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   6.00-7.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   7.00-8.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   8.00-9.00   sec   112 MBytes   112 MBytes/sec                  
[  5]   9.00-10.00  sec   112 MBytes   112 MBytes/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  1.10 GBytes   112 MBytes/sec    1             sender
[  5]   0.00-10.00  sec  1.10 GBytes   112 MBytes/sec                  receiver

 

So to see if the vendor kernel was different I built with the new 6.1.43-vendor-rk35xx kernel, but It will not boot.

build command:  ./compile.sh build BOARD=orangepi3b BRANCH=edge BUILD_DESKTOP=no BUILD_MINIMAL=yes KERNEL_CONFIGURE=no RELEASE=bookworm J=4

 

It seems to be something with power to the NPU. ?

"rockchip-pm-domain fdd90000.power-management:power-controller: failed to get ack on domain 'npu', target_idle = 0, target_ack = 0, val=0x6"

 

uart log attachment



 

kernelpanic.log

Edited by Protx
To long log file
0
Link to comment
Share on other sites
  • Solution
Protx

Protx

Posted
  • Author
  • Solution
Posted (edited)

I now have the Armbian-unofficial_24.5.0-trunk_Orangepi3b_bookworm_vendor_6.1.43_minimal image booting on my orangpi3b. This is an unsupported board, and the images is built from trunk and bleeding edge, so I guess it is expected to have some issues. Anyway I post my findings in case other have the same problem.

Iper3 works without any retries. So next is to install OMV7 and see the file transfer speed.

 

  

orangepi@orangepi3b:~$ iperf3 -c 192.168.10.10 -f M
Connecting to host 192.168.10.10, port 5201
[  5] local 192.168.10.227 port 41766 connected to 192.168.10.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   114 MBytes/sec    0    407 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   112 MBytes/sec    0    407 KBytes       
[  5]   2.00-3.00   sec   113 MBytes   113 MBytes/sec    0    428 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   112 MBytes/sec    0    428 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   112 MBytes/sec    0    451 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   112 MBytes/sec    0    451 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   112 MBytes/sec    0    451 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   112 MBytes/sec    0    451 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   112 MBytes/sec    0    451 KBytes       
[  5]   9.00-10.00  sec   113 MBytes   113 MBytes/sec    0    451 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   112 MBytes/sec    0             sender
[  5]   0.00-10.04  sec  1.10 GBytes   112 MBytes/sec                  receiver

iperf Done.

 

I did not know that the bootloader/u-boot on the SPI and EMMC could have impact on booting of SD card, because all images with edge kernel booted just fine from SD card. I have an EMMC with armbian-bookworm-edge-minimal installed, and have flashed the SPI with the u-boot version installed on the EMMC.

Error when trying to boot a vendor image from SD card:

  

[    9.413438] rockchip-pm-domain fdd90000.power-management:power-controller: failed to get ack on domain 'npu', target_idle = 0, target_ack = 0, val=0x6
[    9.413543] Kernel panic - not syncing: panic_on_set_idle set ...

 

Then I tried other images, armbian with legacy kernel and ubuntu24.04beta-6.1 kernel. Both with the same error.
I then flashed the ubuntu's u-boot on the SPI and removed the EMMC. Then the both images booted.
But the armbian-bookworm-vendor-mininal still would not boot. But this time it stopped with another error:

  

[   23.575850] mtty_probe init device addr: 0x000000004f52d231
[   23.576178] -->rfkill_bluetooth_init
[   23.576456] <--rfkill_bluetooth_init
[   23.577060] sysfs: cannot create duplicate filename '/devices/virtual/tty/ttyBT0'
[   23.577107] CPU: 1 PID: 381 Comm: systemd-modules Not tainted 6.1.43-vendor-rk35xx #1
[   23.577127] Hardware name: Rockchip RK3566 OPi 3B (DT)
[   23.577141] Call trace:
[   23.577153]  dump_backtrace+0xe4/0x108

 

Next I erased the SPI, and with no EMMC the only bootloader on the SBC was on the SD card.
Then the armbian-bookworm-vendor-mininal image stopped with the first kernel panic again:

  

[    9.413438] rockchip-pm-domain fdd90000.power-management:power-controller: failed to get ack on domain 'npu', target_idle = 0, target_ack = 0, val=0x6
[    9.413543] Kernel panic - not syncing: panic_on_set_idle set ...

 

So I made changes and reverted to an older version of uboot: 

	BOOTBRANCH="commit:63073b4af636146d26a7f0f258610eed060c8f34" # specific commit, from "branch:rk3568-2023.10"
	BOOTDIR="u-boot-${BOARD}"                                    # do not share u-boot directory
	BOOTPATCHDIR="v2023.10-orangepi3b"                           # empty, patches are already in Kwiboo's branch:rk3568-2023.10

revert-u-boot-orangepi3b.patch

Again booting from SD card I got the "duplicate filename" bluetooth error again.
So after scratching my head and poking around I noticed that not all the UW5622 patches was applied?:

  

# Apply patches that adjust the driver only for rockchip platforms
		if [[ "$LINUXFAMILY" == rockchip* ]]; then
			if linux-version compare "${version}" le 6.1; then
				process_patch_file "${SRC}/patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-pre-6.1.patch"
			else
				process_patch_file "${SRC}/patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-post-6.1.patch"
			fi
		fi

 

Since the vendor kernel is defined as "$LINUXFAMILY" == rk35xx ? this part of the patch does not get applied.
And since the vendor kernel is 6.1.43, it should use the post version of the patch?, aka the "le" won't work.?
I changed the script to: 

		# Apply patches that adjust the driver only for rockchip platforms
		if [[ "$LINUXFAMILY" == rockchip* || "$LINUXFAMILY" == "rk35xx" ]]; then
			if linux-version compare "${version}" ge 6.1; then
 				process_patch_file "${SRC}/patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-post-6.1.patch"
			else
				process_patch_file "${SRC}/patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-pre-6.1.patch"
		fi

uw5622-fix-adjust-for-rockchip-post-6.1.patch

Compiled, now it boots from SD and seems to work just fine.

Edited by Protx
1
Link to comment
Share on other sites
darcyg

darcyg

Posted
Posted

hi Protx:

Hello, I have just started learning the compilation of Armbian. I am also using the OrangePI3b with the RK3566. I need to study the startup of rknpu in version 6.1.43. I downloaded the latest repository from https://github.com/armbian/build, specifically the main branch. I compiled it according to your compilation configuration. There was an error message upon booting. 

** Booting bootflow 'mmc@fe2b0000.bootdev.part_1' with script
Boot script loaded from mmc 1:1
155 bytes read in 3 ms (49.8 KiB/s)
8598033 bytes read in 724 ms (11.3 MiB/s)
37388800 bytes read in 3133 ms (11.4 MiB/s)
167999 bytes read in 47 ms (3.4 MiB/s)
Working FDT set to a100000
Trying kaslrseed command... Info: Unknown command can be safely ignored since kaslrseed does not apply to all boards.
Unknown command 'kaslrseed' - try 'help'
Moving Image from 0x2080000 to 0x2200000, end=4660000
## Loading init Ramdisk from Legacy Image at 0a200000 ...
   Image Name:   uInitrd
   Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
   Data Size:    8597969 Bytes = 8.2 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 0a100000
   Booting using the fdt blob at 0xa100000
Working FDT set to a100000
   Loading Ramdisk to ec68e000, end ecec11d1 ... OK
ERROR: reserving fdt memory region failed (addr=0 size=0 flags=0)
ERROR: reserving fdt memory region failed (addr=0 size=0 flags=0)
   Loading Device Tree to 00000000ec5fc000, end 00000000ec68dfff ... OK
Working FDT set to ec5fc000

Starting kernel ...

[   14.607208] Internal error: Oops: 0000000096000044 [#1] SMP
[   14.607281] pwm-backlight backlight: Looking up power-supply from device tree
[   14.607780] Modules linked in:
[   14.607848] pwm-backlight backlight: Looking up power-supply property in node /backlight failed
[   14.608469]  sprdbt_tty(+)
[   14.608536] pwm-backlight backlight: supply power not found, using dummy regulator
[   14.608569]  fuse dm_mod ip_tables ipv6 panel_simple pwm_bl
[   14.611186] CPU: 1 PID: 312 Comm: systemd-modules Tainted: G        W          6.1.43-vendor-rk35xx #1
[   14.612092] Hardware name: Rockchip RK3566 OPi 3B (DT)
[   14.612601] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   14.613282] pc : tty_unregister_driver+0x44/0x78
[   14.613752] lr : tty_unregister_driver+0x40/0x78
[   14.614212] sp : ffff80000d02b7c0
[   14.614544] x29: ffff80000d02b7c0 x28: ffff80000a10f0c8 x27: 0000000000000000
[   14.615242] x26: 0000000000000000 x25: ffff80000a23f000 x24: ffff0000092f8400
[   14.615937] x23: ffff80000122d000 x22: ffff80000122e178 x21: 00000000ffffffef
[   14.616632] x20: ffff80000a2272e0 x19: ffff000008c7fb00 x18: 0000000000000000
[   14.617328] x17: 7420797274207427 x16: 6e6f64202c545349 x15: 5845452d20687469
[   14.618028] x14: 7720305442797474 x13: 2e79726f74636572 x12: 696420656d617320
[   14.618724] x11: 656874206e692065 x10: 6d616e20656d6173 x9 : ffff80000915ee50
[   14.619420] x8 : 696166206c616e72 x7 : 65746e695f646461 x6 : 0000000000000000
[   14.620115] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[   14.624209] x2 : ffff000001731c80 x1 : dead000000000100 x0 : dead000000000122
[   14.628287] Call trace:
[   14.631921]  tty_unregister_driver+0x44/0x78
[   14.635732]  mtty_probe+0x1e8/0x2b0 [sprdbt_tty]
[   14.639572]  platform_probe+0x70/0xc0
[   14.643282]  really_probe+0x1cc/0x390
[   14.646981]  __driver_probe_device+0x140/0x158
[   14.650736]  driver_probe_device+0x48/0xd0
[   14.654452]  __device_attach_driver+0xd8/0x128
[   14.658174]  bus_for_each_drv+0xa0/0xc8
[   14.661870]  __device_attach+0xf8/0x180
[   14.665557]  device_initial_probe+0x1c/0x28
[   14.669266]  bus_probe_device+0x38/0x9c
[   14.672951]  device_add+0x550/0x688
[   14.676590]  platform_device_add+0xec/0x224
[   14.680286]  mtty_pdev_init+0x50/0x1000 [sprdbt_tty]
[   14.684061]  do_one_initcall+0x94/0x1e4
[   14.687709]  do_init_module+0x58/0x1e0
[   14.691329]  load_module+0x1818/0x18e0
[   14.694906]  __do_sys_finit_module+0xf8/0x118
[   14.698524]  __arm64_sys_finit_module+0x24/0x30
[   14.702132]  invoke_syscall+0x8c/0x128
[   14.705606]  el0_svc_common.constprop.0+0xd8/0x128
[   14.709161]  do_el0_svc+0xac/0xbc
[   14.712551]  el0_svc+0x2c/0x54
[   14.715888]  el0t_64_sync_handler+0xac/0x13c
[   14.719319]  el0t_64_sync+0x19c/0x1a0
[   14.722684] 
[   14.722684] PC: 0xffff80000875f79c:
[   14.723515] rk_pcie_establish_link: 132 callbacks suppressed


I found that the latest repository does not match your operating instructions. For example, the commit 63073b4af636146d26a7f0f258610eed060c8f34 in the Kwiboog's uboot repository branch rk3568-2023.10 is no longer found. However, the compilation does not report any errors. Could you please share your compilation repository and configuration?
Thank you.

@Protx

kernel.log

0
Link to comment
Share on other sites
Protx

Protx

Posted
  • Author
Posted (edited)

@darcyg

 

First let me say that I am no expert on this. I have not tried to compile it lately, but I see that that commit does not longer exist on kwiboo's github. So I don't now if you need to revert to an older version of u-boot. And if you have an EmmC og SPI with u-boot, then it should work to boot from SC card with the new uboot anyway.

 

My log was with debug enabled (and from uart), and I dont have that log anymore. 

mtty_probe+0x1e8/0x2b0 [sprdbt_tty]

So I am not sure the error is the same wireless / bluetooth related error that I got . But I think it is.

Make sure that the patch uw5622-fix-adjust-for-rockchip-post-6.1.patch got applied.

 

Take a look in the /output/logs, If you have build the kernel several times it could be using a cached kernel and not included in the log!?, (look in the largest log files). And verify that uw5622-fix-adjust-for-rockchip-post-6.1.patch  was included in your compilation: 

--> (15) INFO: Adding [ Drivers for Unisoc uwe5622 found on some Allwinner and Rockchip boards ]
--> (15) INFO: * applying patch/misc/wireless-uwe5622/uwe5622-allwinner.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/uwe5622-allwinner-bugfix.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/uwe5622-warnings.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/uwe5622-park-link-v6.1-post.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/uwe5622-v6.1.patch
--> (15) INFO: *  patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-post-6.1.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/wireless-uwe5622-Fix-compilation-with-6.7-kernel.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/wireless-uwe5622-reduce-system-load.patch
--> (15) INFO: * applying patch/misc/wireless-uwe5622/w-wireless-fix-out-of-tree-build.patch
--> (15) INFO: Preparing driver [ driver_rtl8723cs ]

 

Edited by Protx
0
Link to comment
Share on other sites
Protx

Protx

Posted
  • Author
Posted (edited)

@darcyg

 

What i meant was that you manually apply the uw5622-fix-adjust-for-rockchip-post-6.1.patch before ./compile command.

Then after compilation check that : 

--> (15) INFO: *  patch/misc/wireless-uwe5622/uwe5622-adjust-for-rockchip-post-6.1.patch

is included in the log file.

 

BTW: Do not mind the 

* applying patch/misc/wireless-uwe5622/w-wireless-fix-out-of-tree-build.patch

That was a patch I saw was included in the ubuntu rockchip kernel. I do not think that is needed, does not seem to make a difference.

Edited by Protx
0
Link to comment
Share on other sites
darcyg

darcyg

Posted
Posted

@Protx 
Thank you very much. I am currently compiling other systems, and I will test your solution later. Additionally, I have found that the latest official trunk403 build, as well as the previous trunk250, both exhibit this error. I am testing using an SD card.

0
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept