OPI5Plus - 32GB version issue

[Charlie Romer]

 

2 minutes ago, Igor said:

We are on the same side.

This is what I've been trying to say the whole time... I know many distros are small projects with no full time developers. I saw in this thread that it looked like you guys could use some help and given how new this board is I wasn't expecting anyone to just solve something for me. I wanted to jump in and help if and where I could. 

What did you expect? Truth or fake happiness and prompt resolving of your problem?

The opposite literally. I figured that if I have the ability to help and hardware in hand that I should help (or at least offer to be a guinea pig) - I wasn't and haven't been looking for someone to solve this thing for me but literally offering my assistance as I happen to have the hardware and wanted to give back if I could. This whole conversation has felt like I've been gaslit by people who assume everyone posting for the first time on this forum must be demanding fixes for everything right now. That may be many (or even most) folks but it's not a great feeling when all you're doing is trying to offer help.

[going]

@Charlie Romer Hi. Let's try to figure it out.

 

5 часов назад, Charlie Romer сказал:

I figured that if I have the ability to help and hardware in hand that I should help

Core panic - how is this problem solved?
It is assumed that you have the kernel source code.

You look at the kernel messages, and then using utilities like grep you find the function in the source code.

The very function that could not cope and caused panic. This appears in the kernel panic messages.

 

The first probable reason that led to this is that a patch was incorrectly applied to a file with this function.

I.e. we are looking for all patches that modified this source code file.
In fact, we will need to apply all the patches in the series to the source code in the core git repository.

Apply as the "git am" command.
We pay special attention to those patches that change our target file.
Then we extract a series of our patches from the repository using a script and place it in the target patch folder.
I.e., we replace the existing series with a new one, which is applied without offsets and fuzziness.

 

The second reason is the changes in the upstream of the kernel that led to the situation our patch has aged.
In this case, we are fixing this patch.
We simply make our new changes to the fixation after applying the patch in the process of applying the series step by step.

 

The complexity of the core panic situation in the Armbian project lies in the fact that it is impossible to reproduce this particular problem.
Therefore, if you did everything as I wrote and made the necessary corrections from your point of view,

then assembled the kernel and the panic repeated, you simply push your changes to your repository on github

and only after that you can write here on the forum or ask a question here by specifying a link to your repository

and the kernel version to which corrections have been applied.
Only after that, the maintainer or the interested user will be able to check it.

 

The third possible option is that the u-boot code does not match the kernel code due to their development.

We may have to monitor this loader build process as well.

 

@Igor Did I miss something? Where did the fixed kernel version go in these settings?

How is the correct application of patches controlled today?

 

 

 

[Igor]

14 minutes ago, going said:

How is the correct application of patches controlled today?

Hasn't changed. 
https://github.com/armbian/build/tree/main/patch/kernel

Just symlinking of subfolder are perhaps adding some confusion.

[going]

1 минуту назад, Igor сказал:

Hasn't changed. 
https://github.com/armbian/build/tree/main/patch/kernel

Just symlinking of subfolder are perhaps adding some confusion.

 

I meant this line:

KERNELBRANCH="tag:v6.1.69"

which uniquely determines which version of the kernel patches are applied cleanly.
I.e. to version 6.1.69 and not to the current 6.1.90 for today.

 

 

[Igor]

1 minute ago, going said:

which uniquely determines which version of the kernel patches are applied cleanly.
I.e. to version 6.1.69 and not to the current 6.1.90 for today.

Oh, yes. This must also work. or "commit:xxxxxxxxxxxxxxxxx"

[rosbeef]

Hi ! i am in this situation: Orangepi 5plus 32gb ram , uboot on spi and root on nvme

i experiment the crash on Armbian 24.5.1 Bookworm with Linux 5.10.160-legacy-rk35xx and 6.1.43-vendor-rk35xx and 6.8.10-edge-rk3588 too

I would help you to resolve my bug but could you help me to send you all you could need.

 

i recover some datas from the serial

 

  409.837122] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[  409.837821] CPU: 3 PID: 1 Comm: systemd Not tainted 6.8.11-edge-rockchip-rk3588 #1
[  409.838494] Hardware name: Xunlong Orange Pi 5 Plus (DT)
[  409.838967] Call trace:
[  409.839191]  dump_backtrace+0x94/0x114
[  409.839538]  show_stack+0x18/0x24
[  409.839840]  dump_stack_lvl+0x38/0xf0
[  409.840173]  dump_stack+0x18/0x24
[  409.840473]  panic+0x3b8/0x3f8
[  409.840754]  do_exit+0x898/0x9bc
[  409.841050]  do_group_exit+0x34/0x90
[  409.841376]  copy_siginfo_to_user+0x0/0xc8
[  409.841746]  do_notify_resume+0x27c/0x1430
[  409.842114]  el0_da+0x8c/0x90
[  409.842387]  el0t_64_sync_handler+0xe4/0x158
[  409.842772]  el0t_64_sync+0x1a4/0x1a8
[  409.843104] SMP: stopping secondary CPUs
[  409.843559] Kernel Offset: disabled
[  409.843871] CPU features: 0x0,c0000000,7002814a,2100720b
[  409.844345] Memory Limit: none
[  409.844629] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

 

other dump

 

  279.842385] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
[  279.842949] Modules linked in: macvlan nft_masq bridge snd_seq_dummy rfcomm snd_seq snd_seq_device nft_fib_inet nft_fib_ipv4 nft_fib_ipvb
[  279.843297]  libarc4 gpu_sched drm_shmem_helper drm_exec rfkill_gpio rfkill dm_mod ip_tables x_tables autofs4 uas rockchipdrm drm_dma_hes
[  279.853805] CPU: 2 PID: 190 Comm: kworker/2:1 Not tainted 6.8.11-edge-rockchip-rk3588 #1
[  279.854522] Hardware name: Xunlong Orange Pi 5 Plus (DT)
[  279.854995] Workqueue: events kernfs_notify_workfn
[  279.855439] pstate: a0400009 (NzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  279.856057] pc : find_inode_fast+0x38/0xe4
[  279.856430] lr : ilookup+0x8c/0x118
[  279.856747] sp : ffff80008353bca0
[  279.857044] x29: ffff80008353bca0 x28: 0000000000000000 x27: ffff800081f57980
[  279.857686] x26: ffff800081d5e000 x25: ffff00010048ebb0 x24: ffff800081172078
[  279.858327] x23: ffff0001008a5800 x22: ffff0007d9300c00 x21: ffff0001008a5800
[  279.858968] x20: 00000000000027c8 x19: 90d48fefa3653def x18: ffff80008c523c58
[  279.859609] x17: 0000000000000000 x16: 1fffe0002046dc41 x15: 00000000000002e9
[  279.860249] x14: 0000000000000000 x13: 0000000000000030 x12: 0101010101010101
[  279.860889] x11: 7f7f7f7f7f7f7f7f x10: fefefefefefefeff x9 : 7f7f7f7f7f7f7f7f
[  279.861530] x8 : 8080808080808080 x7 : ffff00010048ebb0 x6 : ffff000100aa7380
[  279.862170] x5 : d83827dd7f7cc000 x4 : 0000000000000015 x3 : d9bf05c4657e16ae
[  279.862810] x2 : 00000000000027c8 x1 : ffff0007d9300c00 x0 : ffff0001008a5800
[  279.863451] Call trace:
[  279.863675]  find_inode_fast+0x38/0xe4
[  279.864016]  ilookup+0x8c/0x118
[  279.864303]  kernfs_notify_workfn+0x12c/0x220
[  279.864698]  process_one_work+0x160/0x3a8
[  279.865064]  worker_thread+0x32c/0x438
[  279.865404]  kthread+0x118/0x11c
[  279.865701]  ret_from_fork+0x10/0x20
[  279.866030] Code: aa0003f5 f9001bf7 d503201f 54000140 (f9402263) 
[  279.866441] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[  279.866448] SMP: stopping secondary CPUs
[  279.866573] ---[ end trace 0000000000000000 ]---
[  279.866610] Kernel Offset: disabled
[  279.868331] CPU features: 0x0,c0000000,7002814a,2100720b
[  279.868799] Memory Limit: none
[  279.869076] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

 

Edited May 30 by rosbeef
more spec

[Pierre C.]

Hi all,

 

Well, I've been having some issue with my OPi5+ and I managed to solve most of them so far.

 

As to the issue at hand, I've been running with Armbian 24.2.1 Bookworm with Linux 5.10.160-legacy-rk35xx for a while on my 16GB no worries.

As for the 32GB, I can't say for Armbian as I haven't tested. But with J-R's, my booting issues and memory issues went away when I replaced my PSU.

 

It seems the 32GB needs a little more power than the 16GB, I replaced it with a proper 5v 6A PSU and all my worries went away, it boots and hasn't crashed yet !

It might work with an original 5v 5A, but mine was from Ali and might be over-rated as most is on this site ...

 

Hope this might help someone...

 

All my OPi5+ (16+32) are installed with a Force MP600 2tB Nvme

Edited May 30 by Pierre C.

[rosbeef]

Damn ! the opi5+ bundle from aliexpress includes a 5V-4A which should be short for a 32G+nvme.

I will try a 5V-5A as i am not able to find a 5V-6A on amazon.

Edited May 30 by rosbeef

[Werner]

I use an mean well HRP-200-5...but for all my stuff

 

https://media.it-tronics.de/Datasheets/Power_Supplies/MeanWell/HRP-200.pdf

[rosbeef]

More current (5A instead of 4A) did not resolve the problem for me.

I  tried the vendor distro (debian) and problem is not solved.

Writing from SD to emmc makes crash too. even if no usb connected.

A simple copy command make debian crash. i will go to orangepi forum 😕 for help.

Edited Sunday at 12:07 PM by rosbeef

[Werner]

Voltage might be an issue. Most SBCs while rated for 5 volts prefer slight overvoltage, last but not least to compensate for voltage loss across wiring, connectors and the PCB itself. Even PSU sold by xunlong in combination with the OPi5, also while rated for 5v actually outputs about 5.2V.

 

I verified this on an OPi5 month ago by measuring at the voltage source which was between 5.2 and 5.3 volts and the USB-A connector on the board which dropped to 5.0 volts when putting on a heavy CPU load. Since the PSU itself has way more than enough power voltage there were constant.

Now think about what happens when your PSU only outputs barely enough wattage. Not only the voltage drop mentioned earlier will be there but also the voltage will break down at PSU level which increases the overall voltage drop to a (from a electronics perspective) crazy level.

 

So if your input voltage is 5 volts only it will drop below this and malfunctioning is VERY likely then.

 

I'd test this on the 5+ 32G model if I had one. But mine has 16G memory only.

[usual user]

1 hour ago, Werner said:

Voltage might be an issue.

The issue is probably due to the fact that rk35xx SoCs have to carve out some memory areas above 16 GB as reserved. Access to those areas leads to crashes.
Because no fully open source TF-A is available yet, current mainline U-Boot has mechanisms landed to take the area information from the closed source TPL.
Chances are the firmware you're using hasn't backported this yet, because devices larger than 16GB are only now becoming widely available in the open market.

[rosbeef]

1 hour ago, usual user said:

Chances are the firmware you're using hasn't backported this yet, because devices larger than 16GB are only now becoming widely available in the open market.

i don't need 32 for now, i jut bought it for the future.

Should it be possible to lie on the size of the memory and only declare 16Gb usable ?

 

[rosbeef]

3 hours ago, Werner said:

So if your input voltage is 5 volts only it will drop below this and malfunctioning is VERY likely then.

i don't have the tools to test the voltages and don't have a precise tunable powersource.

as you seems to know about electronic can i ask you ? Is 6V will be too much overvoltage ?

[usual user]

30 minutes ago, rosbeef said:

Should it be possible to lie on the size of the memory and only declare 16Gb usable ?

Setup mem=16G at kernel cmdline.

[Werner]

1 hour ago, rosbeef said:

Is 6V will be too much overvoltage ?

yes, it is. 5.2V to .3 seems to be a reasonable value.

[rosbeef]

Hum i saw that my power is 5.1v

I removed the mvne and try to cp a big file, it crash too.

Maybe the right way to explore is the bad memory managment.

 

20 hours ago, usual user said:

Setup mem=16G

I could not reduce visible memory, sorry i'm noob here.

I put the mem=12G in the /boot/armbianEnv.txt file then "update-initramfs -u"

I tried directly in /boot/boot.cmd on the bootagrs parameter too without success.

, i read that too :

Quote

mem= is for manual override or prehistoric bootloaders not supporting
atags or device tree.

source https://github.com/linux-sunxi/u-boot-sunxi/issues/11#issuecomment-6358135

 

maybe there is an other way to shrink the memory or accept the 32G.

Edited Monday at 09:12 AM by rosbeef

[Efe Çetin]

@rosbeef have you ever tried to update DDR and BL31 blobs? 5plus still uses old blobs that may cause the issue. You can update blobs by putting 

```

DDR_BLOB='rk35/rk3588_ddr_lp4_2112MHz_lp5_2400MHz_v1.16.bin'

BL31_BLOB='rk35/rk3588_bl31_v1.45.elf

```

to the https://github.com/armbian/build/blob/main/config/boards/orangepi5-plus.conf and compile the new uboot spi image/package or new image.

[MichaIng]

Am 3.6.2024 um 10:13 schrieb rosbeef:

I put the mem=12G in the /boot/armbianEnv.txt file then "update-initramfs -u"

I tried directly in /boot/boot.cmd on the bootagrs parameter too without success.

 

Either, you need to add it to the "extraargs=" (optionally "extraboardargs=") line in /boot/armbianEnv.txt, in case create those lines, if missing:

 

extraargs=mem=16G

 

Or, when adding it to /boot/boot.cmd directly, you need to generate /boot/boot.scr from it:

 

mkimage -C none -A arm64 -T script -d /boot/boot.cmd /boot/boot.scr

The initramfs does not need to be regenerated in any case.

Edited Tuesday at 12:43 PM by MichaIng

[rosbeef]

3 hours ago, MichaIng said:

extraargs=mem=12G line in /boot/armbianEnv.txt,

ok thankyou, that makes debian boot with 12G effectively and big file (250GB tested) coping is fully working.

So it discard a hardware power failure. Sorry @Werner

 

Now i have a half working memory. And if i want to get my fully working 32G memory i have to try the @Efe Çetin solution.

On 6/3/2024 at 1:11 PM, Efe Çetin said:

DDR_BLOB='rk35/rk3588_ddr_lp4_2112MHz_lp5_2400MHz_v1.16.bin'

BL31_BLOB='rk35/rk3588_bl31_v1.45.elf

 

just let me time to compile the new armbian with those blobs

Thanks all.

[Eddie]

On 6/2/2024 at 3:44 PM, usual user said:

mem=16G

 

Crashed.

[rosbeef]

@Eddie as @MichaIng explain me did you add the next in the armbienEnv.txt ?

 

23 hours ago, MichaIng said:

extraargs=mem=16G

 

[Eddie]

3 minutes ago, rosbeef said:

explain me did you add the next in the armbienEnv.txt ?

 

Yes. Added it to other arguments, like this:

extraargs=fsck.mode=force fsck.repair=yes mem=16G

 

free -h shows 16G after this.

[MichaIng]

Since a board with 16G RAM practically has a little less than 16 GiB RAM, probably mem=16G limits it a little too less to prevent the crash 100% reliably, while mem=15G would do. I don't know how exactly those limits are applied, and at which point exactly the power usage of the RAM raises above the critical point (in case really this is the reason for the crashes), so this is just a guess which can be tested.

Edited 16 hours ago by MichaIng

[rosbeef]

@usual user@Efe Çetin

it seem to work with

Quote

 

DDR_BLOB='rk35/rk3588_ddr_lp4_2112MHz_lp5_2400MHz_v1.16.bin'

BL31_BLOB='rk35/rk3588_bl31_v1.45.elf

 

 

i burned an sd card with the new compiled Armbian-unofficial_24.8.0-trunk_Orangepi5-plus_bookworm_legacy.5.10.160 image .

I boot from the sdcard

It seems to work as i have no crash copying a 250GB file between a usb ssd and the mvne.

 

I'm trying now memtester 31G. ...

  ... trying mlock ...Killed

BUT not crash

 

trying memtester 30G is working ...

... and it does not crash.

 

@Efe Çetin Is it possible to make the official release working for the 32Gb version ?

Edited 15 hours ago by rosbeef