Kernel - Internal Error Oops: 0000000096000004 [#1] PREEMPT SMP

[Abdul Badalambadad]

Hello,

 

I've problem...

 

Message from syslogd@rock-5b at Feb 23 21:58:00 ...
 kernel:[  443.674152] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP

Message from syslogd@rock-5b at Feb 23 21:58:00 ...
 kernel:[  443.674152] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP

Message from syslogd@rock-5b at Feb 23 21:58:00 ...
 kernel:[  443.718480] Code: b40001b3 f1036273 d503201f 54000140 (f9402261)

Message from syslogd@rock-5b at Feb 23 21:58:00 ...
 kernel:[  443.718480] Code: b40001b3 f1036273 d503201f 54000140 (f9402261)

 

I have exactly zero experience with debugging this. But from my trial-error it will have something to do with NVME as this is the only thing that remains. This error happen after copying data to SSD in M.2 slot. I tried copying USB -> SSD and SFTP -> SSD . Happens everytime after about 5 GB is copied.

 

Configuration:

* Rock Pi 5B, 32 GB RAM

* SSD: Samsung 990 PRO 2TB
* Power Supply: Dummy 12V DC adapter 3A

* Up-to-date Armbian with 6.8.0-rc1-edge-rockchip-rk3588

* Latest bootloader in NAND

* Boot from NVME

 

The only NOT up-to-date thing is the Samsung firmware. I haven't found a way how to update it inside the Pi.

 

What can I do? Please help.

Edited February 23 by Abdul Badalambadad

[Werner]

Providing logs with 

armbianmonitor -u

 helps with troubleshooting and significantly raises chances that issue gets addressed.

[foxx1337]

It's the memory. You'll notice that the machine with the Armbian kernels can't access more than 8-9 gigabytes. I have the same issue. The Ubuntu 5.10 kernels published by Joshua Riek work fine, not sure about Radxa's 6.1 yet.

 

Type this into a file called test.cpp:

#include <iostream>
#include <vector>

int main()
{
    std::vector<unsigned char> test(29ull * 1024 * 1024 * 1024);
    std::cin.get();
    return 0;
}

 

then launch htop in a session and type in a different session:

g++ test.cpp
./a.out

 

and watch the kernel die after 2-3 seconds when the vector gets to zeroing past the 8-9 gigabyte mark.

 

3 posts under this one -

 

Edited February 24 by foxx1337

[Abdul Badalambadad]

Thank you guys, but the SSD is probably dead. I've tried Arch based on 5.10 kernel and after I've started the copy process it crashed as well but this time the SSD doesn't appear on any other PC. So I am going for a service with it.

 

So this might actually be a HW failure. Good thing is that the Pi survived (crossed fingers) and that I won't have to address refund with China.

[Igor]

8 hours ago, foxx1337 said:

You'll notice that the machine with the Armbian kernels can't access more than 8-9 gigabytes.

 

Using this snippet. No issues whatsoever, latest Gnome image from the download section.

 

[foxx1337]

std::vector also zeroes it, also it's the 32 GB Rock 5Bs I think.

[Igor]

4 minutes ago, foxx1337 said:

also it's the 32 GB Rock 5Bs I think.

 

OK. I don't have any 32Gb rk3588 device here, 16Gb works fine ... Did you perhaps tried with most recent 24.2 images? Also I am running GitHub runners on two 16Gb Rock5 with 6.x kernel ... where memory is abused dramatically. And it works. Perhaps related to u-boot or ddr blobs?

[foxx1337]

I think it's the DTBs from Armbian. Joshua Riek's 5.10 works fine.

[Werner]

Tried doing a comparison?

[Igor]

10 hours ago, foxx1337 said:

I think it's the DTBs from Armbian. Joshua Riek's 5.10 works fine.

I tested his image and it works the same as Armbian. As expected.

[Abdul Badalambadad]

I've got another update: the SSD isn't dead. I was just tired and didn't notice that the USB cable fell from USB-A to USB-C adapter. That was the reason it didn't appear on another PC.

 

Then I decided to use the new SSD in my main machine and use my old SSD in Rock Pi. Didn't help. Now I've got two different SSDs which behave the same. This rules out a hardware fault on SSD side.

 

I've tested different filesystems (ext4 and exfat) to rule out a bug in the filesystem implementation. Both behave the same except ext4 has much more consistent performance.

 

Then I installed memtester. And boom. The fault is not caused by NVMe, but rather faulty memory as @foxx1337 suggested. This happens all the time even in big PC world.

 

So I guess that I just wasn't lucky and my Pi has faulty memory. Does this make sense?

[Abdul Badalambadad]

On 2/24/2024 at 1:47 PM, foxx1337 said:

It's the memory. You'll notice that the machine with the Armbian kernels can't access more than 8-9 gigabytes. I have the same issue. The Ubuntu 5.10 kernels published by Joshua Riek work fine, not sure about Radxa's 6.1 yet.

 

Type this into a file called test.cpp:

#include <iostream>
#include <vector>

int main()
{
    std::vector<unsigned char> test(29ull * 1024 * 1024 * 1024);
    std::cin.get();
    return 0;
}

 

then launch htop in a session and type in a different session:

g++ test.cpp
./a.out

 

and watch the kernel die after 2-3 seconds when the vector gets to zeroing past the 8-9 gigabyte mark.

 

I confirm this. But I still think it is not a kernel problem as all (radaxa, joshua, armbian 6.8) behave exactly the same. On my Pi the mark is about 11 GB. I already raised a complaint at Radaxa so wish me luck.

 

Thank you guys, I won't bug you here any more.

Edited February 25 by Abdul Badalambadad

[foxx1337]

My bet is that it's a DTB issue - so software.

[Abdul Badalambadad]

@foxx1337 If you are interested somebody on Radxa forum has the same opinion. I am not experienced enough with DTOs so maybe he can help you as well https://forum.radxa.com/t/samsung-990-pro-2tb-ssd/20166/6?u=bamboo

Edited February 26 by Abdul Badalambadad

[foxx1337]

Thanks, I'm watching, bamboo.