Possible bootloader bug for NVME boot?

[ScottN]

Hello, I have roughly 30 Orange Pi 5 here in the office setting up for deploying in the field.

 

NVMe boot is working great after using 'nand-sata-install' to flash the MTD.

 

But sometimes (rarely, but happens), when issuing a reboot of Armbian, after full kernel shutdown, the player does not boot back up.  Power light is on, no activity blinking LED.  Network link and activity blinking.  Just won't boot.  Requires physically disconnecting power for a few seconds then re-plug to get to boot.

 

I have watchdog set in Armbian and tested that works.  So this seems to be before Linux kernel has taken control.

 

Is there any way to enable any logging or debug mode on the bootloader to try to get some logs or anything to help see what's happening?

 

Thanks.

Edited June 2, 2023 by ScottN
Added a sentence

[Werner]

1 hour ago, ScottN said:

Is there any way to enable any logging or debug mode on the bootloader to try to get some logs or anything to help see what's happening?

Yes. Hook up a serial console and set verbosity to 7 in /boot/armbianEnv.txt

[ScottN]

OK, I will do this and report back with the console output.  Thanks.

[ScottN]

Just to clarify what I need to do on my end.  Do I need to connect serial cable to an external machine with serial port, open Putty and connect the other end to the OPi serial connections?  Or do I need to plug a USB to serial adapter into the Pi, then connect the other to my external machine.  Any documentation or more detailed steps for this?  Thanks!

[Werner]

Instructions are similar to this: https://www.instructables.com/Setup-Orange-Pi-Using-Serial-Port/

Use baudrate 1500000 instead of 115200.

[ArmBoy1988]

This one has some good info too:

 

https://wiki.radxa.com/Rockpi4/dev/serial-console

 

Has info for connecting using Linux, Windows and MacOS.

[ScottN]

Thank you for the instructions.

 

I had to buy a USB to serial/TTL cable and now have it hooked up.  But I have something weird, getting gibberish in the putty output.  I've logged a full reboot to a log file and attached it.

 

My putty config:

2023-06-05 11:49:14    Configuring baud rate 1500000
2023-06-05 11:49:14    Configuring 8 data bits
2023-06-05 11:49:14    Configuring 1 data bits
2023-06-05 11:49:14    Configuring no parity
2023-06-05 11:49:14    Configuring no flow control

 

What do I have wrong here?

putty.log

Edited June 5, 2023 by ScottN

[Werner]

The adapter you bought supports this rather high baud rate, right?

Also double-check your connection and make sure ground (GND) is connected as well.

[ScottN]

Hmmm, I'm not sure actually.  Didn't know to look for that honestly.  This is what I bought: https://www.amazon.com/dp/B07R8BQYW1

 

Do you have a recommendation for a different one?

 

On a side note, I have found a workaround for this reboot issue.

 

Firstly, you need watchdog enabled.

Then you issue a halt command: shutdown -H 0

What happens, I believe, Linux shuts down normally, but isn't a full shutdown and power off.  Then, the watchdog kicks shortly after and reboots the board.

 

If I issue a reboot with 'reboot now', the bootloader bug appears rather frequently.  Sometimes it'll boot back up, sometimes not.  I'm seeing well over 50% of the times it's not.  Hopefully I can get this serial console output with verbosity and give more information.  I'm just happy I have a temporary workaround.

[ArmBoy1988]

I think the TX of the cable needs to plug into the RX on the board and then the RX on the cable plugs into the TX on the board.

 

PS:  the listing for the cable you purchased shows it supports up to 3M baud rate so it should handle 1500000.

Edited June 6, 2023 by ArmBoy1988
add PS

[ScottN]

11 hours ago, ArmBoy1988 said:

I think the TX of the cable needs to plug into the RX on the board and then the RX on the cable plugs into the TX on the board.

 

I did make this mistake at first, didn't get anything coming through Putty and the connection, so I swapped them and that's when this garbage started.  I'll play around with it.  I do have the connections on the 3 pin header for UART, which is correct, right?  Looking at the GPIO there are other UART pins there too.  The Orange Pi 5 manual covers them.

 

So, not sure what's happening here.  This isn't some text encoding thing is it?

Edited June 6, 2023 by ScottN
More info and link

[ArmBoy1988]

Not sure.  Maybe try other software.  In that link I posted it shows how to use mincom or minicom.  I'll be doing this some time in the near future so I hope you figure it out 😀  When I try it I'll be able to provide my experiences.

[Werner]

On Windows PuTTY Or KiTTY working nicely, on Linux I simply use screen

[ScottN]

Little update.  I connected the USB to TTL to my Ubuntu laptop and used tio.  Same result as Putty.

 

sudo tio /dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_A\:AZt114J20-if00-port0 -b 1500000 -d 8 -f none -p none -s 1

[ScottN]

I found the other thread trying to figure out the UART serial communication

 

I will pick up this one that is recommended.

 

Question is, what's the correct way to do this?  The thread I linked mentions 3.3v?  Any more clarification on how to do this?

[Werner]

19 minutes ago, ScottN said:

d 8 -f none -p none -s 1

Hm I never had to set any of that stuff, it just worked.

Could you try like sudo screen /dev/ttyUSB0 1500000 ?

[ScottN]

It connects, but when I hit ENTER on the keyboard, I get:

 

PAcbwo`@8

 

When the player reboots, I get the same garbled output as previous.  Below is a brief snip on when the OPi5 starts to shutdown.

 

`@@@HAD@@$    ,@@h@`@p0@@``p`@q`HhhNCBAR 0!  $0  `nBKB@@`)@@```@@@HaD@@$    ,`@L@`@p0@@`pp`@p`HHHoBC@ 0 , 6"60 X`OBBB@@@@```@p(@@H@DA@$    ,`@m@`@p0@@``p`@qh@n@HnCC@ Z 2         D[ ;" H H`n``A@ 2 2#0I@`HA`b@`C@sr`onb  2" H HX`n``@@`@`G@` 2 :"0H@`fO`B H     [[ :" h H`N``A@`@`F@` 2 2#0HbE`@[Z) 2ALFR!D  H  2uL D@bb` D[>  h "R `n``@ :R :!2"!D 0I"12) DZ?  @[ 07701100b@@No`8pb`@l`@H`E@h@`wo`C@HAP@sOHq`ANL H`N``ALF1220Ibu``HAB@L` H,

 

And just more similar to that as it boots back up.

[-]

i experienced this too with nvme tho. it used to reboot fine but after some update, it just locks up on reboot with red led on.  have to remember to shutdown and manually press the button for a reboot. it might have to do with a kernel update since the official debian uses an older kernel and it doesnt have this behavior on reboot.

[ScottN]

10 hours ago, - said:

have to remember to shutdown and manually press the button for a reboot

 

That's if you have access to the board

 

We're shipping these players out in the field for use and need to reliably reboot.  As I mentioned above in a reply, having the watchdog enabled and doing a 'shutdown -H 0' halt command, the watchdog will kick and reboot the board.  A little janky for sure, but a workaround.  I'd really like to find why this is happening and then patch the boards in the field remotely.

[ScottN]

An update on the UART serial connection.  I bought this one and used the command 'sudo tio /dev/ttyUSB0 -b 1500000 -d 8 -f none -p none -s 1' and it is working beautifully now and prompts for login and on reboots get full verbose output.  So I will be able to provide some more info shortly on when the reboot hangs.

[ArmBoy1988]

It looks like that one uses an FTDI FT232RL chip.  I think that one is faster than the pl2303 chips.  Perhaps the driver for the ft232 is better or more recent than that for the pl2303 chips or they are knockoff chips that don't work properly.

 

Thanks for letting us know.  I'll have to pick one of those up too.

[ScottN]

I've been having a heck of a time trying to catch this issue with the devices I have as I'm setting them up.  If I'm preparing a batch of them, say of 10, 2 of them will act up then not agaom.  Of course I cannot have a TTY connection open to all of them at once so it's by sheer luck that I can have the serial connected and it'll happen.  I haven't been able to be that lucky yet so I don't know.

 

But what I have now is one that hung on boot, I left it there and it's powered on, so I connected the serial connection and when I hit enter, I get a "opi#" prompt.  Is there any way to print any log files from this that could tell me anything?  Or anything I can do to help diagnose why it did not start the Linux kernel?

 

EDIT:

 

So help shows some things I can do.  Listed below.  But if I do 'nvme device' I get 'no nvme devices available'.  Which could be why it won't boot to an nvme

 

?       - alias for 'help'
android_print_hdr- print android image header
atags   - Dump all atags
base    - print or set address offset
bdinfo  - print Board Info structure
bidram_dump- Dump bidram layout
blk     - Block device sub-system
boot    - boot default, i.e., run 'bootcmd'
boot_android- Execute the Android Bootloader flow.
boot_fit- Boot FIT Image from memory or boot/recovery partition
bootavb - Execute the Android avb a/b boot flow.
bootd   - boot default, i.e., run 'bootcmd'
booti   - boot arm64 Linux Image image from memory
bootm   - boot application image from memory
bootp   - boot image via network using BOOTP/TFTP protocol
bootz   - boot Linux zImage image from memory
charge  - Charge display
charge_pd- Charge select
cmp     - memory compare
coninfo - print console devices and information
cp      - memory copy
crc32   - checksum calculation
crypto_sum- crypto checksum engine
dhcp    - boot image via network using DHCP/TFTP protocol
dm      - Driver model low level access
download- enter rockusb/bootrom download mode
dtimg   - manipulate dtb/dtbo Android image
dump_irqs- Dump IRQs
dump_resource- dump resource list
echo    - echo args to console
editenv - edit environment variable
env     - environment handling commands
exit    - exit script
ext2load- load binary file from a Ext2 filesystem
ext2ls  - list files in a directory (default /)
ext4load- load binary file from a Ext4 filesystem
ext4ls  - list files in a directory (default /)
ext4size- determine a file's size
false   - do nothing, unsuccessfully
fastboot- use USB or UDP Fastboot protocol
fatinfo - print information about filesystem
fatload - load binary file from a dos filesystem
fatls   - list files in a directory (default /)
fatsize - determine a file's size
fatwrite- write file into a dos filesystem
fdt     - flattened device tree utility commands
fstype  - Look up a filesystem type
go      - start application at address 'addr'
gpt     - GUID Partition Table
help    - print command description/usage
iomem   - Show iomem data by device compatible(high priority) or node name
lcdputs - print string on video framebuffer
load    - load binary file from a filesystem
loop    - infinite loop on address range
ls      - list files in a directory (default /)
md      - memory display
mdio    - MDIO utility commands
mii     - MII utility commands
mm      - memory modify (auto-incrementing address)
mmc     - MMC sub system
mmcinfo - display MMC info
mtd_blk - MTD Block device sub-system
mw      - memory write (fill)
nfs     - boot image via network using NFS protocol
nm      - memory modify (constant address)
nvme    - NVM Express sub-system
part    - disk partition related commands
pci     - list and access PCI Configuration Space
ping    - send ICMP ECHO_REQUEST to network host
printenv- print environment variables
pxe     - commands to get and boot from pxe files
rbrom   - Perform RESET of the CPU
reboot  - Perform RESET of the CPU, alias of 'reset'
reset   - Perform RESET of the CPU
rkimgtest- Test if storage media have rockchip image
rockchip_show_bmp- load and display bmp from resource partition
rockchip_show_logo- load and display log from resource partition
rockusb - Use the rockusb Protocol
run     - run commands in an environment variable
save    - save file to a filesystem
saveenv - save environment variables to persistent storage
scsi    - SCSI sub-system
scsiboot- boot from SCSI device
setcurs - set cursor position within screen
setenv  - set environment variables
setexpr - set environment variable as the result of eval expression
sf      - SPI flash sub-system
showvar - print local hushshell variables
size    - determine a file's size
source  - run script from memory
sspi    - SPI utility command
sysboot - command to get and boot from syslinux files
sysmem_dump- Dump sysmem layout
sysmem_search- Search a available sysmem region
test    - minimal test like /bin/sh
tftp    - download image via network using TFTP protocol
tftpbootm- tftpbootm aosp/uImage/FIT image via network using TFTP protocol
tftpflash- flash image via network using TFTP protocol
tftpput - TFTP put command, for uploading files to a server
true    - do nothing, successfully
ums     - Use the UMS [USB Mass Storage]
usb     - USB sub-system
usbboot - boot from USB device
version - print monitor, compiler and linker version
opi# version
U-Boot 2017.09-armbian (Feb 17 2023 - 22:22:49 +0000)

aarch64-linux-gnu-gcc (Linaro GCC 7.4-2019.02) 7.4.1 20181213 [linaro-7.4-2019.02 revision 56ec6f6b99cc167ff0c2f8e1a2eed33b1edc85d4]
GNU ld (Linaro_Binutils-2019.02) 2.28.2.20170706

 

Edited July 17, 2023 by ScottN
help list

[ArmBoy1988]

If the NVME is not showing up, perhaps it's an issue with the NVME?  Do they all use the same NVME?  Does the issue happen with the same boards all the time?  Mostly?

 

Sounds like this is for a product you're going to be selling.  Perhaps try a few of the OPi with different NVME and see if the issue happens on those as well.

Could even be one or several OPi with issues? 

 

Keep good stats and then closely review them at regular intervals.

 

Just throwing out ideas.

[ScottN]

Hi ArmBoy1988, yes we use these as small media players, think of like signage and other things in public spaces.

 

I've been reading up on U-Boot more, which seems to be the boot loader here.  I found another boot loader from a small project here which I downloaded the image, wrote to a microsd, and copied off the 'rkspi_loader.img' file from there, and then used dd to write that to '/dev/mtdblock0'.  That boots the OPi on the NVMe just fine.  I created a '@reboot' crontab that waits a couple minutes and issues a normal reboot.  I'm monitoring the serial output doing this to a file.  What I've noticed (but still testing to confirm) is if I do NOT connect the network cable, this thing reboots great for hours on end, no issues.  So I have to rule out the NVMe at that point I suppose?  I'm about to try to test by connecting the network cable back up and see if after a bit it hangs.

 

Btw, when it did hang with the different boot loader, I was at the u-boot command prompt, I did the NVMe command to list devices and the storage is there.  I then just issued a reset command and it booted just fine.

 

So with that, my suspicion is maybe U-Boot is trying a network boot and failing?

 

I'm starting to now research doing my own custom U-Boot configuration, but I cannot find the OPi5 'defconfig'.  The github repo doesn't seem to include one there.  So curious if anyone here knows where I can obtain that?  I don't want to just start trying different configs, because I could brick the SBC I think with a bad boot loader.

 

This is what I know so far after trial and errors and testing

[ScottN]

Oh, the reason to go down the U-Boot path is to customize its behavior to our needs.  Namely disable any network boot attempts and also setting an environment variable for 'bootdelay' that will issue that reset command after say 30 seconds.  If that was put in there, it is at least a last ditch effort for the board to try again if it failed to boot like I'm experiencing.

 

I looked into modifying the boot loader image I found at the link above, but I can't even mount the .img file env partition. I get "mount: wrong fs type, bad option, bad superblock missing codepage or helper program, or other error", even using a loop device and offset.  So even if I could get that, I would need to be able to mount in write mode.  Seems like a bad path to go down if I can just get the "defconfig" for the OPi5 SBC and just build a new U-Boot image and flash the MTD storage.

[ArmBoy1988]

If this u-boot doesn't work out, perhaps there is another option.  When I was initially setting up my NVME, I was having an issue.  In order to make it work, a small FAT boot partition was created and a 2nd partition for the root fs.  I believe there was an early issue with ext4 partitions on the SPI.  In any case, perhaps using this configuration, you can more easily modify the boot partition to do what you need it to do.

[ScottN]

I was able to get some info on a failure.  See below.  Is this stack trace from the Linux/Armbian kernel or was this U-Boot crash?

 

[  OK  ] Finished System Reboot.
[  OK  ] Reached target System Reboot.
[  133.995572] watchdog: watchdog0: watchdog did not stop!
[  134.573158] reboot: Restarting system
DDR Version V1.08 20220617
LPDDR4X, 2112MHz
channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS=1 Die BW=16 Size=1024MB
channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS=1 Die BW=16 Size=1024MB
channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS=1 Die BW=16 Size=1024MB
channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS=1 Die BW=16 Size=1024MB
Manufacturer ID:0x1 Samsung
CH0 RX Vref:32.7%, TX Vref:18.8%,0.0%
CH1 RX Vref:34.7%, TX Vref:17.8%,0.0%
CH2 RX Vref:30.7%, TX Vref:18.8%,0.0%
CH3 RX Vref:31.7%, TX Vref:17.8%,0.0%
change to F1: 528MHz
change to F2: 1068MHz
change to F3: 1560MHz
change to F0: 2112MHz
out
U-Boot SPL board init
U-Boot SPL 2017.09-orangepi-rk3588 (Jun 27 2023 - 23:16:07)
Trying to boot from MMC1
spl: mmc init failed with error: -123
Trying to boot from MMC2
Card did not respond to voltage select!
spl: mmc init failed with error: -95
Trying to boot from MTD2
Trying fit image at 0x4000 sector
## Verified-boot: 0
## Checking atf-1 0x00040000 ... sha256(d21d14728d...) + OK
## Checking uboot 0x00200000 ... sha256(b6a01cccca...) + OK
## Checking fdt 0x0034a198 ... sha256(6a0656d703...) + OK
## Checking atf-2 0x000f0000 ... sha256(c00c7fd75b...) + OK
## Checking atf-3 0xff100000 ... sha256(71c3a5841b...) + OK
## Checking atf-4 0xff001000 ... sha256(2301cf73be...) + OK
Jumping to U-Boot(0x00200000) via ARM Trusted Firmware(0x00040000)
Total: 506.477 ms

INFO:    Preloader serial: 2
NOTICE:  BL31: v2.3():v2.3-391-g856309329:derrick.huang
NOTICE:  BL31: Built : 14:15:50, Jul 18 2022
INFO:    ext 32k is valid
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    system boots from cpu-hwid-0
INFO:    idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001
INFO:    dfs DDR fsp_params[0].freq_mhz= 2112MHz
INFO:    dfs DDR fsp_params[1].freq_mhz= 528MHz
INFO:    dfs DDR fsp_params[2].freq_mhz= 1068MHz
INFO:    dfs DDR fsp_params[3].freq_mhz= 1560MHz
INFO:    BL31: Initialising Exception Handling Framework
INFO:    BL31: Initializing runtime services
WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
ERROR:   Error initializing runtime service opteed_fast
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x200000
INFO:    SPSR = 0x3c9


U-Boot 2017.09-orangepi-rk3588 (Jun 27 2023 - 23:16:07 +0000)

Model: Orange Pi 5
PreSerial: 2, raw, 0xfeb50000
DRAM:  3.7 GiB
Sysmem: init
Relocation Offset: eda2c000
Relocation fdt: eb9f8e48 - eb9fecb0
CR: M/C/I
Using default environment

mmc@fe2c0000: 0, mmc@fe2e0000: 1

Device 0: unknown device
Card did not respond to voltage select!

Device 0: unknown device

Device 1: 
Device 2: SF: Detected sfc_nor with page size 256 Bytes, erase size 4 KiB, total 16 MiB
Vendor: 0x2207 Rev: V1.00 Prod: sfc_nor
            Type: Hard Disk
            Capacity: 16.0 MB = 0.0 GB (32768 x 512)
... is now current device
Bootdev(scan): mtd 2
PartType: EFI
DM: v2
boot mode: normal
Model: Orange Pi 5
CLK: (sync kernel. arm: enter 1008000 KHz, init 1008000 KHz, kernel 0N/A)
  b0pll 24000 KHz
  b1pll 24000 KHz
  lpll 24000 KHz
  v0pll 24000 KHz
  aupll 24000 KHz
  cpll 1500000 KHz
  gpll 1188000 KHz
  npll 24000 KHz
  ppll 1100000 KHz
  aclk_center_root 702000 KHz
  pclk_center_root 100000 KHz
  hclk_center_root 396000 KHz
  aclk_center_low_root 500000 KHz
  aclk_top_root 750000 KHz
  pclk_top_root 100000 KHz
  aclk_low_top_root 396000 KHz
Net:   No ethernet found.
Hit key to stop autoboot('CTRL+C'):  0 
mmc@fe2c0000: 0
mmc@fe2e0000: 1
Card did not respond to voltage select!

Device 0: Vendor: 0x10ec Rev: VC2S038E Prod: C24DA05003111044201 
            Type: Hard Disk
            Capacity: 976762.3 MB = 953.8 GB (2000409264 x 512)
... is now current device
Scanning nvme 0:1...
Found U-Boot script /boot.scr
reading /boot.scr
3476 bytes read in 0 ms
## Executing script at 00500000
Boot script loaded from nvme 0
reading /armbianEnv.txt
213 bytes read in 1 ms (208 KiB/s)
reading /uInitrd
18472747 bytes read in 48 ms (367 MiB/s)
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
reading /dtb/rockchip/rk3588s-orangepi-5.dtb
234767 bytes read in 2 ms (111.9 MiB/s)
** Unable to read file /dtb/rockchip/overlay/rockchip-rk3588-fixup.scr **
Unknown command 'kaslrseed' - try 'help'
Fdt Ramdisk skip relocation
## Loading init Ramdisk from Legacy Image at 0a200000 ...
   Image Name:   uInitrd
   Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
   Data Size:    18472683 Bytes = 17.6 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 0x0a100000
   Booting using the fdt blob at 0x0a100000
   reserving fdt memory region: addr=a100000 size=9f000
  'reserved-memory' ramoops@110000: addr=110000 size=f0000
   Using Device Tree in place at 000000000a100000, end 000000000a1a1fff
Adding bank: 0x00200000 - 0xf0000000 (size: 0xefe00000)
Total: 2064.838 ms

Starting kernel ...

"Synchronous Abort" handler, esr 0x02000000

* Reason:        Exception from an unknown reason
* PC         =   ffffffff14054000
* LR         =   00000000002022b8
* SP         =   00000000eb9f6ba0
* ESR_EL2    =   0000000002000000
* Reloc Off  =   00000000eda2c000

x0 : 000000000a100000 x1 : 0000000000000000
x2 : 0000000000000000 x3 : 0000000000000000
x4 : 0000000000400000 x5 : 0000000000000001
x6 : 0000000000000008 x7 : 0000000000000004
x8 : 00000000eb9f6d28 x9 : 0000010000000000
x10: 00000000ffffffd0 x11: 0000000000000006
x12: 000000000001869f x13: 00000000eb9ffce6
x14: 000000000a100000 x15: 0000000000000002
x16: 00000000edca156c x17: 0000000000000005
x18: 00000000eb9ffcd0 x19: 0000000000000400
x20: 00000000edd76d70 x21: 0000000000000000
x22: 0000000000000003 x23: 00000000ebc3d338
x24: 00000000ebc3d338 x25: 00000000edd511b8
x26: 0000000000000000 x27: 0000000000000400
x28: 00000000ebc3d360 x29: 00000000eb9f6d80


Call trace:
  PC:	[< ffffffff14054000 >]
  LR:	[< 002022b8 >]

Stack:
	[< ffffffff14054000 >]
	[< 0021b084 >]
	[< 0021ab44 >]
	[< 0022f970 >]
	[< 00218d28 >]
	[< 00218edc >]
	[< 0021862c >]
	[< 00218be0 >]
	[< 00218edc >]
	[< 002185f0 >]
	[< 0022ee90 >]
	[< 00208908 >]
	[< 00208a10 >]
	[< 0022f970 >]
	[< 00218d28 >]
	[< 00218edc >]
	[< 0021862c >]
	[< 00218be0 >]
	[< 00218edc >]
	[< 002185f0 >]
	[< 0022ef20 >]
	[< 0022f970 >]
	[< 00218d28 >]
	[< 002189bc >]
	[< 002189bc >]
	[< 00218edc >]
	[< 002185f0 >]
	[< 0022ef20 >]
	[< 0022f970 >]
	[< 00218d28 >]
	[< 002189bc >]
	[< 00218edc >]
	[< 002185f0 >]
	[< 0022ef20 >]
	[< 0022f970 >]
	[< 00218d28 >]
	[< 002189bc >]
	[< 002189bc >]
	[< 00218edc >]
	[< 002185f0 >]
	[< 0022ef20 >]
	[< 0022f970 >]
	[< 00218d28 >]
	[< 002189bc >]
	[< 00218edc >]
	[< 002185f0 >]
	[< 0022ef20 >]
	[< 0022f970 >]
	[< 00218d28 >]
	[< 00218edc >]
	[< 002185f0 >]
	[< 0022ef20 >]
	[< 0022f970 >]
	[< 00218d28 >]
	[< 00218edc >]
	[< 0021862c >]
	[< 00218be0 >]
	[< 002189bc >]
	[< 00218edc >]
	[< 002185f0 >]
	[< 0022ef20 >]
	[< 0022f970 >]
	[< 00218d28 >]
	[< 00218edc >]
	[< 002185f0 >]
	[< 0022ee90 >]
	[< 0021957c >]
	[< 0021741c >]
	[< 00219c1c >]
	[< 002a78bc >]
	[< 00219e58 >]
	[< 00201e18 >]

Copy info from "Call trace..." to a file(eg. dump.txt), and run
command in your U-Boot project: ./scripts/stacktrace.sh dump.txt 

Resetting CPU ...

### ERROR ### Please RESET the board ###

 

[ScottN]

I'm not sure what to think about that crash above.  I reset the board (unplug/replug) and let it go for almost 24 hours of it rebooting every 2 minutes (network or no network) and it worked good.  So I believe this other Orange Pi 5 U-Boot build I found may work better for us.  I'll be testing more boards with this bootloader as well.  If continued success I'll probably just write this off and use this one going forward.

[HYEOK]

Hi ScottN,

 

A couple of days ago, I bought an Orange Pi 5 PLUS + 980 NVMe M.2 SSD and I am experiencing exactly the same symptom like yours. I was wondering if you could find some solution because your last comment was already 4 months ago. Could you please let us know how you have dealt with the problem so far? 

[ScottN]

On 11/25/2023 at 3:44 PM, HYEOK said:

Could you please let us know how you have dealt with the problem so far?

 

That is correct, I've managed to deal with it but have no concrete solution yet.  I'm not using the OPi 5 Plus, but I have about 70 regular OPi 5's built so far and running in embedded environments, so have some experience with what is working and what isn't.  I'm not sure if the OPi 5 Plus has the same bootloader and potentially the same issue.

 

1. I believe the Armbian u-boot binary may have issues but I cannot point to a specific issue.  I had more reliability when I downloaded and flashed the bootloader with an alternative u-boot from this project: https://github.com/Joshua-Riek/ubuntu-rockchip/releases and I've noticed it is more reliable when booting up and detecting a storage device to boot from (I even did a crontab delay for a few days with no issues booting back up).
    
2. I've also enabled the watchdog within Ubuntu and combined with 'shutdown -H 0' as a way to reboot, the watchdog seems to power cycle the board more fully so storage devices are found on boot to choose from for u-boot.  This is just my educated guess with my own experience and have no assurances that it has fixed anything, I was trying multiple angles here and still use this method though.
    
3. I've started buying a different brand NVMe drive as maybe the one I bought for those 50-ish boards is the cause all along.  But again, not 100% sure and haven't used enough of the new brand NVMe drives to test yet.  Although if you're using a 980 Samsung, that's a pretty well established NVMe brand and makes me wonder now if it isn't the NVMe brand after all...
    
4. I looked into doing my own build of u-boot that could potentially have its own watchdog that if it doesn't boot within 30-60 seconds, it will issue its own watchdog reboot.  But I have no experience in building a custom u-boot.  I'd still like to investigate this route but I haven't had time.  If you know anyone that has done this, I'd like to chat

 

Hope this helps!

 

Scott