ScottN

That is correct, I've managed to deal with it but have no concrete solution yet. I'm not using the OPi 5 Plus, but I have about 70 regular OPi 5's built so far and running in embedded environments, so have some experience with what is working and what isn't. I'm not sure if the OPi 5 Plus has the same bootloader and potentially the same issue. I believe the Armbian u-boot binary may have issues but I cannot point to a specific issue. I had more reliability when I downloaded and flashed the bootloader with an alternative u-boot from this project: https://github.com/Joshua-Riek/ubuntu-rockchip/releases and I've noticed it is more reliable when booting up and detecting a storage device to boot from (I even did a crontab delay for a few days with no issues booting back up). I've also enabled the watchdog within Ubuntu and combined with 'shutdown -H 0' as a way to reboot, the watchdog seems to power cycle the board more fully so storage devices are found on boot to choose from for u-boot. This is just my educated guess with my own experience and have no assurances that it has fixed anything, I was trying multiple angles here and still use this method though. I've started buying a different brand NVMe drive as maybe the one I bought for those 50-ish boards is the cause all along. But again, not 100% sure and haven't used enough of the new brand NVMe drives to test yet. Although if you're using a 980 Samsung, that's a pretty well established NVMe brand and makes me wonder now if it isn't the NVMe brand after all... I looked into doing my own build of u-boot that could potentially have its own watchdog that if it doesn't boot within 30-60 seconds, it will issue its own watchdog reboot. But I have no experience in building a custom u-boot. I'd still like to investigate this route but I haven't had time. If you know anyone that has done this, I'd like to chat Hope this helps! Scott

I'm not sure what to think about that crash above. I reset the board (unplug/replug) and let it go for almost 24 hours of it rebooting every 2 minutes (network or no network) and it worked good. So I believe this other Orange Pi 5 U-Boot build I found may work better for us. I'll be testing more boards with this bootloader as well. If continued success I'll probably just write this off and use this one going forward.

I was able to get some info on a failure. See below. Is this stack trace from the Linux/Armbian kernel or was this U-Boot crash? [ OK ] Finished System Reboot. [ OK ] Reached target System Reboot. [ 133.995572] watchdog: watchdog0: watchdog did not stop! [ 134.573158] reboot: Restarting system DDR Version V1.08 20220617 LPDDR4X, 2112MHz channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS=1 Die BW=16 Size=1024MB channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS=1 Die BW=16 Size=1024MB channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS=1 Die BW=16 Size=1024MB channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS=1 Die BW=16 Size=1024MB Manufacturer ID:0x1 Samsung CH0 RX Vref:32.7%, TX Vref:18.8%,0.0% CH1 RX Vref:34.7%, TX Vref:17.8%,0.0% CH2 RX Vref:30.7%, TX Vref:18.8%,0.0% CH3 RX Vref:31.7%, TX Vref:17.8%,0.0% change to F1: 528MHz change to F2: 1068MHz change to F3: 1560MHz change to F0: 2112MHz out U-Boot SPL board init U-Boot SPL 2017.09-orangepi-rk3588 (Jun 27 2023 - 23:16:07) Trying to boot from MMC1 spl: mmc init failed with error: -123 Trying to boot from MMC2 Card did not respond to voltage select! spl: mmc init failed with error: -95 Trying to boot from MTD2 Trying fit image at 0x4000 sector ## Verified-boot: 0 ## Checking atf-1 0x00040000 ... sha256(d21d14728d...) + OK ## Checking uboot 0x00200000 ... sha256(b6a01cccca...) + OK ## Checking fdt 0x0034a198 ... sha256(6a0656d703...) + OK ## Checking atf-2 0x000f0000 ... sha256(c00c7fd75b...) + OK ## Checking atf-3 0xff100000 ... sha256(71c3a5841b...) + OK ## Checking atf-4 0xff001000 ... sha256(2301cf73be...) + OK Jumping to U-Boot(0x00200000) via ARM Trusted Firmware(0x00040000) Total: 506.477 ms INFO: Preloader serial: 2 NOTICE: BL31: v2.3():v2.3-391-g856309329:derrick.huang NOTICE: BL31: Built : 14:15:50, Jul 18 2022 INFO: ext 32k is valid INFO: GICv3 without legacy support detected. INFO: ARM GICv3 driver initialized in EL3 INFO: system boots from cpu-hwid-0 INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001 INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz INFO: BL31: Initialising Exception Handling Framework INFO: BL31: Initializing runtime services WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK ERROR: Error initializing runtime service opteed_fast INFO: BL31: Preparing for EL3 exit to normal world INFO: Entry point address = 0x200000 INFO: SPSR = 0x3c9 U-Boot 2017.09-orangepi-rk3588 (Jun 27 2023 - 23:16:07 +0000) Model: Orange Pi 5 PreSerial: 2, raw, 0xfeb50000 DRAM: 3.7 GiB Sysmem: init Relocation Offset: eda2c000 Relocation fdt: eb9f8e48 - eb9fecb0 CR: M/C/I Using default environment mmc@fe2c0000: 0, mmc@fe2e0000: 1 Device 0: unknown device Card did not respond to voltage select! Device 0: unknown device Device 1: Device 2: SF: Detected sfc_nor with page size 256 Bytes, erase size 4 KiB, total 16 MiB Vendor: 0x2207 Rev: V1.00 Prod: sfc_nor Type: Hard Disk Capacity: 16.0 MB = 0.0 GB (32768 x 512) ... is now current device Bootdev(scan): mtd 2 PartType: EFI DM: v2 boot mode: normal Model: Orange Pi 5 CLK: (sync kernel. arm: enter 1008000 KHz, init 1008000 KHz, kernel 0N/A) b0pll 24000 KHz b1pll 24000 KHz lpll 24000 KHz v0pll 24000 KHz aupll 24000 KHz cpll 1500000 KHz gpll 1188000 KHz npll 24000 KHz ppll 1100000 KHz aclk_center_root 702000 KHz pclk_center_root 100000 KHz hclk_center_root 396000 KHz aclk_center_low_root 500000 KHz aclk_top_root 750000 KHz pclk_top_root 100000 KHz aclk_low_top_root 396000 KHz Net: No ethernet found. Hit key to stop autoboot('CTRL+C'): 0 mmc@fe2c0000: 0 mmc@fe2e0000: 1 Card did not respond to voltage select! Device 0: Vendor: 0x10ec Rev: VC2S038E Prod: C24DA05003111044201 Type: Hard Disk Capacity: 976762.3 MB = 953.8 GB (2000409264 x 512) ... is now current device Scanning nvme 0:1... Found U-Boot script /boot.scr reading /boot.scr 3476 bytes read in 0 ms ## Executing script at 00500000 Boot script loaded from nvme 0 reading /armbianEnv.txt 213 bytes read in 1 ms (208 KiB/s) reading /uInitrd 18472747 bytes read in 48 ms (367 MiB/s) Failed to mount ext2 filesystem... ** Unrecognized filesystem type ** reading /dtb/rockchip/rk3588s-orangepi-5.dtb 234767 bytes read in 2 ms (111.9 MiB/s) ** Unable to read file /dtb/rockchip/overlay/rockchip-rk3588-fixup.scr ** Unknown command 'kaslrseed' - try 'help' Fdt Ramdisk skip relocation ## Loading init Ramdisk from Legacy Image at 0a200000 ... Image Name: uInitrd Image Type: AArch64 Linux RAMDisk Image (gzip compressed) Data Size: 18472683 Bytes = 17.6 MiB Load Address: 00000000 Entry Point: 00000000 Verifying Checksum ... OK ## Flattened Device Tree blob at 0x0a100000 Booting using the fdt blob at 0x0a100000 reserving fdt memory region: addr=a100000 size=9f000 'reserved-memory' ramoops@110000: addr=110000 size=f0000 Using Device Tree in place at 000000000a100000, end 000000000a1a1fff Adding bank: 0x00200000 - 0xf0000000 (size: 0xefe00000) Total: 2064.838 ms Starting kernel ... "Synchronous Abort" handler, esr 0x02000000 * Reason: Exception from an unknown reason * PC = ffffffff14054000 * LR = 00000000002022b8 * SP = 00000000eb9f6ba0 * ESR_EL2 = 0000000002000000 * Reloc Off = 00000000eda2c000 x0 : 000000000a100000 x1 : 0000000000000000 x2 : 0000000000000000 x3 : 0000000000000000 x4 : 0000000000400000 x5 : 0000000000000001 x6 : 0000000000000008 x7 : 0000000000000004 x8 : 00000000eb9f6d28 x9 : 0000010000000000 x10: 00000000ffffffd0 x11: 0000000000000006 x12: 000000000001869f x13: 00000000eb9ffce6 x14: 000000000a100000 x15: 0000000000000002 x16: 00000000edca156c x17: 0000000000000005 x18: 00000000eb9ffcd0 x19: 0000000000000400 x20: 00000000edd76d70 x21: 0000000000000000 x22: 0000000000000003 x23: 00000000ebc3d338 x24: 00000000ebc3d338 x25: 00000000edd511b8 x26: 0000000000000000 x27: 0000000000000400 x28: 00000000ebc3d360 x29: 00000000eb9f6d80 Call trace: PC: [< ffffffff14054000 >] LR: [< 002022b8 >] Stack: [< ffffffff14054000 >] [< 0021b084 >] [< 0021ab44 >] [< 0022f970 >] [< 00218d28 >] [< 00218edc >] [< 0021862c >] [< 00218be0 >] [< 00218edc >] [< 002185f0 >] [< 0022ee90 >] [< 00208908 >] [< 00208a10 >] [< 0022f970 >] [< 00218d28 >] [< 00218edc >] [< 0021862c >] [< 00218be0 >] [< 00218edc >] [< 002185f0 >] [< 0022ef20 >] [< 0022f970 >] [< 00218d28 >] [< 002189bc >] [< 002189bc >] [< 00218edc >] [< 002185f0 >] [< 0022ef20 >] [< 0022f970 >] [< 00218d28 >] [< 002189bc >] [< 00218edc >] [< 002185f0 >] [< 0022ef20 >] [< 0022f970 >] [< 00218d28 >] [< 002189bc >] [< 002189bc >] [< 00218edc >] [< 002185f0 >] [< 0022ef20 >] [< 0022f970 >] [< 00218d28 >] [< 002189bc >] [< 00218edc >] [< 002185f0 >] [< 0022ef20 >] [< 0022f970 >] [< 00218d28 >] [< 00218edc >] [< 002185f0 >] [< 0022ef20 >] [< 0022f970 >] [< 00218d28 >] [< 00218edc >] [< 0021862c >] [< 00218be0 >] [< 002189bc >] [< 00218edc >] [< 002185f0 >] [< 0022ef20 >] [< 0022f970 >] [< 00218d28 >] [< 00218edc >] [< 002185f0 >] [< 0022ee90 >] [< 0021957c >] [< 0021741c >] [< 00219c1c >] [< 002a78bc >] [< 00219e58 >] [< 00201e18 >] Copy info from "Call trace..." to a file(eg. dump.txt), and run command in your U-Boot project: ./scripts/stacktrace.sh dump.txt Resetting CPU ... ### ERROR ### Please RESET the board ###

Oh, the reason to go down the U-Boot path is to customize its behavior to our needs. Namely disable any network boot attempts and also setting an environment variable for 'bootdelay' that will issue that reset command after say 30 seconds. If that was put in there, it is at least a last ditch effort for the board to try again if it failed to boot like I'm experiencing. I looked into modifying the boot loader image I found at the link above, but I can't even mount the .img file env partition. I get "mount: wrong fs type, bad option, bad superblock missing codepage or helper program, or other error", even using a loop device and offset. So even if I could get that, I would need to be able to mount in write mode. Seems like a bad path to go down if I can just get the "defconfig" for the OPi5 SBC and just build a new U-Boot image and flash the MTD storage.

Hi ArmBoy1988, yes we use these as small media players, think of like signage and other things in public spaces. I've been reading up on U-Boot more, which seems to be the boot loader here. I found another boot loader from a small project here which I downloaded the image, wrote to a microsd, and copied off the 'rkspi_loader.img' file from there, and then used dd to write that to '/dev/mtdblock0'. That boots the OPi on the NVMe just fine. I created a '@reboot' crontab that waits a couple minutes and issues a normal reboot. I'm monitoring the serial output doing this to a file. What I've noticed (but still testing to confirm) is if I do NOT connect the network cable, this thing reboots great for hours on end, no issues. So I have to rule out the NVMe at that point I suppose? I'm about to try to test by connecting the network cable back up and see if after a bit it hangs. Btw, when it did hang with the different boot loader, I was at the u-boot command prompt, I did the NVMe command to list devices and the storage is there. I then just issued a reset command and it booted just fine. So with that, my suspicion is maybe U-Boot is trying a network boot and failing? I'm starting to now research doing my own custom U-Boot configuration, but I cannot find the OPi5 'defconfig'. The github repo doesn't seem to include one there. So curious if anyone here knows where I can obtain that? I don't want to just start trying different configs, because I could brick the SBC I think with a bad boot loader. This is what I know so far after trial and errors and testing

I've been having a heck of a time trying to catch this issue with the devices I have as I'm setting them up. If I'm preparing a batch of them, say of 10, 2 of them will act up then not agaom. Of course I cannot have a TTY connection open to all of them at once so it's by sheer luck that I can have the serial connected and it'll happen. I haven't been able to be that lucky yet so I don't know. But what I have now is one that hung on boot, I left it there and it's powered on, so I connected the serial connection and when I hit enter, I get a "opi#" prompt. Is there any way to print any log files from this that could tell me anything? Or anything I can do to help diagnose why it did not start the Linux kernel? EDIT: So help shows some things I can do. Listed below. But if I do 'nvme device' I get 'no nvme devices available'. Which could be why it won't boot to an nvme ? - alias for 'help' android_print_hdr- print android image header atags - Dump all atags base - print or set address offset bdinfo - print Board Info structure bidram_dump- Dump bidram layout blk - Block device sub-system boot - boot default, i.e., run 'bootcmd' boot_android- Execute the Android Bootloader flow. boot_fit- Boot FIT Image from memory or boot/recovery partition bootavb - Execute the Android avb a/b boot flow. bootd - boot default, i.e., run 'bootcmd' booti - boot arm64 Linux Image image from memory bootm - boot application image from memory bootp - boot image via network using BOOTP/TFTP protocol bootz - boot Linux zImage image from memory charge - Charge display charge_pd- Charge select cmp - memory compare coninfo - print console devices and information cp - memory copy crc32 - checksum calculation crypto_sum- crypto checksum engine dhcp - boot image via network using DHCP/TFTP protocol dm - Driver model low level access download- enter rockusb/bootrom download mode dtimg - manipulate dtb/dtbo Android image dump_irqs- Dump IRQs dump_resource- dump resource list echo - echo args to console editenv - edit environment variable env - environment handling commands exit - exit script ext2load- load binary file from a Ext2 filesystem ext2ls - list files in a directory (default /) ext4load- load binary file from a Ext4 filesystem ext4ls - list files in a directory (default /) ext4size- determine a file's size false - do nothing, unsuccessfully fastboot- use USB or UDP Fastboot protocol fatinfo - print information about filesystem fatload - load binary file from a dos filesystem fatls - list files in a directory (default /) fatsize - determine a file's size fatwrite- write file into a dos filesystem fdt - flattened device tree utility commands fstype - Look up a filesystem type go - start application at address 'addr' gpt - GUID Partition Table help - print command description/usage iomem - Show iomem data by device compatible(high priority) or node name lcdputs - print string on video framebuffer load - load binary file from a filesystem loop - infinite loop on address range ls - list files in a directory (default /) md - memory display mdio - MDIO utility commands mii - MII utility commands mm - memory modify (auto-incrementing address) mmc - MMC sub system mmcinfo - display MMC info mtd_blk - MTD Block device sub-system mw - memory write (fill) nfs - boot image via network using NFS protocol nm - memory modify (constant address) nvme - NVM Express sub-system part - disk partition related commands pci - list and access PCI Configuration Space ping - send ICMP ECHO_REQUEST to network host printenv- print environment variables pxe - commands to get and boot from pxe files rbrom - Perform RESET of the CPU reboot - Perform RESET of the CPU, alias of 'reset' reset - Perform RESET of the CPU rkimgtest- Test if storage media have rockchip image rockchip_show_bmp- load and display bmp from resource partition rockchip_show_logo- load and display log from resource partition rockusb - Use the rockusb Protocol run - run commands in an environment variable save - save file to a filesystem saveenv - save environment variables to persistent storage scsi - SCSI sub-system scsiboot- boot from SCSI device setcurs - set cursor position within screen setenv - set environment variables setexpr - set environment variable as the result of eval expression sf - SPI flash sub-system showvar - print local hushshell variables size - determine a file's size source - run script from memory sspi - SPI utility command sysboot - command to get and boot from syslinux files sysmem_dump- Dump sysmem layout sysmem_search- Search a available sysmem region test - minimal test like /bin/sh tftp - download image via network using TFTP protocol tftpbootm- tftpbootm aosp/uImage/FIT image via network using TFTP protocol tftpflash- flash image via network using TFTP protocol tftpput - TFTP put command, for uploading files to a server true - do nothing, successfully ums - Use the UMS [USB Mass Storage] usb - USB sub-system usbboot - boot from USB device version - print monitor, compiler and linker version opi# version U-Boot 2017.09-armbian (Feb 17 2023 - 22:22:49 +0000) aarch64-linux-gnu-gcc (Linaro GCC 7.4-2019.02) 7.4.1 20181213 [linaro-7.4-2019.02 revision 56ec6f6b99cc167ff0c2f8e1a2eed33b1edc85d4] GNU ld (Linaro_Binutils-2019.02) 2.28.2.20170706

An update on the UART serial connection. I bought this one and used the command 'sudo tio /dev/ttyUSB0 -b 1500000 -d 8 -f none -p none -s 1' and it is working beautifully now and prompts for login and on reboots get full verbose output. So I will be able to provide some more info shortly on when the reboot hangs.

That's if you have access to the board We're shipping these players out in the field for use and need to reliably reboot. As I mentioned above in a reply, having the watchdog enabled and doing a 'shutdown -H 0' halt command, the watchdog will kick and reboot the board. A little janky for sure, but a workaround. I'd really like to find why this is happening and then patch the boards in the field remotely.

It connects, but when I hit ENTER on the keyboard, I get: PAcbwo`@8 When the player reboots, I get the same garbled output as previous. Below is a brief snip on when the OPi5 starts to shutdown. `@@@HAD@@$ ,@@h@`@p0@@``p`@q`HhhNCBAR 0! $0 `nBKB@@`)@@```@@@HaD@@$ ,`@L@`@p0@@`pp`@p`HHHoBC@ 0 , 6"60 X`OBBB@@@@```@p(@@H@DA@$ ,`@m@`@p0@@``p`@qh@n@HnCC@ Z 2 D[ ;" H H`n``A@ 2 2#0I@`HA`b@`C@sr`onb 2" H HX`n``@@`@`G@` 2 :"0H@`fO`B H [[ :" h H`N``A@`@`F@` 2 2#0HbE`@[Z) 2ALFR!D H 2uL D@bb` D[> h "R `n``@ :R :!2"!D 0I"12) DZ? @[ 07701100b@@No`8pb`@l`@H`E@h@`wo`C@HAP@sOHq`ANL H`N``ALF1220Ibu``HAB@L` H, And just more similar to that as it boots back up.

I found the other thread trying to figure out the UART serial communication I will pick up this one that is recommended. Question is, what's the correct way to do this? The thread I linked mentions 3.3v? Any more clarification on how to do this?

Little update. I connected the USB to TTL to my Ubuntu laptop and used tio. Same result as Putty. sudo tio /dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_A\:AZt114J20-if00-port0 -b 1500000 -d 8 -f none -p none -s 1

I did make this mistake at first, didn't get anything coming through Putty and the connection, so I swapped them and that's when this garbage started. I'll play around with it. I do have the connections on the 3 pin header for UART, which is correct, right? Looking at the GPIO there are other UART pins there too. The Orange Pi 5 manual covers them. So, not sure what's happening here. This isn't some text encoding thing is it?

Hmmm, I'm not sure actually. Didn't know to look for that honestly. This is what I bought: https://www.amazon.com/dp/B07R8BQYW1 Do you have a recommendation for a different one? On a side note, I have found a workaround for this reboot issue. Firstly, you need watchdog enabled. Then you issue a halt command: shutdown -H 0 What happens, I believe, Linux shuts down normally, but isn't a full shutdown and power off. Then, the watchdog kicks shortly after and reboots the board. If I issue a reboot with 'reboot now', the bootloader bug appears rather frequently. Sometimes it'll boot back up, sometimes not. I'm seeing well over 50% of the times it's not. Hopefully I can get this serial console output with verbosity and give more information. I'm just happy I have a temporary workaround.

Thank you for the instructions. I had to buy a USB to serial/TTL cable and now have it hooked up. But I have something weird, getting gibberish in the putty output. I've logged a full reboot to a log file and attached it. My putty config: 2023-06-05 11:49:14 Configuring baud rate 1500000 2023-06-05 11:49:14 Configuring 8 data bits 2023-06-05 11:49:14 Configuring 1 data bits 2023-06-05 11:49:14 Configuring no parity 2023-06-05 11:49:14 Configuring no flow control What do I have wrong here? putty.log

Just to clarify what I need to do on my end. Do I need to connect serial cable to an external machine with serial port, open Putty and connect the other end to the OPi serial connections? Or do I need to plug a USB to serial adapter into the Pi, then connect the other to my external machine. Any documentation or more detailed steps for this? Thanks!