Johson Posted December 4, 2025 Posted December 4, 2025 (edited) I've already benchmarked a computer-interactivbe task (like image processing) on the Orangepi 5 plus's processor——and it wrapped up in just 40 minutes, which is impressively fast. Therefore, I' like to try the NPU. Anyone know how to tap into its NPU for AI workloads? Edited December 4, 2025 by Johson 0 Quote
robertoj Posted December 4, 2025 Posted December 4, 2025 (edited) I would like to know too. The orange pi zero 3 has a GPU that's available for SIMD acceleration through the latest OpenGLES library, but I haven't had time for that: https://ai.google.dev/edge/mediapipe/framework/getting_started/gpu_support Try it in your opi5+, then run the mediapipe python examples. Then look for other neural network tasks that use the same NN engine: tensor flow light (tensor flow, pytorch will need a different method) Other examples: https://forum.armbian.com/topic/28895-efforts-to-develop-firmware-for-h96-max-v56-rk3566-8g64g/#comment-167001 https://opencv.org/blog/working-with-neural-processing-units-npus-using-opencv/ Edited December 4, 2025 by robertoj 0 Quote
usual user Posted December 5, 2025 Posted December 5, 2025 FWIW, on my rk3588 devices the NPUs are working with recent mainline releases: [ 5.967316] [drm] Initialized rocket 0.0.0 for rknn on minor 0 [ 5.975499] rocket fdab0000.npu: Rockchip NPU core 0 version: 1179210309 [ 5.978652] rocket fdac0000.npu: Rockchip NPU core 1 version: 1179210309 [ 5.985602] rocket fdad0000.npu: Rockchip NPU core 2 version: 1179210309 This script runs the Mesa example with the latest available working versions: Spoiler #!/bin/bash IMAGE="grace_hopper.bmp" WORKBENCH="." ENVIRONMENT="${WORKBENCH}/python/3.11" [ "${1}" == "setup" ] || [ ! -f ${ENVIRONMENT}/bin/activate ] && BOOTSTRAP="true" [ -v BOOTSTRAP ] && python3.11 -m venv ${ENVIRONMENT} source ${ENVIRONMENT}/bin/activate [ -v BOOTSTRAP ] && pip install numpy==1.26.4 [ -v BOOTSTRAP ] && pip install pillow==12.0.0 [ -v BOOTSTRAP ] && pip install tflite-runtime==2.14.0 TEFLON_DEBUG=verbose ETNA_MESA_DEBUG=ml_dbgs python ${WORKBENCH}/classification-tflite.py \ -i ${WORKBENCH}/${IMAGE} \ -m ${WORKBENCH}/mobilenet_v1_1_224_quant.tflite \ -l ${WORKBENCH}/labels_mobilenet_quant_v1_224.txt \ -e /usr/lib64/libteflon.so deactivate And with this script, the Mesa example runs, with a small adjustment, also with the TFLite successor LiteRT: Spoiler #!/bin/bash IMAGE="grace_hopper.bmp" WORKBENCH="." ENVIRONMENT="${WORKBENCH}/python/3.13" [ "${1}" == "setup" ] || [ ! -f ${ENVIRONMENT}/bin/activate ] && BOOTSTRAP="true" [ -v BOOTSTRAP ] && python3.13 -m venv ${ENVIRONMENT} source ${ENVIRONMENT}/bin/activate [ -v BOOTSTRAP ] && pip install pillow [ -v BOOTSTRAP ] && pip install ai-edge-litert-nightly TEFLON_DEBUG=verbose ETNA_MESA_DEBUG=ml_dbgs python ${WORKBENCH}/classification-litert.py \ -i ${WORKBENCH}/${IMAGE} \ -m ${WORKBENCH}/mobilenet_v1_1_224_quant.tflite \ -l ${WORKBENCH}/labels_mobilenet_quant_v1_224.txt \ -e /usr/lib64/libteflon.so deactivate A MediaPipe sample can also be set up easily: Spoiler #!/bin/bash WORKBENCH="." ENVIRONMENT="${WORKBENCH}/python/3.12" [ "${1}" == "setup" ] || [ ! -f ${ENVIRONMENT}/bin/activate ] && BOOTSTRAP="true" [ -v BOOTSTRAP ] && python3.12 -m venv ${ENVIRONMENT} source ${ENVIRONMENT}/bin/activate [ -v BOOTSTRAP ] && pip install mediapipe [ -v BOOTSTRAP ] && pip install pillow [ -v BOOTSTRAP ] && pip install ai-edge-litert-nightly python ${WORKBENCH}/detect.py --model efficientdet_lite0.tflite But unfortunately, the MediaPipe framework does not support the extended delegate functionality of LiteRT (TFLite). And therefore no NPU support. classification-3.11-tflite.logclassification-3.13-litert.logobject_detection-3.12-litert.log 1 Quote
flappyjet Posted March 1 Posted March 1 Do you have rocket kernel with mesa that enable NPU working now? Or with rknn kernel? I'm a hobbyist working on a script to detect the NPU environment for ARM64 edge devices. Since I only have limited hardware, I’m looking for volunteers to run a quick test and see what is the environment on different NPU-equipped boards. This is a personal experimental project with no guarantees, just a survey to see what works and what doesn't. GitHub Repository: [npu-toolbox](https://github.com/flappyjet/npu-toolbox/) Test Script: `curl -sSL https://github.com/flappyjet/npu-toolbox/raw/refs/heads/main/scripts/npu_probe.sh | bash` Report Results: Please drop a comment in this survey issue: [Call for Testing: NPU detection Script](https://github.com/flappyjet/npu-toolbox/issues/1) Here's my A311d device info Quote # npu-toolbox probe [platform] comatible="onethingcloud,oes,amlogic,a311d,amlogic,g12b," model="OneThing Cloud OES," family="Amlogic Meson" soc_id="G12B (A311D)" arch="aarch64" kernel="6.12.59-ophub" [modules] galcore="" etanviv="/sys/module/etnaviv" [devices] npu_device="/dev/dri/renderD128" # Driver: etnaviv devices_not_support=[ "/dev/dri/renderD129", # Driver: panfrost ] [libraries] delegate="/usr/local/lib/aarch64-linux-gnu/libteflon.so" # npu-toolbox benchmark Quote INFO: Created TensorFlow Lite XNNPACK delegate for CPU. ================================================================================ Model: bird Image | Backend | Time (ms) | Result -------------------------------------------------------------------------------- bird.jpg | CPU | 47.40 | ID:671|Passer domesticus (House Sparrow) -------------------------------------------------------------------------------- bird.jpg | NPU | 13.04 | ID:671|Passer domesticus (House Sparrow) -------------------------------------------------------------------------------- people.jpg | CPU | 47.32 | ID:964|background -------------------------------------------------------------------------------- people.jpg | NPU | 7.79 | ID:964|background -------------------------------------------------------------------------------- street.jpg | CPU | 47.28 | ID:964|background -------------------------------------------------------------------------------- street.jpg | NPU | 7.82 | ID:964|background -------------------------------------------------------------------------------- ================================================================================ Model: mobilenet Image | Backend | Time (ms) | Result -------------------------------------------------------------------------------- bird.jpg | CPU | 47.39 | ID:11|brambling, Fringilla montifringilla -------------------------------------------------------------------------------- bird.jpg | NPU | 13.53 | ID:11|brambling, Fringilla montifringilla -------------------------------------------------------------------------------- people.jpg | CPU | 47.38 | ID:646|maypole -------------------------------------------------------------------------------- people.jpg | NPU | 8.00 | ID:646|maypole -------------------------------------------------------------------------------- street.jpg | CPU | 47.56 | ID:921|traffic light, traffic signal, stoplight -------------------------------------------------------------------------------- street.jpg | NPU | 8.02 | ID:921|traffic light, traffic signal, stoplight -------------------------------------------------------------------------------- ================================================================================ Model: ssdlite Image | Backend | Time (ms) | Result -------------------------------------------------------------------------------- bird.jpg | CPU | 208.39 | Detected 1 -------------------------------------------------------------------------------- bird.jpg | NPU | 31.77 | Detected 1 -------------------------------------------------------------------------------- people.jpg | CPU | 208.49 | Detected 10 -------------------------------------------------------------------------------- people.jpg | NPU | 21.96 | Detected 10 -------------------------------------------------------------------------------- street.jpg | CPU | 208.35 | Detected 10 -------------------------------------------------------------------------------- street.jpg | NPU | 21.87 | Detected 10 -------------------------------------------------------------------------------- 0 Quote
usual user Posted March 1 Posted March 1 8 hours ago, flappyjet said: Do you have rocket kernel with mesa that enable NPU working now? This is what my software stack looks like: My kernel is build as a generic one, hence my OS is working on any device equipped with a VeriSilicon VIPNano, a Rockchip RK3588 or an Arm Ethos-U65/U85 NPU. The application can be written NPU-agnostic, as long as a model.tflite file suitable for the NPU is used. 0 Quote
flappyjet Posted March 2 Posted March 2 What a nice picture! Would you be generous to post your NPU device info? I'm also looking for some w8a8 llm models working on NPU (the GEMM capability is perfect, isn't it). 0 Quote
usual user Posted Friday at 05:11 PM Posted Friday at 05:11 PM (edited) On 3/2/2026 at 2:22 AM, flappyjet said: Would you be generous to post your NPU device info? Since the hardware support for Rockchip SoCs in the mainline kernelis generally already very outstanding and their further development is also being actively pursued, I only have SBCs with integrated NPUs that are based on them. Among them are ODROID-M2, NanoPC-T6, and ROCK-5-ITX. But since the NPU is an integral part of the SoC, the board manufacturer and the design of the SBC are not necessarily of importance. On 3/2/2026 at 2:22 AM, flappyjet said: I'm also looking for some w8a8 llm models working on NPU (the GEMM capability is perfect, isn't it). As far as I understand, edge-class NPUs are best suited for computer vision tasks. I am therefore engaged in object detection: and super-resolution: Edited Friday at 05:59 PM by usual user 0 Quote
mircsicz Posted 12 hours ago Posted 12 hours ago (edited) Hi all, I kinda set myself in a corner: Last year in October I installed the UEFI image in SPI on my OrangePi5 Plus, then I installed the uefi armbian image on the Nvme ssd. I did all of that after a system failure during my 3 month absence from where I've got that OPi5 setup. As I couldn't get it back to work I went and tried with the Uefi Image... But it seems that this was a major mistake: I could never get my frigate container back to running as there's no NPU's detected! Just now I took the leap of faith and reinstalled uboot into the SPI and flashed a freshly loaded "Armbian_25.11.1_Orangepi5-plus_trixie_current_6.12.58_minimal.img.xz"... And still there's no NPU: I can't see a thing about a NPU: mircsicz@orangepi5-plus:~$ sudo dmesg|grep rocket mircsicz@orangepi5-plus:~$ sudo dmesg|grep NPU mircsicz@orangepi5-plus:~$ sudo dmesg|grep GPU mircsicz@orangepi5-plus:~$ sudo dmesg|grep gpu [ 6.133753] panthor fb000000.gpu: [drm] clock rate = 198000000 [ 6.136216] panthor fb000000.gpu: [drm] Mali-G610 id 0xa867 major 0x0 minor 0x0 status 0x5 [ 6.136222] panthor fb000000.gpu: [drm] Features: L2:0x7120306 Tiler:0x809 Mem:0x301 MMU:0x2830 AS:0xff [ 6.136226] panthor fb000000.gpu: [drm] shader_present=0x50005 l2_present=0x1 tiler_present=0x1 [ 6.160653] panthor fb000000.gpu: [drm] Firmware protected mode entry not be supported, ignoring [ 6.160711] panthor fb000000.gpu: [drm] Firmware git sha: 814b47b551159067b67a37c4e9adda458ad9d852 [ 6.160868] panthor fb000000.gpu: [drm] CSF FW using interface v1.1.0, Features 0x0 Instrumentation features 0x71 [ 6.161137] [drm] Initialized panthor 1.5.0 for fb000000.gpu on minor 1 mircsicz@orangepi5-plus:~$ uname -a Linux orangepi5-plus 6.18.0-rc6-edge-rockchip64 #1 SMP PREEMPT Sun Nov 16 22:25:38 UTC 2025 aarch64 GNU/Linux Nor with the npu-probe.sh script: # Auto-generated NPU detection record # Generated at: Mon Mar 9 07:32:45 PM EST 2026 [platform] comatible="xunlong,orangepi-5-plus,rockchip,rk3588," model="Xunlong Orange Pi 5 Plus," family="unknown" soc_id="unknown" arch="aarch64" kernel="6.18.0-rc6-edge-rockchip64" [modules] rknpu="" rocket="" [devices] no_detected_npu=1 devices_not_support=[ "/dev/dri/renderD128", # Driver: panthor ] [libraries] no_delegate=1 vendor_libs=[] So I came here hoping one of you guys will be able to gi'me a hint... After flashing a recent dev build of v26.2 with vendor Kernel to another SD I finally can see the NPU again: # Auto-generated NPU detection record # Generated at: Mon Mar 9 08:20:56 PM EST 2026 [platform] comatible="rockchip,rk3588-orangepi-5-plus,rockchip,rk3588," model="Orange Pi 5 Plus," family="unknown" soc_id="unknown" arch="aarch64" kernel="6.1.115-vendor-rk35xx" [modules] rknpu="/sys/module/rknpu" rocket="" [devices] npu_device="/dev/dri/renderD129" # Driver: RKNPU devices_not_support=[ "/dev/dri/renderD128", # Driver: rockchip-drm ] [libraries] no_delegate=1 vendor_libs=[] Also added my entry to the Github mentioned above 😉 So now I need to ask @usual user what specific kernel are you running? It seems to me that with the kernels available on the repo's only 6.1.115 allows me to use the NPU... Edited 11 hours ago by mircsicz 0 Quote
flappyjet Posted 2 hours ago Posted 2 hours ago Quote Quote As far as I understand, edge-class NPUs are best suited for computer vision tasks. I agree with you. Neural network works well on NPU. I tried to load tiny llm model, but it's still too large for NPU. The bottleneck is the sram in NPU, not computing capacity. And I found an interesting project https://github.com/NotPunchnox/rkllama 0 Quote
usual user Posted 1 hour ago Posted 1 hour ago 10 hours ago, mircsicz said: So now I need to ask @usual user what specific kernel are you running? I am currently at 7.0.0-rc1. I can upload my jump-start image so you can check if my kernel build works with your device. If you like what you see, it is only a 'prepare-jump-start ${target-mount-point}' away to install the kernel package alongside your existing system. 1 hour ago, flappyjet said: I found an interesting project https://github.com/NotPunchnox/rkllama I know about it, but since it is just another not mainline solution with another dependency mess, I am not particularly interested. 0 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.