Boardcon_yang

Your split between the vendor and mainline paths matches what we see on RK3588 in production, and the AV1 gap is real: 10-bit AV1 needs NV15 and it isn't there yet, so on a mainline kernel the only honest options are falling back to 8-bit or staying on the vendor stack for that format. Worth saying out loud in your app's capability detection rather than letting it fail at runtime. On the vendor side, two things that bite in deployed units rather than on the bench. First, CMA: the default allocation is fine for one stream and falls over somewhere around the third concurrent 4K decode. We ship with cma=256M on anything doing multi-stream and it removes a whole class of allocation failures that otherwise look like driver bugs. Second, if you're using VA-API rather than talking to MPP directly, the driver name has to be set explicitly or you silently get software decode and wonder why the CPU is pegged. Your udev rules are the right ones. One addition that saves time later — put mpp_service and rga in the same group as your app user, not just video, because a service started before the session gets its groups from a different place than an interactive login, and you end up with permissions that work when you test and fail under systemd. The WiFi difference you noticed between branches is worth its own thread if you have time to bisect it.

Hi Abdul, Nice work — the out-of-tree route is honestly the sane one for the vendor kernel. The imx415.c that ships in the 6.1 BSP is an old snapshot, and a couple of things in it bite people: the default mode it probes to is 4K@30, and the sensor's native 4K@60 mode exists in the table but needs the right link frequency and a 4-lane MIPI wiring to sustain ~890 Mbps per lane. If you copied the mode table from a Radxa overlay made for a 2-lane carrier, the driver will load fine and quietly negotiate 30 fps anyway, so it's worth checking the actual sensor-side rate with v4l2-ctl --get-fmt video0 on the sensor subdev, not just the capture node. The other one that wastes days: media-ctl. The rkisp1 pipeline does not inherit formats — every pad (sensor → MIPI CSI2 → isp0) has to be set explicitly with matching formats, or you get a working probe and black frames. Once the mode is real, IMX415's rolling shutter at 4K@60 is quite usable for machine vision; we've shipped it in industrial RK3588 products. Lighting is the only thing that gets ugly — low light and 60 fps means short exposure, so expect to trade gain. Thanks for publishing it properly with the overlay. That's the part everyone keeps in private forks.

Hi Haxce, Direct to eMMC from a fresh current release is fine — you do not need an intermediate image. The 5B installs are "finicky" mostly because of one thing that is not the eMMC itself: the SPI flash. If the board ever carried an older image, the SPI likely holds an old U-Boot that takes precedence in the boot order and hangs before your new install even gets touched. The reliable sequence is: boot the Armbian image from a friend's SD card just once, run armbian-install (nand-sata-install) to write the system to eMMC, then wipe the SPI bootloader so the only U-Boot on the board is the one armbian-install just wrote. After that, power off, remove the card, done. On branch choice for a server that has to last: vendor (6.1 BSP) is the feature-complete one — good if you need MPP video, NPU, camera. Current/edge is mainline and gets the security fixes continuously, which matters more for a box that sits on a network for years. For a headless server doing storage and services I'd take current. If you want the same image line to still be around in three years, that's exactly the reason we moved our industrial boards off vendor BSP years ago — vendor branches freeze and then die, mainline keeps moving. You can even skip the SD card: rkdeveloptool in MASKROM mode flashes eMMC directly from a PC over USB-C. Just don't touch SPI from there unless you know why.

Your debugging is already spot on — journald only stays on /run/log/journal until something runs journalctl --flush, and the fact that a manual flush works proves the storage side is fine. So this is purely an ordering problem: systemd-journal-flush.service runs early in boot, and on Armbian images /var/log.hdd is typically mounted by the armbian-ramlog machinery, which replaces /var/log with its tmpfs/overlay quite late compared to when the flush service expects a real /var/log/journal to exist. If the flush fires while ramlog hasn't finished restoring the /var/log/journal symlink into /var/log.hdd, it silently gives up and journald keeps the runtime journal open — exactly what you're seeing in /proc. You can confirm by comparing timestamps: systemctl status systemd-journal-flush.service versus armbian-ramlog (or mount time of /var/log.hdd). The clean fix is a drop-in for the flush unit, something like After=armbian-ramlog.service plus RequiresMountsFor=/var/log.hdd, so flush waits until the persistent path is actually there. Alternatively, an override on the ramlog unit adding Before=systemd-journal-flush.service achieves the same ordering from the other side. Either way the runtime journal gets flushed on the next boot instead of being stranded in /run until a manual flush.

The identical failure across two boards with nothing in common except the SD card is the tell here. RK3588S and H618 share no kernel, no GPU driver, no display stack — but they do share the first-boot procedure, and that's the most I/O-heavy moment in the whole process: the rootfs gets resized, SSH keys regenerate, caches build. One detail worth noticing: even the Minimal image "failed" on your Zero 3, and Minimal has no desktop to start — so what's hanging is likely the first-boot stage itself, not the desktop. A slow or counterfeit card dies exactly there. They often benchmark fine sequentially and crawl on the small random writes first boot does, so a "fast" card can still be the culprit. Give it a genuine 10-15 minutes watching the activity LED (regular bursts = still working; long dead silence = stuck), and if possible try a known-good name-brand card written with the imager's verify pass enabled. The working text console is your diagnostic window. When it stalls, log in from there and pull journalctl -b for the display manager and dmesg for the tail end — mmc I/O errors confirm the card theory, while a drm or GPU oops on the 6 Plus would point at the graphics side instead. Post those two outputs and the next step usually becomes obvious. Since other OSes run fine, the hardware is almost certainly healthy — this smells like media and first-boot timing rather than a board fault.

The old blaster isn't coming back on 26.04, and it's worth knowing why before chasing it: "odroid blaster" wasn't a lirc configuration, it was an out-of-tree Hardkernel module (lirc_odroid) built for the 3.10 vendor kernel, and hardware.conf died with lirc 0.10 anyway. There's no modern driver for that path, so the clean way is to rebuild the transmit side on the generic kernel infrastructure. Your LED circuit can stay exactly as it is. Declare a gpio-ir-tx node in a DT overlay on the GPIO behind pin 7 of the 40-pin header (Hardkernel's pinout will give you the GPIOX number), keep driving the LED through the same transistor/resistor from the 5V pin, and the TX side shows up as an extra /dev/lirc device. From there ir-ctl replaces irsend: capture the power code from your LG remote with any IR receiver (a $1 gpio-ir on a spare pin, or the onboard receiver if your image enables it) or pull the LG address/command from irdb, verify the LED blinks with a phone camera, then ir-ctl -S lg:: sends it. If you specifically want lircd and irsend back, modern lirc can sit on top of the same /dev/lirc0 — but for a single SEND_ONCE use case, ir-ctl alone is fewer moving parts. One gotcha: gpio-ir-tx is bit-banged, so a heavily loaded system can stretch the first transmission now and then — a repeat usually fixes it, that's not a config error.

Thanks for the detailed write-up — the 50.25 vs 74.25 MHz contrast is exactly the kind of data point the hdptx work needs. We've seen the same pattern on EM3588 (RK3588, dual HDMI 2.1, vendor 6.1): the PHY locks without complaint at standard clocks but gets flaky below ~75 MHz on some panels, which lines up with what the upstream phy-rockchip-samsung-hdptx series has been chasing around PLL and clock-ratio edge cases. The vendor kernel here clearly predates those fixes. Your EDID approach is also the practical answer for industrial displays. Most of the low-resolution panels we run into in signage and access-control products (1024x600, 800x480) sit right in that fragile clock range, and inflating blanking to reuse a known-good dclk keeps the native active area untouched — much cleaner than letting the scaler upscale. The initramfs gotcha you hit is worth repeating too: the connector asks for EDID well before the rootfs is up, so the override has to live in initramfs or it silently misses.

Yes, they're related — I'd treat the PPU probe failure as the direct reason you're not getting renderD128. On the H616 the Mali node in the device tree references a power domain provided by sun50i-h6-prcm-ppu. That driver registers three domains (PLL, ANA, GPU) out of two PRCM registers, and panfrost won't create its render node until the GPU domain exists and is powered. Your log shows the driver bailing out during probe, so the GPU domain never gets registered and panfrost keeps deferring forever. The -22 comes from the PLL entry: the driver marks it always-on, and the PM core refuses to register an always-on domain that reads as off at probe time — that's exactly what the "always-on PM domain PLL is not on" line means. The PLL gate is the active-low bit 2 in the PRCM VDD_SYS register, so you can check the real state with devmem 0x7010250 (bit 2 set = rail gated). Dev boards usually come up with that rail on because their bootloader enables it; plenty of TV boxes like the T95MAX leave it gated, which is why this mostly bites box owners. The clean workaround is to clear that bit before the kernel probes the PPU, e.g. from U-Boot: read the register with md.l, write it back with bit 2 cleared via mw.l. After that the driver should probe cleanly and panfrost should pick up the GPU domain and create the render node. Might also be worth reporting upstream that the driver hard-fails here instead of switching the always-on rail on itself — it would save every H616 box owner this detour.

Nice writeup — following this one closely, RK3576 boards are popping up everywhere lately. One thing worth mentioning: MiniLoaderAll.bin is fine for getting the vendor branch running, but for the mainline path you'll want the idbloader built from mainline U-Boot with the rkbin blobs instead. Armbian refreshed the bl31/bl32 and DDR blobs for RK3576 recently, so a mainline build is closer than it looks — the BL32 hang thread in this forum last month was exactly that stage. Once lubancat-3-v2 boots on vendor, it might be worth a quick try on the edge kernel to see what's still missing before submitting the board config upstream. Saves a round of review comments later. We do RK3576/RK3588 work on the industrial side at Boardcon, so if you run into anything odd during DDR init or PMIC sequencing, happy to compare notes.

Hi Avatar, You don't actually need U-Boot to understand Btrfs. Keep kernel + initrd + extlinux.conf on a small separate /boot (ext4 or ESP) and put only the rootfs on Btrfs subvolumes — booting a snapshot is then just a cmdline change: rootflags=subvol=@snapshots/42/snapshot. U-Boot loads the kernel from ext4 like everywhere else, the kernel mounts the snapshot. That's also why SUSE only needs their enhanced GRUB for convenience. For failure detection, U-Boot already ships the primitive: CONFIG_BOOTCOUNT_LIMIT with bootcmd/altbootcmd. Set fw_setenv upgrade_available 1 before an update, clear bootcount from a systemd oneshot at multi-user.target, and let altbootcmd boot the previous snapshot's extlinux label when bootcount exceeds BOOT_LIMIT. snapper pre/post snapshots, and the rest is just scripting around extlinux.conf — the grub-btrfs menu logic ports over nicely. One footgun to watch: /boot isn't captured by snapshots, so after a rollback make sure initrd/dtb still match the rolled-back kernel.

Nice find, Stefan — and thanks for including the LTSSM state, that makes the root cause unambiguous. LTSSM 0x3 is Detect.Quiet: the PHY never even starts link training because the PCIe rail never comes up. That is exactly what you expect when the regulator's enable GPIO points at a pin that does not drive the M.2 power switch.For anyone wondering how this happened: gpio0 RK_PC5 is a leftover from the Rockchip EVB reference design. The RK3588 EVB routes vcc3v3_pcie20's enable through GPIO0_C5, but the Orange Pi 5 board wires the M.2 power switch to GPIO2_A1. When a board DTS is derived from the EVB, the power-tree GPIOs are the first things to silently go wrong, because everything else (I2C, UART, display) still matches the silicon reference. A couple of points from the industrial/product side, since we maintain a DTS review checklist for our own RK3588 boards going through Armbian bring-up: 1. Cross-check enable GPIOs against the schematic, not against the SoC reference. The EVB-derived pins are the single most common source of "regulator never enables" bugs on Rockchip platforms. 2. While you are in there, check power sequencing too — not just the pin number. On designs where PERST# is also GPIO-controlled, the enable -> PERST# release -> link training order matters, and regulator-ramp-delay / startup-delay can mask or fix marginal boards. A rail that powers up in the wrong order can still enumerate NVMe but fail intermittently at cold boot. 3. Worth pushing the fix upstream: a local overlay is a fine workaround for your own board, but the wrong pin is still in the DTS shipped with mainline kernels (and it reproduced across 6.1.115 / 6.18.x / 7.1.8, so it is the shared DTS, not a kernel regression). A one-line patch to the board DTS would fix this for every user.

On the "intentional?" question: your mirroring experiment is the strongest evidence I have seen that RK3588 does not need the converge. If the layer_sel latch race existed on the rk3588 path,back-porting the rk3568 wait would either fix visible tearing or change timing observably. It did neither — inert, as you said.Combined with the masked cfg-done bits (BIT(vp->id) << 16) making each VP's done signal atomic, my working assumption is also "by design", and it would be interesting to hear from the Rockchip side whether the rk3588 path was deliberately simplified during bring-up.We reproduce the PD accounting pair on every mode change as well: [drm:vop2_power_domain_off_by_disabled_vp] *ERROR* unexpected power on pd6 [drm:vop2_power_domain_off_by_disabled_vp] *ERROR* unexpected power on pd5 On our side it is tied to VP enable/disable ordering during hotplug and dual-HDMI mode switching (EM3588: 2x HDMI 2.1 + DP 1.4 + 2x MIPI-DSI). One note for the industrial angle: in fanless enclosures we watch PD state indirectly through current draw, and these "unexpected power on" events make power telemetry look noisy even though the behaviour is benign. In our data the domains stay latched for a few ms and it does not move the thermal budget — but it would be worth quantifying if anyone is measuring per-domain power on RK3588. On the mainline relationship: since Ciocaltea's multi-output series restructured the cfg_done path, the BSP-only question stands. There is also a related thread on dri-devel where Igor Paunovic is scaling the VOP2 AXI clock (v2 series); in our on-list reply we noted that commit_tail ordering can affect power-domain handoffs on multi-CRTC configs. If that change lands, it may be worth re-testing whether the pd5/pd6 errors shift on the BSP side as well.

As Werner noted, the maintained project is edk2-porting/edk2-rockchip on GitHub: https://github.com/edk2-porting/edk2-rockchip Check the Releases page — the Orange Pi 5 family (RK3588S) builds are there. If there's no specific "5 Max" entry, the Orange Pi 5 build usually matches (same SoC variant), but verify the DDR init config covers your LPDDR5x bin. One thing worth noting: edk2-rockchip ships its own DTB, which can drift from the kernel DTB that Armbian provides. If you hit device-level quirks after switching (NPU, GPU, USB) that's usually the cause — the openSUSE MicroOS thread on this forum has a clean fix using sdbootutil DEVICETREE_SOURCE. Alternatively, Armbian 26.8 (just released) can build UEFI images as bootable Live ISOs, and the rewritten armbian-config installer can flash a bootloader directly to SPI. Worth considering if you're reinstalling anyway — you might not need a separate edk2 flash at all.

Hi thai hoang, Since this is console-only, one thing worth ruling out first: nothing may ever be requesting a mode on that connector at cold boot. status=connected just mirrors HPD — fbcon only sets a mode after the DRM hotplug path fires, which is exactly what your replug gives you. The standard test is to force it from the kernel cmdline: video=DP-1:1366x768e (the trailing e forces the connector enabled at boot, bypassing the hotplug dependency). If that lights the screen, you have the answer and the workaround in one — and modetest -s from libdrm-tests does the same from userspace if you'd rather test without a reboot. If it stays dark with e, then it's genuinely link training, and drm.debug=0x1e will show which side gives up (PHY training vs. cdn-dp firmware handshake). The early-boot dmesg lines (cdn-dp / tcpm / fusb302) would tell which case you're in — Werner's armbianmonitor covers it. On the unbind segfault — that's a real bug in cdn-dp's remove path, not just noise on your end. The driver doesn't tear down cleanly when the PHY/firmware is in a partial state. Worth reporting separately with the oops trace if you have it; either way, unbind isn't a safe reset on this driver.

Hi cu6apum, You already guessed right about the pre-DTB — armbianEnv.txt can't touch this stage. And since BL31 printed at all, DDR init in TPL/SPL succeeded, so it's the trust chain that's dying.On RK3576 that line usually means one of two things: 1. The trust image on flash has no OP-TEE payload (some vendor SDK builds are TEE-disabled), or one built for a different DRAM map — BL31 jumps, nothing answers, silence. 2. BL31 and BL32 aren't from the same rkbin build — happens when a vendor flash and an Armbian image each rewrote part of the chain. Maskrom + rkdeveloptool to dump what's actually on flash, then rewrite idbloader + uboot + trust as a matched set from the closest supported RK3576 board. If it turns out TEE-less, a no-TEE trust combo is a legitimate workaround — you lose the secure boot path but gain a booting board. Happy to compare notes if you post the full UART log.