eselarm

root@rock3a:/lan/tmp# ffmpeg -i file1.AV1.mkv -c:v hevc_rkmpp -c:a copy file1.x265.mkv ffmpeg version 5.1.6-0+deb12u1 Copyright (c) 2000-2024 the FFmpeg developers ... Unknown encoder 'hevc_rkmpp' root@rock3a:/lan/tmp# /usr/share/jellyfin-ffmpeg/ffmpeg -i file1.AV1.mkv -c:v hevc_rkmpp -c:a copy file1.x265.mkv ffmpeg version 7.0.2-Jellyfin Copyright (c) 2000-2024 the FFmpeg developers ... [out#0/matroska @ 0xaaaaeff1e660] video:223157KiB audio:6460KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.220596% frame=27421 fps=138 q=-0.0 Lsize= 230124KiB time=00:15:21.52 bitrate=2045.7kbits/s speed=4.64x video plays fine You might have power issues, that is my thirst thought, as I had several issues with my rock3a when I used USB-C (so with default PD with 5V). Now 12V in and user on-board DC-DC to do proper feeding. OPi5 does not have this AFAIK from schematics. But is is a wild guess. Many other things can be wrong, like who knows something with iommu (I guess not). I might try same command sometime later on my NanoPi-R6C when I will upgrade Armbian from Bookworm to Testing/Trixie, it also has RK3588S, same as OPi5, but check this, it is just top-of-my-head. I use mainline kernels mostly now, so need to tune grub or so first.

My pihole is a virtual machine; I do not use SD-card for it, I keep virtual machine 'objects' around on various machines, so I do not have to wait 'burning' SD-cards. I can tell way more to prevent further assumptions being made about how I operate computers in my home and foreign country, but this is getting boring. I won't click all the various URLs. I would rather start using bcachefs (again) and see how long it will take before Kent Overstreet will have added a send|receive in there like now is available in ZFS and Btrfs. But it seems you did not get the point that I use the word 'snapshot' to identify both: (pseudo code) raspi1:/.snapshots/numberX/snapshot and othermachine:/backups/numberY/snapshot (and multiple othermachines, clone NAS as I said) On raspi1 it is local on SD-card, on othermachine(s) is the exact same filetree and same subvol (received) UUID. If I would not trust btrfs send|receive, which could be the case sometimes when doing btrfs development kernel patching around 2013 timeframe, I could do a an rsync -avxc between the 2, where 1 side is a just taken RW snapshot of the otherwise RO snapshot. So you will see rsync do writes to the RW subvol if something had gone wrong with kernel implementation or userspace. Or if i manually poked 1 sector random data in some some random file at some random offset (simulating a HDD bad sector). I have done that as test several times. It take time to do that between 2 4TB HDDs but not a problem.

do you have correct permissions? maybe run as root on my rock3a content is fine but got [Jul 4 20:19] warn_alloc: 19 callbacks suppressed [ +0.000026] dec0:0:hevc_rkm: page allocation failure: order:5, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=user.slice,mems_allowed=0 [ +0.000115] CPU: 2 PID: 2339021 Comm: dec0:0:hevc_rkm Not tainted 6.1.115-vendor-rk35xx #1 [ +0.000011] Hardware name: Radxa ROCK3 Model A (DT) [ +0.000011] Call trace: [ +0.000008] dump_backtrace+0xf0/0x12c [ +0.000023] show_stack+0x20/0x30 [ +0.000011] dump_stack_lvl+0x7c/0xa0 [ +0.000016] dump_stack+0x18/0x34 [ +0.000010] warn_alloc+0xe0/0x17c [ +0.000012] __alloc_pages+0x524/0x854 [ +0.000010] __kmalloc_large_node+0xb8/0x114 [ +0.000012] __kmalloc+0x4c/0x100 [ +0.000009] __regset_get+0x60/0xd8 [ +0.000013] regset_get_alloc+0x1c/0x28 [ +0.000010] elf_core_dump+0x500/0xbc8 [ +0.000011] do_coredump+0xabc/0x100c [ +0.000012] get_signal+0x1b8/0x634 [ +0.000013] do_notify_resume+0x194/0xda8 [ +0.000010] el0_da+0x5c/0x70 [ +0.000012] el0t_64_sync_handler+0xc0/0x13c [ +0.000010] el0t_64_sync+0x19c/0x1a0 [ +0.000013] Mem-Info: [ +0.000010] active_anon:112109 inactive_anon:125820 isolated_anon:0 active_file:31285 inactive_file:45270 isolated_file:0 unevictable:0 dirty:2210 writeback:0 slab_reclaimable:16665 slab_unreclaimable:16446 mapped:15455 shmem:105 pagetables:1861 sec_pagetables:510 bounce:0 kernel_misc_reclaimable:0 free:82604 free_pcp:125 free_cma:35560 [ +0.000022] Node 0 active_anon:448436kB inactive_anon:503280kB active_file:125140kB inactive_file:181080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:61820kB dirty:8840kB writeback:0kB shmem:420kB writeback_tmp:0kB kernel_stack:4192kB pagetables:7444kB sec_pagetables:2040kB all_unreclaimable? no [ +0.000017] DMA free:330416kB boost:14364kB min:19988kB low:21964kB high:23940kB reserved_highatomic:2048KB active_anon:448188kB inactive_anon:503568kB active_file:125312kB inactive_file:181216kB unevictable:0kB writepending:8840kB present:2095104kB managed:2014252kB mlocked:0kB bounce:0kB free_pcp:544kB local_pcp:0kB free_cma:142240kB [ +0.000018] lowmem_reserve[]: 0 0 0 0 [ +0.000015] DMA: 20626*4kB (UMEHC) 16291*8kB (UMEHC) 4462*16kB (UMHC) 1263*32kB (UMHC) 92*64kB (MHC) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 330528kB [ +0.000049] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ +0.000010] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=32768kB [ +0.000008] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ +0.000009] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=64kB [ +0.000008] 129311 total pagecache pages [ +0.000007] 52636 pages in swap cache [ +0.000006] Free swap = 11240188kB [ +0.000007] Total swap = 11792380kB [ +0.000006] 523776 pages RAM [ +0.000006] 0 pages HighMem/MovableOnly [ +0.000006] 20213 pages reserved [ +0.000007] 135168 pages cma reserved command used /usr/share/jellyfin-ffmpeg/ffmpeg -y -c:v hevc_rkmpp -i 2025-06-29_22-05_rec.ts -map v -c:v h264 _rkmpp -b:v 8000k -map 0:1 -c:a copy -t 300 dvbt2.h264.ts

I have a BananaPi M1, same SoC, same kernel (the upgrade went fine for me last week). Although I only use it for its SATA connector (big HDD as NBD, OS on SD-card), audio 3.5mm plug did work. Maybe I could see what mine does, but rather with a default Debian player, like mpv, works via ssh CLI. I can generate some FLAC file, but I don't want to search for the specific song/title. Also I could do a temp exfat on a loopdev or so or just play from NFS, Btrfs or Ext4 loopdev.

I know, but who cares if it is impractical IMO and various distros by default use that path. And RPiOS needs /boot/firmware, I myself use also /boot/uboot so I am more flexible w.r.t bootloaders. What is on boot FAT or in SPI-flash or tftpboot or else is secondary, the master (from package manager) is in rootfs. Read back my 1st message; there is option -p A snapshot is identified by its UUID, whether local or remote/other/foreign filesystem. So my backup tool/script figures that out. If an SD-card would be end of life or damaged scrub will notice it and then I regenerate the whole thing on a new SD-card from the latest transferred snapshot. I do not use /home for any meaningful data. All is on NAS. Some have no local storage except SPI for bootloader. I also treat all Linux computers the same, except those that are NAS or a clone of it. For rare Windows I indeed use 'imaging' although there is WinBTRFS.

I just mention the very basic, Btrfs has way to many features and options to make a generic image creating script. I see already 2 issues in the shrink-backup scrip, simple test: root@rock5b:~ # btrfs filesystem du -s --raw / Total Exclusive Set shared Filename ERROR: not a btrfs filesystem: /boot/efi WARNING: cannot access 'efi': Invalid argument WARNING: cannot access 'boot': Invalid argument ERROR: cannot check space of '/': Invalid argument This is on Opensuse Tumbleweed, but distro should not matter as I have a custom simple subvolume scheme. It is after 10+ years that I have a fully automated methods that just backup the structure I use in every Linux system so that it also can be 'played back' to a (sparse) image file or just raw device. I don't do any resizing, that is not essential to backup; I just re-create that same partition table (also backed up) so I have the same machine after a failed SD-card or HDD with too many bad sectors or just a clone template for a extra new SBC. I use a tool called snapper to create snapshots, both automatic and ad-hoc. It is also integrated in apt and zypper. So before the upgrade to pihole v6, a (read-only) snapshot is created. Also I was bothered by buggy v6, so I just take a read-write snapshot of the last know good/running read-only snapshot and set that that to default, then reboot, or even sync and power cycle. Of course that snapshot (UUID) is also as a differential created tree/folder on my NAS (in case the SD-card or so would fail). Long time ago installers created a lot of subvolumes, like for /home /var/log and many more. That indeed gets into big problems. Just use 1 for the root and snapshot that. That works fine for a workstation like machine with no real data stored locally (all is on NAS). Note that since kernel 6.1, btrfs send can send compressed data, so as it is stored on storage device. That saves time (and money) on mobile links for example. You can compress and decompress via rsync or ssh or vpn, but it costs extra battery energy. Also how long does "make atomic snapshots" take on a 4TB HDD of a subvolume of 2TB you think?

The foreign added SPI-flash makes it custom in terms of Armbian bootloader. Look how to build U-Boot from Armbian build environment (development basically) or find howto on denx.de. It been a while since I last did that myself, so things might have changed a bit.

That is because it is the linux kernel that is driving you own/custom SPI-flash chip. When you power on the board, it follows a certain priority of devices, see schematics of the board. It might be that SPI-flash is not included. And if included, the U-Boot you put in there might assume different wiring or none at all. It is can be because the U-Boot config it is compiled from does not include the right options. So create a custom U-Boot for your custom board.

@Geoffrey Schaller error is "No space left on device (28)", from rsync The github info mentions: Rsync WILL cross filesystem boundries, and you have Ubuntu snapd active it seems, so my guess is the script copies double; bot snapd image file and mounted content. mount command on your OPI3 will show that. -x option of rsync shall be used, e.g. test it with mkdir -p /mnt/backups/PiHole/test/ rsync -avx / /mnt/backups/PiHole/test/

check those files /usr/lib/systemd/system/armbian-resize-filesystem.service /usr/lib/armbian/armbian-resize-filesystem

# dpkg -L linux-u-boot-nanopi-r6c-current | grep platform_install.sh /usr/lib/u-boot/platform_install.sh that script writes the uboot binary

See https://opensource.rock-chips.com/wiki_Partitions The ROM in Rockchips read what is at sector 64 and executes that (if valid ARM code of course). Can be SD-card or eMMC, which priority depends on some resistors AFAIK, see schematics of a particular SBC. You can flash just the uboot to emmc and then that is loaded, that uboot understands PCIe/NVMe and FAT and Ext4 so kan load linux kernel and other files needed. uboot programs HW and kernel programs HW. They might not interact and conflict, is not uncommon for several SBCs.

Then also make sure what that is; I do not know exact HW and storage devices for OPi5Pro, but let's say wipe eMMC and SPI-flash first, so you are sure that only the U-Boot on the SD-card can be started. That one is easier to rewrite using another computer. OrangePi has virtually no support is my impression, just 1 image on googledrive then they go on to make a new SBC variant. But that 1 version might have good enough U-Boot so maybe rip that out the image and patch that into some own/Armbian image. Still no guarantee for success, but otherwise you need to develop/change DeviceTree and/or overlays yourself. I have a NanoPi-R6C, also RK3588S based, and it does well with 3 bootloader variants (FriendlyElec's, Armbian variants, EDK2-UEFI) for Armbian mainline based kernels.

@salas What U-Boot are you using?

Most SBC backup tools I have seen are based on rsync and they operate then on/with Ext4 filesystem. Is fine when you don't have complex servers and databases running. I am not sure about pihole, but I saw that by default it keeps a 1 year history so a database file of about 1GB. I am not sure what happens to integrity of such a file on the target if during copy/rsync the source file also changes. AFAIK databases have their ways to handle it, when power-loss or so, but not sure. If you want to be sure, use Btrfs as filesystem, then you can make atomic snapshots and use those for source of rsync or use: btrfs send -p <old snapshotnumber> <snapshot number>| ssh <remote_host> btrfs receive <backup folder>. That is the basic manual option. Complete tool is btrbk, it is standard in Debian repo, see https://digint.ch/btrbk/index.html docs of author. You still need some partition and bootloader handling I think. Or assume that a total crash is rare so only backup to NAS or so and reconstruct manually if you need a new SD-card or so or want to copy thing so eMMC of the OPi3.

This might be about PCI-E bifurcation not working or can also be something mixed up with other SERDES ports. ROCK3A just uses 1x PCIe3x2-lanes (for NVME) but the R5C 2x PCIe3x1-lanes. The second lane might not be enabled or the 1st 2.5GbE chip might get 2-lanes allocated. And/or any other mix-up, but less likely I think. If the USB3 ports work and also if you have a working M.2 E-key WiFi, it is likely that bifurcation not working. For my ROCK3A I have to use original Radxa U-Boot and Armbian vendor kernel 6.1.x, otherwise It doesn't do what I want. So you might use other U-Boot and other kernel. Or just the FriendlyElec OpenWrt based image, but that is not easy generic Debian Linux. For my R6C I got Armbian userspace/rootfs working with Armbian mainline based kernel was about 17 months ago. Also with Armbian vendor based kernel. I see in Armbian Bookworm with beta repo: # apt list | grep u-boot-nanopi-r5c linux-u-boot-nanopi-r5c-current/bookworm 25.8.0-trunk.244 arm64 linux-u-boot-nanopi-r5c-edge/bookworm 25.8.0-trunk.244 arm64 You can do similar grep for kernels. Install an alternative and see on which U-Boot it is based. Or maybe first pick an older image from archive. At least you must know what U-Boot and what kernel. For my ROCK3A i tried 3 U-Boot binaries and also 3 kernel flavor, so many combinations possible.

I am not sure if this line in the yaml file makes sense: all-wan-interfaces: # include interfaces that are renamed to 'wanX' by udev, e.g. nanopi-r1 There is usually only 1 WAN interface on routers. The print on my R6C is "WAN" and "LAN1". WAN is clearly the Rockchip GMAC. Actually I use it as 'LAN', as the only RJ45 cable. Only temporary I also use the PCI-E / 2.5GbE port while bridging with the GMAC, so I have a real 2.5GbE port as peer for my ROCK5B (just for speedtests). For the R5C, the Rockchip GMAC is not used. A rename from WAN to WAN1 I would refuse, but I do not use netplan.io, so not affected. I have the R6C now flashed with EDK2-UEFI and running Tumbleweed. 70-persistent-net.rules is empty, but Armbian 6.15.2-edge-rockchip64 kernel. I am not sure why the naming is still OK (wan and lan1, so my .nmconnection files keep working). With other kernel it gets end0 and enP3p49s0, I might use those eventually, ignoring what is printed on the case, depends how often I am going to unplug/plug network cables. For the R5C, the ports in the schematics are named "LAN1" and "LAN2", not sure how they map to what is printed on the case. They are each 1 lane of the PCIE3x2, that might be a problem for some kernel and/or bootloader.

Somehow the SD-card contents and SSD contents could be mixed-up or partly double. I do not know what armbian-update does, but what is your plan or what do you want? Should Armbian run from SD-card or SSD?

@StanleyLIM I see it should run a variant of OpenWrt, see https://www.friendlyelec.com/index.php?route=product/product&path=69&product_id=290 My NanoPi-R6C got delivered with that FriendlyWrt pre-installed on eMMC. I bought the R6C as alternative for RPi5, nice metal passive cooling and M.2 M-key as well. FriendlyWrt uses a bunch of partitions to store various objects related to the boot stage and although it would be nice for a router, I wanted generic Linux as base. But R6C is using RK3588S as SoC, yours RK3568B2. Based on experience with my ROCK3A (RK3568), you could first check if both NICs are available on PCI-E, 'sudo lspci' should show. If available, 'sudo ip link' should also show the 2 NICs as network ports. Then it is a matter of high-level networking, not really specific to Armbian.

Those use systemd-network and it is called 'minimal' so it might be that only 1 port is handled by default (the LAN only). Rest you need to figure out yourself. What do you want to do with the NanoPi-R5C ? I have replaced systemd-network with NetworkManager as is the default in Debian Bookworm once for a newly downloaded image (was for ROCK3A AFAIR). The Ubuntu based images use netplan.io, that is nice for Canonical, not for me. I do not want to waste my time on yet another configuring/scripting/layering. It made me just clone an existing Armbian/Debian/RPi Bookworm rootfs as I use NetworkManager for almost all Linux computers and is easy copying of .nmconnection files between platforms (x86, arm, etc).

I consider the btrfs issue 'solved', from past experience I know sometimes explicit balancing is needed for certain blockdevice level or lowlevel operations, like conversion of profiles, for example single into raid1, there must be enough unallocated space so new 1GB chunks can be created, especially on the originating device as it is all fundamentally copy-on-write so at least extra/double space for 1 system, meta, data chunk shall be available and I simply forgot to check/realize that. Further enhancements to Btrfs that this is done automatically somehow means likely that I first need to browse the kernel mailing list and see what people like David Sterba are doing nowadays. The CMA issue I suspect maybe it is related to power supply. Because of testing, I needed to take the NanoPi-R6C to another room where i used a 65W 'white-label branded' USB-C PD PSU that I got delivered with a refurbished laptop. That PSU has once shown under-voltage on an RPi4, so I expect too much ripple (when 5V). Besides the CMA issue, lateron also sudden hard reset of the NanoPi-R6C when I was doing networked backup, was 2x ethernet + HDMI + NVMe + eMMC active and that could mean >3A when only 5V. When I also took my RPI5 27W USB-C PD PSU to that room and used that, no strange issues anymore, so running fine at least with EDK2-UEFI and Armbian kernel 6.15.2-edge-rockchip64 and Bookworm or Tumbleweed userspace (and KDE Wayland). I have never seen PD working on the NanoPi-R6C and the powering circuit looks a lot like ROCK3A w.r.t. staged DC-DC conversion. And I use Armbian mainline based U-Boot or EDK2-UEFI, not the original FriendlyWrt installation with some 2017 based U-Boot. Also for that original one, I doubt that it does USB-C PD to request 9V or higher. I assume not, looking at: https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R6C#PD_Power_Adapters_We_Tested https://wiki.friendlyelec.com/wiki/index.php/Template:PDPowerWeTested There is also nothing about USB-C PD under https://www.friendlyelec.com/Forum/viewforum.php?f=81 so I assume: Great HW but SW/OS support is abandoned and DIY. That also means, same as for ROCK3A and ROCK5B, solder some 12V wire to a male USB-C connector and use that in order to be sure power won't drop too much when e.g. some USB3 HDD/SSD is inserted. Or limit cables to only RPI5 PSU and WAN/1GB ethernet (and still Samsung 970 NVMe, that works OK, even when 800% CPUload). Or check/see if PD handling is available in denx.de/mainline U-Boot.

Restarted with vendor kernel, then same actions: [ 580.015977] BTRFS info (device loop0p2): dev_replace from /dev/loop0p2 (devid 1) to /dev/loop1p2 started [ 599.501402] cma: cma_alloc: cma: alloc failed, req-size: 1024 pages, ret: -12 [ 599.502431] kworker/6:4: page allocation failure: order:10, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0 [ 599.502452] CPU: 6 PID: 239 Comm: kworker/6:4 Tainted: G W 6.1.115-vendor-rk35xx #1 [ 599.502457] Hardware name: FriendlyElec NanoPi R6C/NanoPi R6C, BIOS v1.1 04/09/2025 [ 599.502461] Workqueue: events atomic_pool_work_fn [ 599.502471] Call trace: [ 599.502474] dump_backtrace+0xf0/0x12c [ 599.502480] show_stack+0x20/0x30 [ 599.502484] dump_stack_lvl+0x7c/0xa0 [ 599.502489] dump_stack+0x18/0x34 [ 599.502493] warn_alloc+0xe0/0x17c [ 599.502498] __alloc_pages+0x524/0x854 [ 599.502501] atomic_pool_expand+0x8c/0x268 [ 599.502506] atomic_pool_resize+0x50/0x64 [ 599.502510] atomic_pool_work_fn+0x44/0x54 [ 599.502515] process_one_work+0x1c0/0x274 [ 599.502521] worker_thread+0x1dc/0x274 [ 599.502525] kthread+0xc4/0xd4 [ 599.502529] ret_from_fork+0x10/0x20 [ 599.502534] Mem-Info: [ 599.502537] active_anon:465 inactive_anon:13222 isolated_anon:0 active_file:37602 inactive_file:1830958 isolated_file:0 unevictable:4 dirty:1679 writeback:166667 slab_reclaimable:15924 slab_unreclaimable:27052 mapped:16566 shmem:499 pagetables:621 sec_pagetables:0 bounce:0 kernel_misc_reclaimable:0 free:12582 free_pcp:127 free_cma:0 [ 599.502546] Node 0 active_anon:1860kB inactive_anon:52888kB active_file:150408kB inactive_file:7323832kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:66264kB dirty:6716kB writeback:666668kB shmem:1996kB writeback_tmp:0kB kernel_stack:4416kB pagetables:2484kB sec_pagetables:0kB all_unreclaimable? no [ 599.502554] DMA free:24920kB boost:0kB min:5448kB low:9248kB high:13048kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:956kB inactive_file:3784292kB unevictable:0kB writepending:29608kB present:3929344kB managed:3825428kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 599.502561] lowmem_reserve[]: 0 0 3902 3902 [ 599.502567] Normal free:25408kB boost:0kB min:5712kB low:9696kB high:13680kB reserved_highatomic:2048KB active_anon:1860kB inactive_anon:52888kB active_file:149060kB inactive_file:3539556kB unevictable:16kB writepending:643228kB present:4194304kB managed:3996056kB mlocked:16kB bounce:0kB free_pcp:556kB local_pcp:0kB free_cma:0kB [ 599.502574] lowmem_reserve[]: 0 0 0 0 [ 599.502580] DMA: 97*4kB (UM) 103*8kB (UM) 95*16kB (UME) 101*32kB (UME) 61*64kB (UME) 22*128kB (UME) 7*256kB (U) 7*512kB (UME) 5*1024kB (UME) 1*2048kB (U) 0*4096kB = 25228kB [ 599.502601] Normal: 962*4kB (UME) 377*8kB (UME) 140*16kB (UMEH) 43*32kB (UMEH) 22*64kB (UME) 25*128kB (UME) 17*256kB (UM) 4*512kB (ME) 5*1024kB (UME) 0*2048kB 0*4096kB = 26608kB [ 599.502621] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ 599.502624] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=32768kB [ 599.502628] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 599.502631] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=64kB [ 599.502634] 1869148 total pagecache pages [ 599.502636] 0 pages in swap cache [ 599.502639] Free swap = 10873596kB [ 599.502641] Total swap = 10873852kB [ 599.502643] 2030912 pages RAM [ 599.502645] 0 pages HighMem/MovableOnly [ 599.502648] 75541 pages reserved [ 599.502650] 2048 pages cma reserved [ 624.681251] BTRFS info (device loop0p2): dev_replace from /dev/loop0p2 (devid 1) to /dev/loop1p2 finished So although it works, a rather strange side effect with CMA makes me feel that the system could be unstable now. I have seen CMA related errors before on RK35xx, but that was with U-Boot, this is with UEFI firmware, where I maybe forgot to change a setting or so. At least HDMI display is blank, serial console and ssh works.

OK, as I thought, not enough 'headroom' rock5b:/local/ssdata/nocow # btrfs inspect-internal list-chunks 2 btrfs inspect-internal list-chunks 2 Devid PNumber Type/profile PStart Length PEnd LNumber LStart Usage% ----- ------- ----------------- -------- --------- -------- ------- -------- ------ 1 1 System/single 1.00MiB 4.00MiB 5.00MiB 4 2.30GiB 0.39 1 2 Metadata/single 5.00MiB 8.00MiB 13.00MiB 1 5.00MiB 35.74 1 3 Data/single 13.00MiB 8.00MiB 21.00MiB 2 13.00MiB 100.00 1 4 Data/single 21.00MiB 1.00GiB 1.02GiB 3 21.00MiB 53.20 1 5 Metadata/single 1.02GiB 208.00MiB 1.22GiB 5 2.31GiB 23.51 1 6 Data/single 1.22GiB 512.00MiB 1.72GiB 6 2.51GiB 84.35 1 7 Data/single 1.72GiB 512.00MiB 2.22GiB 7 3.01GiB 86.48 1 8 Metadata/single 2.22GiB 256.00MiB 2.47GiB 8 3.51GiB 33.67 1 9 Metadata/single 2.47GiB 256.00MiB 2.72GiB 9 3.76GiB 14.43 1 10 Data/single 2.72GiB 1.00GiB 3.72GiB 10 4.01GiB 83.88 1 11 Data/single 3.72GiB 1.00GiB 4.72GiB 11 5.01GiB 97.23 1 12 Data/single 4.72GiB 1.00GiB 5.72GiB 12 6.01GiB 55.02 1 13 Data/single 5.72GiB 1.00GiB 6.72GiB 13 7.01GiB 85.21 1 14 Data/single 6.72GiB 1.00GiB 7.72GiB 14 8.01GiB 58.04 1 15 Data/single 7.72GiB 1.00GiB 8.72GiB 15 9.01GiB 82.03 1 16 Data/single 8.72GiB 1.00GiB 9.72GiB 16 10.01GiB 28.15 1 17 Metadata/single 9.72GiB 256.00MiB 9.97GiB 17 11.01GiB 10.30 1 18 Data/single 9.97GiB 1.00GiB 10.97GiB 18 11.26GiB 99.83 1 19 Metadata/single 10.97GiB 256.00MiB 11.22GiB 19 12.26GiB 0.58 1 20 Metadata/single 11.22GiB 256.00MiB 11.47GiB 20 12.51GiB 5.22 1 21 Metadata/single 11.47GiB 256.00MiB 11.72GiB 21 12.76GiB 0.49 1 22 Metadata/single 11.72GiB 256.00MiB 11.97GiB 22 13.01GiB 0.27 1 23 Metadata/single 11.97GiB 27.00MiB 12.00GiB 23 13.26GiB 0.06 Then doing various balance actions, then again replace action, see result below: [334356.515225] [ T207531] BTRFS info (device loop0p2): balance: start -mlimit=3 -slimit=3 [334356.516261] [ T207531] BTRFS info (device loop0p2): relocating block group 14236516352 flags metadata [334356.608520] [ T207531] BTRFS info (device loop0p2): found 1 extents, stage: move data extents [334356.655369] [ T207531] BTRFS info (device loop0p2): relocating block group 13968080896 flags metadata [334356.882397] [ T207531] BTRFS info (device loop0p2): found 44 extents, stage: move data extents [334356.974962] [ T207531] BTRFS info (device loop0p2): relocating block group 13699645440 flags metadata [334357.145812] [ T207531] BTRFS info (device loop0p2): found 77 extents, stage: move data extents [334357.223337] [ T207531] BTRFS info (device loop0p2): balance: ended with status: 0 [334402.346978] [ T207580] BTRFS info (device loop0p2): balance: start -mlimit=6 -slimit=6 [334402.347912] [ T207580] BTRFS info (device loop0p2): relocating block group 13431209984 flags metadata [334403.120645] [ T207580] BTRFS info (device loop0p2): found 841 extents, stage: move data extents [334403.372968] [ T207580] BTRFS info (device loop0p2): relocating block group 13162774528 flags metadata [334403.613848] [ T207580] BTRFS info (device loop0p2): found 94 extents, stage: move data extents [334403.743610] [ T207580] BTRFS info (device loop0p2): relocating block group 11820597248 flags metadata [334404.526435] [ T207580] BTRFS info (device loop0p2): found 1666 extents, stage: move data extents [334404.798554] [ T207580] BTRFS info (device loop0p2): relocating block group 4035969024 flags metadata [334406.059117] [ T207580] BTRFS info (device loop0p2): found 2363 extents, stage: move data extents [334406.374011] [ T207580] BTRFS info (device loop0p2): relocating block group 3767533568 flags metadata [334407.879438] [ T207580] BTRFS info (device loop0p2): found 5433 extents, stage: move data extents [334408.284506] [ T207580] BTRFS info (device loop0p2): relocating block group 2475687936 flags metadata [334413.884083] [ T207580] BTRFS info (device loop0p2): found 9463 extents, stage: move data extents [334414.690177] [ T207580] BTRFS info (device loop0p2): balance: ended with status: 0 [334530.694011] [ T207693] BTRFS info (device loop0p2): balance: start -dlimit=2 [334530.694839] [ T207693] BTRFS info (device loop0p2): relocating block group 12089032704 flags data [334534.205097] [ T207693] BTRFS info (device loop0p2): found 1897 extents, stage: move data extents [334534.491352] [ T207693] BTRFS info (device loop0p2): found 1897 extents, stage: update data pointers [334534.625126] [ T207693] BTRFS info (device loop0p2): relocating block group 10746855424 flags data [334536.605759] [ T207693] BTRFS info (device loop0p2): found 16316 extents, stage: move data extents [334537.056028] [ T207693] BTRFS info (device loop0p2): found 16316 extents, stage: update data pointers [334537.361629] [ T207693] BTRFS info (device loop0p2): balance: ended with status: 0 [334614.951029] [ T207803] BTRFS info (device loop0p2): dev_replace from /dev/loop0p2 (devid 1) to /dev/loop1p2 started [334654.572435] [ T207803] BTRFS info (device loop0p2): dev_replace from /dev/loop0p2 (devid 1) to /dev/loop1p2 finished So I would need to do the same with vendor kernel on same images, I made exact copies, so can be done, however starting with vendor kernel requires system reboot (and other bootloader), so not sure when I can do this.

I could not reproduce it on a vanilla OpenSuse Tumbleweed installation (EDK2-UEFI in SPI-flash on ROCK5B) when using a rather newly created filesystem that is only used for 15 minutes or so. However, I can reproduce if the image is my virtual machine armbian64 machine/test/template and between 1 to 2 years old and has had many operations, like resize shrink extend etc. So it seems some complex out-of-space issue for a certain part: rock5b:/local/ssdata/nocow # uname -a ; grep VERSION_ID /etc/os-release Linux rock5b 6.15.1-1-default #1 SMP PREEMPT_DYNAMIC Thu Jun 5 14:29:05 UTC 2025 (75961ad) aarch64 aarch64 aarch64 GNU/Linux VERSION_ID="20250610" - so depends on image as rather fresh image newly constructed serv.img no problem - -28 is no space error AFAIK, check internal of image rock5b:/local/ssdata/nocow # df | grep loop0p2 /dev/loop0p2 12582912 7904480 2852032 74% /local/ssdata/nocow/2 rock5b:/local/ssdata/nocow # btrfs fi df 2 Data, single: total=10.01GiB, used=7.29GiB System, single: total=4.00MiB, used=16.00KiB Metadata, single: total=1.99GiB, used=218.05MiB GlobalReserve, single: total=26.95MiB, used=0.00B rock5b:/local/ssdata/nocow # btrfs de us 2 /dev/loop0p2, ID: 1 Device size: 12.00GiB Device slack: 0.00B Data,single: 10.01GiB Metadata,single: 1.99GiB System,single: 4.00MiB Unallocated: 1.00MiB rock5b:/local/ssdata/nocow # btrfs fi us 2 Overall: Device size: 12.00GiB Device allocated: 12.00GiB Device unallocated: 1.00MiB Device missing: 0.00B Device slack: 0.00B Used: 7.51GiB Free (estimated): 2.72GiB (min: 2.72GiB) Free (statfs, df): 2.72GiB Data ratio: 1.00 Metadata ratio: 1.00 Global reserve: 26.95MiB (used: 0.00B) Multiple profiles: no Data,single: Size:10.01GiB, Used:7.29GiB (72.82%) /dev/loop0p2 10.01GiB Metadata,single: Size:1.99GiB, Used:229.44MiB (11.27%) /dev/loop0p2 1.99GiB System,single: Size:4.00MiB, Used:16.00KiB (0.39%) /dev/loop0p2 4.00MiB Unallocated: /dev/loop0p2 1.00MiB From 10+ years Btrfs usage and a quick look at those numbers, I think the problem occurs because it is not possible to easily/directly create a new chunk. The filesystem is still mounted. If I unmount, a new mount fails because of undefined state (replace ongoing but no mounted filesystem and target device has temp ID 0 and cannot be found ?). So will see if balancing first with the proper options gets it done.

I think now for the 3rd time this happens. OS is Armbian Bookworm with vendor ,current, edge kernels installed. it turns out that with vendor kenel there is no issue. But fails for mainline based rockchip64 kernels, at least edge. I am not sure about current (6.6 or 6.12) as I am not really using those. I discovered first time with 6.13.0-rc* kernel, I assumed something else might have been wrong, so more or les ignored and rebooted with vendor kernel. That happened both with nanopi-r6c and rock-3a. I also already rebooted the rock-3a on a later occasion just t make sure problem would not occur. Yesterday was on nanopi-r6c with kernel 6.15.1-edge-rockchip64. I booted with this kernel but Opensuse Tumbleweed userspace (VERSION_ID="20250610"). Following command run-as-root should move Armbian Bookworm rootfs from eMMC to NVMe: btrfs replace start /dev/mmcblk1p2 /dev/nvme0n1p2 /local/armbid Then in dmesg: Jun 19 18:44:38 ranc kernel: BTRFS info (device mmcblk1p2): dev_replace from /dev/mmcblk1p2 (devid 1) to /dev/nvme0n1p2 started Jun 19 18:45:49 ranc kernel: BTRFS warning (device mmcblk1p2): failed setting block group ro: -28 Jun 19 18:45:49 ranc kernel: BTRFS error (device mmcblk1p2): btrfs_scrub_dev(/dev/mmcblk1p2, 1, /dev/nvme0n1p2) failed -28 After 5 minutes or so I discovered that the filesystem was corrupted, would not open: open_ctree failed: -5 As I do (very) frequent backups (differential send|receive to NAS) recreate and play back the latest single subvolume snapshot is easy, but it would become quite a drama if large multi-TB with hundreds of snapshots/subvolumes. As the OS userspace was rolling release Tumbleweed, the btrfs-progs is very new, not the 6.2 from Bookworm. So it seems something mainline kernel rockchip64. I will try to further pinpoint. The NanoPi-R6C is a bit DUT ATM, but I could test as well with various other combinations, ROCK3A or ROCK5B, although those have a dedicated task/use-case in my home. The 5B has 16G RAM, so easy to run VMs and loopdevs etc. I think this is not HW related, but whos knows, maybe some detailed RK35xx optimization or so.