Orange Pi 5 plus RK3588 IOMMU SMMU VFIO PCI passthrough

Ilya A · June 22, 2023

Hello!

Can anyone comment whether PCIE Passthrough is even theoretically possible on an Orange Pi 5+? Or should I just cease and desist?

#1) The hardware seems to be there

#2) IOMMU seems to be working at least for platform/PHP devices

#3) VFIO looks to be on

#4) no cigar

The missing piece looks to be MMU600_PCIE support, as non-platform devices don't seem to be getting assigned iommu groups?

---

#1
"There are two MMU600 instances in RK3588, MMU600_PCIE and MMU600_PHP respectively "

https://github.com/FanX-Tek/rk3588-TRM-and-Datasheet/blob/master/Rockchip RK3588 TRM V1.0-Part1-20220309.pdf

#2

root@oppa:~# dmesg | grep -i iommu
[    4.106039] iommu: Default domain type: Translated
[    4.106116] rk_iommu fdab9000.iommu: version = 2
[    4.106520] rk_iommu fdb50800.iommu: version = 2
[    4.106772] rk_iommu fdb60f00.iommu: version = 2
[    4.106972] rk_iommu fdb70f00.iommu: version = 2
[    4.107142] rk_iommu fdb90480.iommu: version = 2
[    4.107330] rk_iommu fdba0800.iommu: version = 2
[    4.107517] rk_iommu fdba4800.iommu: version = 2
[    4.107706] rk_iommu fdba8800.iommu: version = 2
[    4.107893] rk_iommu fdbac800.iommu: version = 2
[    4.108080] rk_iommu fdbb0800.iommu: version = 2
[    4.108282] rk_iommu fdbdf000.iommu: version = 2
[    4.108568] rk_iommu fdbef000.iommu: version = 2
[    4.108871] rk_iommu fdc38700.iommu: version = 2
[    4.109162] rk_iommu fdc48700.iommu: version = 2
[    4.109386] rk_iommu fdcb7f00.iommu: version = 2
[    4.109567] rk_iommu fdce0800.iommu: version = 2
[    4.109682] rk_iommu fdd97e00.iommu: version = 2
[    5.095442] mpp_vdpu2 fdb50400.vdpu: Adding to iommu group 1
[    5.096226] mpp_vepu2 fdb50000.vepu: Adding to iommu group 1
[    5.096639] mpp_vepu2 fdba0000.jpege-core: Adding to iommu group 5
[    5.097143] mpp_vepu2 fdba4000.jpege-core: Adding to iommu group 6
[    5.097610] mpp_vepu2 fdba8000.jpege-core: Adding to iommu group 7
[    5.098073] mpp_vepu2 fdbac000.jpege-core: Adding to iommu group 8
[    5.098725] mpp-iep2 fdbb0000.iep: Adding to iommu group 9
[    5.099329] mpp_jpgdec fdb90000.jpegd: Adding to iommu group 4
[    5.100281] mpp_rkvdec2 fdc38100.rkvdec-core: Adding to iommu group 12
[    5.101031] mpp_rkvdec2 fdc48100.rkvdec-core: Adding to iommu group 13
[    5.101989] mpp_rkvenc2 fdbd0000.rkvenc-core: Adding to iommu group 10
[    5.102543] mpp_rkvenc2 fdbe0000.rkvenc-core: Adding to iommu group 11
[    5.103910] mpp_av1dec av1d-master: av1_iommu_of_xlate,784
[    5.103927] av1_iommu_probe_device,736, consumer : av1d-master, supplier : fdca0000.iommu
[    5.103934] mpp_av1dec av1d-master: Adding to iommu group 17
[    5.120930] rockchip-vop2 fdd90000.vop: Adding to iommu group 16
[    5.578177] rkcifhw fdce0000.rkcif: Adding to iommu group 15
[    5.579738] rkisp_hw fdcb0000.rkisp: Adding to iommu group 14
[    5.739969] rga3_core0 fdb60000.rga: Adding to iommu group 2
[    5.740438] rga3_core1 fdb70000.rga: Adding to iommu group 3
[    5.741091] rga2_mmu: IOMMU binding successfully, default mapping core[0x1]
[    5.783349] RKNPU fdab0000.npu: Adding to iommu group 0
[    5.783493] RKNPU fdab0000.npu: RKNPU: rknpu iommu is enabled, using iommu mode
for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort
/sys/kernel/iommu_groups/0/devices/fdab0000.npu
/sys/kernel/iommu_groups/1/devices/fdb50000.vepu
/sys/kernel/iommu_groups/1/devices/fdb50400.vdpu
/sys/kernel/iommu_groups/2/devices/fdb60000.rga
/sys/kernel/iommu_groups/3/devices/fdb70000.rga
/sys/kernel/iommu_groups/4/devices/fdb90000.jpegd
/sys/kernel/iommu_groups/5/devices/fdba0000.jpege-core
/sys/kernel/iommu_groups/6/devices/fdba4000.jpege-core
/sys/kernel/iommu_groups/7/devices/fdba8000.jpege-core
/sys/kernel/iommu_groups/8/devices/fdbac000.jpege-core
/sys/kernel/iommu_groups/9/devices/fdbb0000.iep
/sys/kernel/iommu_groups/10/devices/fdbd0000.rkvenc-core
/sys/kernel/iommu_groups/11/devices/fdbe0000.rkvenc-core
/sys/kernel/iommu_groups/12/devices/fdc38100.rkvdec-core
/sys/kernel/iommu_groups/13/devices/fdc48100.rkvdec-core
/sys/kernel/iommu_groups/14/devices/fdcb0000.rkisp
/sys/kernel/iommu_groups/15/devices/fdce0000.rkcif
/sys/kernel/iommu_groups/16/devices/fdd90000.vop
/sys/kernel/iommu_groups/17/devices/av1d-master

#3

dmesg | grep -i vfio
[    5.183453] VFIO - User Level meta-driver version: 0.3
root@oppa:~# cat /lib/modules/$(uname -r)/modules.builtin | grep vfio
kernel/drivers/vfio/vfio.ko
kernel/drivers/vfio/vfio_virqfd.ko
kernel/drivers/vfio/vfio_iommu_type1.ko
kernel/drivers/vfio/pci/vfio-pci.ko
kernel/drivers/vfio/platform/vfio-platform.ko
kernel/drivers/vfio/platform/vfio-platform-base.ko
kernel/drivers/vfio/mdev/mdev.ko
kernel/drivers/vfio/mdev/vfio_mdev.ko

#lspci

lspci -nn -D | grep -i 79
0002:21:00.0 Network controller [0280]: MEDIATEK Corp. Device [14c3:7961]
lspci -vnk -s 0002:21:00.0
0002:21:00.0 0280: 14c3:7961
        Subsystem: 1a3b:4680
        Flags: fast devsel
        Memory at 980000000 (64-bit, prefetchable) [disabled] [size=1M]
        Memory at 980100000 (64-bit, prefetchable) [disabled] [size=16K]
        Memory at 980104000 (64-bit, prefetchable) [disabled] [size=4K]
        Capabilities: [80] Express Endpoint, MSI 00
        Capabilities: [e0] MSI: Enable- Count=1/32 Maskable+ 64bit+
        Capabilities: [f8] Power Management version 3
        Capabilities: [100] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
        Capabilities: [108] Latency Tolerance Reporting
        Capabilities: [110] L1 PM Substates
        Capabilities: [200] Advanced Error Reporting