Icenowy Posted March 29, 2020 Posted March 29, 2020 As we know, the PCIe on H6 is buggy, which doesn't offer linear address, and Linux cannot support such kind of configuration. However, the Cortex-A53 cores used by H6 supports virtualization, which can be used to change the order of the address space. Recently, I tried to make use of virtualization to provide linear mapping of PCIe, and I succeed in making an Intel 6205 wireless card working. The hypervisor code is at https://github.com/Icenowy/aw-el2-barebone . It's intended to start before U-Boot, and located at 0x40010000. A U-Boot fork that is patched to load the hypervisor is at https://github.com/Icenowy/u-boot/tree/h6-load-hyp , and a kernel that utilizes the wrapped PCIe (and patched to reserve memory for the hypervisor) is at https://github.com/Icenowy/linux/tree/h6-pcie-wrapped . In order to let the hypervisor start before U-Boot, BL31 needs to be built with `PRELOADED_BL33_BASE=0x40010000` in make parameter -- this will change the EL2 entrypoint to the hypervisor. Mainline ATF from ARM works. Contributions to the hypervisor is welcomed. (In addition, abusing virtualization in such way will prevent us from using KVM. But I think more people will want PCIe instead of KVM, right?) 4 Quote
Icenowy Posted March 29, 2020 Author Posted March 29, 2020 Currently it's tested on Pine H64 model A. More tests, e.g. on OPi 3 is welcomed. (And if a PR can be sent, it's better :-) ) 0 Quote
Clément Peron Posted March 29, 2020 Posted March 29, 2020 (edited) Hi @Icenowy, Thanks for explanations and the dev is really nice :). I have rebuilt from scratch my buildroot + adding PCIUTILS but rtw PCI drivers is still not working properly :/. buildroot branch used: https://github.com/clementperon/buildroot/commits/beelink_gs1_pcie_wrapped linux patches are taken from kernel : https://github.com/clementperon/linux/commits/h6_pcie_wrapped Spoiler # dmesg | grep pci [ 0.260904] sunxi-pcie 5400000.pcie: failed to get pcie ref clk [ 0.299353] ehci-pci: EHCI PCI platform driver [ 0.314375] ohci-pci: OHCI PCI platform driver [ 0.654747] sunxi-pcie 5400000.pcie: host bridge /soc/pcie@5400000 ranges: [ 0.654786] sunxi-pcie 5400000.pcie: IO 0x0005e00000..0x0005e0ffff -> 0x0000000000 [ 0.654807] sunxi-pcie 5400000.pcie: MEM 0x0005500000..0x0005cfffff -> 0x0005500000 [ 0.754914] sunxi-pcie 5400000.pcie: Link up [ 0.956812] sunxi-pcie 5400000.pcie: Speed change timeout [ 0.956820] sunxi-pcie 5400000.pcie: PCI-e speed of Gen1 [ 0.956951] sunxi-pcie 5400000.pcie: PCI host bridge to bus 0000:00 [ 0.956963] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.956971] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] [ 0.956980] pci_bus 0000:00: root bus resource [mem 0x05500000-0x05cfffff] [ 0.957013] pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400 [ 0.957064] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref] [ 0.957121] pci 0000:00:00.0: supports D1 [ 0.957129] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold [ 0.958985] pci 0000:01:00.0: [10ec:b822] type 00 class 0x028000 [ 0.959212] pci 0000:01:00.0: reg 0x10: [io 0x0000-0x00ff] [ 0.959384] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit] [ 0.960203] pci 0000:01:00.0: supports D1 D2 [ 0.960211] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.970569] pci 0000:00:00.0: BAR 14: assigned [mem 0x05500000-0x055fffff] [ 0.970580] pci 0000:00:00.0: BAR 6: assigned [mem 0x05600000-0x0560ffff pref] [ 0.970591] pci 0000:00:00.0: BAR 13: assigned [io 0x1000-0x1fff] [ 0.970605] pci 0000:01:00.0: BAR 2: assigned [mem 0x05500000-0x0550ffff 64bit] [ 0.970698] pci 0000:01:00.0: BAR 0: assigned [io 0x1000-0x10ff] [ 0.970731] pci 0000:00:00.0: PCI bridge to [bus 01-ff] [ 0.970740] pci 0000:00:00.0: bridge window [io 0x1000-0x1fff] [ 0.970749] pci 0000:00:00.0: bridge window [mem 0x05500000-0x055fffff] [ 0.971039] pcieport 0000:00:00.0: PME: Signaling with IRQ 248 [ 0.971294] pcieport 0000:00:00.0: AER: enabled with IRQ 248 [ 4.546202] rtw_pci 0000:01:00.0: enabling device (0000 -> 0003) [ 4.559316] rtw_pci 0000:01:00.0: Firmware version 27.2.0, H2C version 13 [ 4.587927] rtw_pci 0000:01:00.0: error beacon valid [ 4.592921] rtw_pci 0000:01:00.0: failed to download rsvd page [ 4.598793] rtw_pci 0000:01:00.0: failed to download firmware [ 4.604762] rtw_pci 0000:01:00.0: failed to setup chip efuse info [ 4.610867] rtw_pci 0000:01:00.0: failed to setup chip information [ 4.620690] rtw_pci: probe of 0000:01:00.0 failed with error -16 Output of lspci -vv Spoiler # lspci -vv 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 248 Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0 I/O behind bridge: 00001000-00001fff [size=4K] Memory behind bridge: 05500000-055fffff [size=1M] Prefetchable memory behind bridge: None Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- [virtual] Expansion ROM at 05600000 [disabled] [size=64K] BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000bab09000 Data: 0000 Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd- AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- AtomicOpsCtl: ReqEn- EgressBlck- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 RootCmd: CERptEn+ NFERptEn+ FERptEn+ RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd- FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0 ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000 Kernel driver in use: pcieport 01:00.0 Network controller: Realtek Semiconductor Co., Ltd. Device b822 Subsystem: Realtek Semiconductor Co., Ltd. Device b822 Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 31 Region 0: I/O ports at 1000 [size=256] Region 2: Memory at 05500000 (64-bit, non-prefetchable) [size=64K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [70] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Via message/WAKE# AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled AtomicOpsCtl: ReqEn- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 0e, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [148 v1] Device Serial Number 00-e0-4c-ff-fe-b8-22-01 Capabilities: [158 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Capabilities: [160 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates- L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1SubCtl2: I have also try to rebuild from my previous work (your old PCIe stash + AW dirty patches) but can't make it works neither :'(. https://github.com/clementperon/linux/commits/h6_pcie Just sayin' that because I'm now convinced the issue is coming from my setup and not from the virtualization layer. Regards, Clement Edited March 29, 2020 by Clément Peron 0 Quote
Icenowy Posted March 30, 2020 Author Posted March 30, 2020 Interestingly the wireless card also uses IO space... could you try to add `num-viewport = <4>;` to the PCIe device node and remove calls for dw_pcie_prog_outbound_atu() in sunxi_pcie_host_init() ? (although I don't know whether this will work...) 0 Quote
Icenowy Posted March 30, 2020 Author Posted March 30, 2020 oh I don't think the previous word will work. RTW88 driver only uses the BAR with ID 2, which is the memory space, not IO space. 0 Quote
Icenowy Posted March 30, 2020 Author Posted March 30, 2020 `SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612]` is tested to work. 2 Quote
Bharat Gohil Posted April 6, 2020 Posted April 6, 2020 Hi, Booted Xen on Orange Pi Lite2 which is based upon H6. want to try PCI pass-through so guest can access, anyone has tried PCI pass-through on Xen,KVM or other hypervisor? @icenowy: will try your hypervisor code on Orange Pi Lite2. Thanks, Bharat 0 Quote
Bharat Gohil Posted April 11, 2020 Posted April 11, 2020 Hi Icenowy, I am able to run your code in PineH64 and OrangePi Lite2. able to run Linux in EL1 Want to know, do you tried running Linux as guest and access PCI address space from Linux guest on any H6 board? Regards, Bharat 0 Quote
ingamedeo Posted June 20, 2020 Posted June 20, 2020 Confirmed working on Orange Pi 3 based on Allwinner H6, kernel 5.4.7 with pcie-wrapped. No particular changes needed apart from new dts. Link here: https://github.com/ingamedeo/orangepi3-h6-mainline 3 Quote
Nick Bishop Posted July 29, 2020 Posted July 29, 2020 On 6/20/2020 at 11:35 AM, ingamedeo said: Confirmed working on Orange Pi 3 based on Allwinner H6, kernel 5.4.7 with pcie-wrapped. No particular changes needed apart from new dts. Link here: https://github.com/ingamedeo/orangepi3-h6-mainline This may be a silly question, but I'm new to compiling linux. I'm guessing you're supposed to compile this kernel to add to a new build image for install on the OPI3. Or is this to add to an existing OPI3 image (i.e updating the kernels and uboot on an already installed armbian build). Even some initial guidance for further research on my part would be appreciated. 0 Quote
Werner Posted July 30, 2020 Posted July 30, 2020 6 hours ago, Nick Bishop said: This may be a silly question, but I'm new to compiling linux. I'm guessing you're supposed to compile this kernel to add to a new build image for install on the OPI3. Or is this to add to an existing OPI3 image (i.e updating the kernels and uboot on an already installed armbian build). Even some initial guidance for further research on my part would be appreciated. To be honest if the given explanations through the link are not enough you should keep considering PCIe broken for now and wait until somebody upstreams her work. 0 Quote
sine Posted August 16, 2020 Posted August 16, 2020 hello guys, Thank you for putting the time and the work on this. You are awesome. I was wondering when do you think this will be available in a stable branch (or unstable for that matter, since AFAIK it is not implemented on any) since I have spent quite some time trying to cross-compile the kernel and install it in my armbian image unsuccesfully. Thanks a lot! 0 Quote
sfx2000 Posted October 30, 2020 Posted October 30, 2020 On 8/16/2020 at 4:38 AM, sine said: I was wondering when do you think this will be available in a stable branch (or unstable for that matter, since AFAIK it is not implemented on any) since I have spent quite some time trying to cross-compile the kernel and install it in my armbian image unsuccesfully. Probably never - as this is a bit of a security concern... There's a reason why there are EL levels in ARM-V8A, and bringing in support for things under the kernel are a bit of a concern. If Mainline picks this up, perhaps, but I suggest that this be kept as a science project - most folks don't need PCI-e support on hobby boards 0 Quote
sfx2000 Posted October 30, 2020 Posted October 30, 2020 On 3/28/2020 at 10:39 PM, Icenowy said: However, the Cortex-A53 cores used by H6 supports virtualization, which can be used to change the order of the address space. Recently, I tried to make use of virtualization to provide linear mapping of PCIe, and I succeed in making an Intel 6205 wireless card working. The hypervisor code is at https://github.com/Icenowy/aw-el2-barebone . It's intended to start before U-Boot, and located at 0x40010000. A U-Boot fork that is patched to load the hypervisor is at https://github.com/Icenowy/u-boot/tree/h6-load-hyp , and a kernel that utilizes the wrapped PCIe (and patched to reserve memory for the hypervisor) is at https://github.com/Icenowy/linux/tree/h6-pcie-wrapped . In order to let the hypervisor start before U-Boot, BL31 needs to be built with `PRELOADED_BL33_BASE=0x40010000` in make parameter -- this will change the EL2 entrypoint to the hypervisor. Mainline ATF from ARM works. As mentioned above - interesting - but outside of wrapping PCI-e addresses - this is a security concern 0 Quote
Frank.L Posted June 18, 2021 Posted June 18, 2021 On 3/30/2020 at 7:00 PM, Icenowy said: `SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612]` is tested to work. where is the code or dts ? 0 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.