3 3
Icenowy

research A try on utilizing H6 PCIe with "Virtualization"

Recommended Posts

As we know, the PCIe on H6 is buggy, which doesn't offer linear address, and Linux cannot support such kind of configuration.

 

However, the Cortex-A53 cores used by H6 supports virtualization, which can be used to change the order of the address space.

 

Recently, I tried to make use of virtualization to provide linear mapping of PCIe, and I succeed in making an Intel 6205 wireless card working.

 

The hypervisor code is at https://github.com/Icenowy/aw-el2-barebone . It's intended to start before U-Boot, and located at 0x40010000.

 

A U-Boot fork that is patched to load the hypervisor is at https://github.com/Icenowy/u-boot/tree/h6-load-hyp , and a kernel that utilizes the wrapped PCIe (and patched to reserve memory for the hypervisor) is at https://github.com/Icenowy/linux/tree/h6-pcie-wrapped .

 

In order to let the hypervisor start before U-Boot, BL31 needs to be built with `PRELOADED_BL33_BASE=0x40010000` in make parameter -- this will change the EL2 entrypoint to the hypervisor. Mainline ATF from ARM works.

 

Contributions to the hypervisor is welcomed.

 

(In addition, abusing virtualization in such way will prevent us from using KVM. But I think more people will want PCIe instead of KVM, right?)

Share this post


Link to post
Share on other sites
Want Armbian t-shirt or a cup?

Currently it's tested on Pine H64 model A.

 

More tests, e.g. on OPi 3 is welcomed. (And if a PR can be sent, it's better :-) )

Share this post


Link to post
Share on other sites
(edited)

Hi @Icenowy,

 

Thanks for explanations and the dev is really nice :).

 

I have rebuilt from scratch my buildroot + adding PCIUTILS but rtw PCI drivers is still not working properly :/.

 

buildroot branch used: https://github.com/clementperon/buildroot/commits/beelink_gs1_pcie_wrapped

linux patches are taken from kernel https://github.com/clementperon/linux/commits/h6_pcie_wrapped

 

Spoiler

# dmesg | grep pci
[    0.260904] sunxi-pcie 5400000.pcie: failed to get pcie ref clk
[    0.299353] ehci-pci: EHCI PCI platform driver
[    0.314375] ohci-pci: OHCI PCI platform driver
[    0.654747] sunxi-pcie 5400000.pcie: host bridge /soc/pcie@5400000 ranges:
[    0.654786] sunxi-pcie 5400000.pcie:       IO 0x0005e00000..0x0005e0ffff -> 0x0000000000
[    0.654807] sunxi-pcie 5400000.pcie:      MEM 0x0005500000..0x0005cfffff -> 0x0005500000
[    0.754914] sunxi-pcie 5400000.pcie: Link up
[    0.956812] sunxi-pcie 5400000.pcie: Speed change timeout
[    0.956820] sunxi-pcie 5400000.pcie: PCI-e speed of Gen1
[    0.956951] sunxi-pcie 5400000.pcie: PCI host bridge to bus 0000:00
[    0.956963] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.956971] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.956980] pci_bus 0000:00: root bus resource [mem 0x05500000-0x05cfffff]
[    0.957013] pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400
[    0.957064] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    0.957121] pci 0000:00:00.0: supports D1
[    0.957129] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold
[    0.958985] pci 0000:01:00.0: [10ec:b822] type 00 class 0x028000
[    0.959212] pci 0000:01:00.0: reg 0x10: [io  0x0000-0x00ff]
[    0.959384] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit]
[    0.960203] pci 0000:01:00.0: supports D1 D2
[    0.960211] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.970569] pci 0000:00:00.0: BAR 14: assigned [mem 0x05500000-0x055fffff]
[    0.970580] pci 0000:00:00.0: BAR 6: assigned [mem 0x05600000-0x0560ffff pref]
[    0.970591] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    0.970605] pci 0000:01:00.0: BAR 2: assigned [mem 0x05500000-0x0550ffff 64bit]
[    0.970698] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    0.970731] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    0.970740] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.970749] pci 0000:00:00.0:   bridge window [mem 0x05500000-0x055fffff]
[    0.971039] pcieport 0000:00:00.0: PME: Signaling with IRQ 248
[    0.971294] pcieport 0000:00:00.0: AER: enabled with IRQ 248
[    4.546202] rtw_pci 0000:01:00.0: enabling device (0000 -> 0003)
[    4.559316] rtw_pci 0000:01:00.0: Firmware version 27.2.0, H2C version 13
[    4.587927] rtw_pci 0000:01:00.0: error beacon valid
[    4.592921] rtw_pci 0000:01:00.0: failed to download rsvd page
[    4.598793] rtw_pci 0000:01:00.0: failed to download firmware
[    4.604762] rtw_pci 0000:01:00.0: failed to setup chip efuse info
[    4.610867] rtw_pci 0000:01:00.0: failed to setup chip information
[    4.620690] rtw_pci: probe of 0000:01:00.0 failed with error -16

 

 

Output of lspci -vv

Spoiler

 


# lspci -vv
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 248
	Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
	I/O behind bridge: 00001000-00001fff [size=4K]
	Memory behind bridge: 05500000-055fffff [size=1M]
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	[virtual] Expansion ROM at 05600000 [disabled] [size=64K]
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000bab09000  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn+ NFERptEn+ FERptEn+
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Kernel driver in use: pcieport

01:00.0 Network controller: Realtek Semiconductor Co., Ltd. Device b822
	Subsystem: Realtek Semiconductor Co., Ltd. Device b822
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 31
	Region 0: I/O ports at 1000 [size=256]
	Region 2: Memory at 05500000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Via message/WAKE#
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 0e, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [148 v1] Device Serial Number 00-e0-4c-ff-fe-b8-22-01
	Capabilities: [158 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [160 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-

		L1SubCtl2:

 

 

 

I have also try to rebuild from my previous work  (your old PCIe stash  + AW dirty patches) but can't make it works neither :'(.

https://github.com/clementperon/linux/commits/h6_pcie

 

Just sayin' that because I'm now convinced the issue is coming from my setup and not from the virtualization layer.

 

Regards,

Clement

Edited by Clément Peron

Share this post


Link to post
Share on other sites

Interestingly the wireless card also uses IO space...

 

could you try to add `num-viewport = <4>;` to the PCIe device node and remove calls for dw_pcie_prog_outbound_atu() in sunxi_pcie_host_init() ? (although I don't know whether this will work...)

 

Share this post


Link to post
Share on other sites

oh I don't think the previous word will work.

 

RTW88 driver only uses the BAR with ID 2, which is the memory space, not IO space.

Share this post


Link to post
Share on other sites

`SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612]` is tested to work.

Share this post


Link to post
Share on other sites

Hi,

 

Booted Xen on Orange Pi Lite2 which is based upon H6.

want to try PCI pass-through so guest can access, anyone has tried PCI pass-through on Xen,KVM or other hypervisor?

 

@icenowy: will try your hypervisor code on Orange Pi Lite2.

 

Thanks,

Bharat

Share this post


Link to post
Share on other sites

Hi Icenowy,

 

I am able to run your code in PineH64 and OrangePi Lite2. able to run Linux in EL1

Want to know, do you tried running Linux as guest and access PCI address space from Linux guest on any H6 board?

 

Regards,

Bharat

Share this post


Link to post
Share on other sites
On 6/20/2020 at 11:35 AM, ingamedeo said:

Confirmed working on Orange Pi 3 based on Allwinner H6, kernel 5.4.7 with pcie-wrapped. No particular changes needed apart from new dts.

Link here: https://github.com/ingamedeo/orangepi3-h6-mainline

This may be a silly question, but I'm new to compiling linux. I'm guessing you're supposed to compile this kernel to add to a new build image for install on the OPI3. Or is this to add to an existing OPI3 image (i.e updating the kernels and uboot on an already installed armbian build). Even some initial guidance for further research on my part would be appreciated.

Share this post


Link to post
Share on other sites
6 hours ago, Nick Bishop said:

This may be a silly question, but I'm new to compiling linux. I'm guessing you're supposed to compile this kernel to add to a new build image for install on the OPI3. Or is this to add to an existing OPI3 image (i.e updating the kernels and uboot on an already installed armbian build). Even some initial guidance for further research on my part would be appreciated.

To be honest if the given explanations through the link are not enough you should keep considering PCIe broken for now and wait until somebody upstreams her work.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
3 3