Jump to content

A try on utilizing H6 PCIe with "Virtualization"


Icenowy

Recommended Posts

As we know, the PCIe on H6 is buggy, which doesn't offer linear address, and Linux cannot support such kind of configuration.

 

However, the Cortex-A53 cores used by H6 supports virtualization, which can be used to change the order of the address space.

 

Recently, I tried to make use of virtualization to provide linear mapping of PCIe, and I succeed in making an Intel 6205 wireless card working.

 

The hypervisor code is at https://github.com/Icenowy/aw-el2-barebone . It's intended to start before U-Boot, and located at 0x40010000.

 

A U-Boot fork that is patched to load the hypervisor is at https://github.com/Icenowy/u-boot/tree/h6-load-hyp , and a kernel that utilizes the wrapped PCIe (and patched to reserve memory for the hypervisor) is at https://github.com/Icenowy/linux/tree/h6-pcie-wrapped .

 

In order to let the hypervisor start before U-Boot, BL31 needs to be built with `PRELOADED_BL33_BASE=0x40010000` in make parameter -- this will change the EL2 entrypoint to the hypervisor. Mainline ATF from ARM works.

 

Contributions to the hypervisor is welcomed.

 

(In addition, abusing virtualization in such way will prevent us from using KVM. But I think more people will want PCIe instead of KVM, right?)

Link to comment
Share on other sites

Hi @Icenowy,

 

Thanks for explanations and the dev is really nice :).

 

I have rebuilt from scratch my buildroot + adding PCIUTILS but rtw PCI drivers is still not working properly :/.

 

buildroot branch used: https://github.com/clementperon/buildroot/commits/beelink_gs1_pcie_wrapped

linux patches are taken from kernel https://github.com/clementperon/linux/commits/h6_pcie_wrapped

 

Spoiler

# dmesg | grep pci
[    0.260904] sunxi-pcie 5400000.pcie: failed to get pcie ref clk
[    0.299353] ehci-pci: EHCI PCI platform driver
[    0.314375] ohci-pci: OHCI PCI platform driver
[    0.654747] sunxi-pcie 5400000.pcie: host bridge /soc/pcie@5400000 ranges:
[    0.654786] sunxi-pcie 5400000.pcie:       IO 0x0005e00000..0x0005e0ffff -> 0x0000000000
[    0.654807] sunxi-pcie 5400000.pcie:      MEM 0x0005500000..0x0005cfffff -> 0x0005500000
[    0.754914] sunxi-pcie 5400000.pcie: Link up
[    0.956812] sunxi-pcie 5400000.pcie: Speed change timeout
[    0.956820] sunxi-pcie 5400000.pcie: PCI-e speed of Gen1
[    0.956951] sunxi-pcie 5400000.pcie: PCI host bridge to bus 0000:00
[    0.956963] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.956971] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.956980] pci_bus 0000:00: root bus resource [mem 0x05500000-0x05cfffff]
[    0.957013] pci 0000:00:00.0: [16c3:abcd] type 01 class 0x060400
[    0.957064] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    0.957121] pci 0000:00:00.0: supports D1
[    0.957129] pci 0000:00:00.0: PME# supported from D0 D1 D3hot D3cold
[    0.958985] pci 0000:01:00.0: [10ec:b822] type 00 class 0x028000
[    0.959212] pci 0000:01:00.0: reg 0x10: [io  0x0000-0x00ff]
[    0.959384] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit]
[    0.960203] pci 0000:01:00.0: supports D1 D2
[    0.960211] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.970569] pci 0000:00:00.0: BAR 14: assigned [mem 0x05500000-0x055fffff]
[    0.970580] pci 0000:00:00.0: BAR 6: assigned [mem 0x05600000-0x0560ffff pref]
[    0.970591] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    0.970605] pci 0000:01:00.0: BAR 2: assigned [mem 0x05500000-0x0550ffff 64bit]
[    0.970698] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    0.970731] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    0.970740] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.970749] pci 0000:00:00.0:   bridge window [mem 0x05500000-0x055fffff]
[    0.971039] pcieport 0000:00:00.0: PME: Signaling with IRQ 248
[    0.971294] pcieport 0000:00:00.0: AER: enabled with IRQ 248
[    4.546202] rtw_pci 0000:01:00.0: enabling device (0000 -> 0003)
[    4.559316] rtw_pci 0000:01:00.0: Firmware version 27.2.0, H2C version 13
[    4.587927] rtw_pci 0000:01:00.0: error beacon valid
[    4.592921] rtw_pci 0000:01:00.0: failed to download rsvd page
[    4.598793] rtw_pci 0000:01:00.0: failed to download firmware
[    4.604762] rtw_pci 0000:01:00.0: failed to setup chip efuse info
[    4.610867] rtw_pci 0000:01:00.0: failed to setup chip information
[    4.620690] rtw_pci: probe of 0000:01:00.0 failed with error -16

 

 

Output of lspci -vv

Spoiler

 


# lspci -vv
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 248
	Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
	I/O behind bridge: 00001000-00001fff [size=4K]
	Memory behind bridge: 05500000-055fffff [size=1M]
	Prefetchable memory behind bridge: None
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	[virtual] Expansion ROM at 05600000 [disabled] [size=64K]
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000bab09000  Data: 0000
	Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn+ NFERptEn+ FERptEn+
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Kernel driver in use: pcieport

01:00.0 Network controller: Realtek Semiconductor Co., Ltd. Device b822
	Subsystem: Realtek Semiconductor Co., Ltd. Device b822
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 31
	Region 0: I/O ports at 1000 [size=256]
	Region 2: Memory at 05500000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Via message/WAKE#
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 0e, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [148 v1] Device Serial Number 00-e0-4c-ff-fe-b8-22-01
	Capabilities: [158 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [160 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-

		L1SubCtl2:

 

 

 

I have also try to rebuild from my previous work  (your old PCIe stash  + AW dirty patches) but can't make it works neither :'(.

https://github.com/clementperon/linux/commits/h6_pcie

 

Just sayin' that because I'm now convinced the issue is coming from my setup and not from the virtualization layer.

 

Regards,

Clement

Edited by Clément Peron
Link to comment
Share on other sites

Interestingly the wireless card also uses IO space...

 

could you try to add `num-viewport = <4>;` to the PCIe device node and remove calls for dw_pcie_prog_outbound_atu() in sunxi_pcie_host_init() ? (although I don't know whether this will work...)

 

Link to comment
Share on other sites

  • Igor pinned this topic
On 6/20/2020 at 11:35 AM, ingamedeo said:

Confirmed working on Orange Pi 3 based on Allwinner H6, kernel 5.4.7 with pcie-wrapped. No particular changes needed apart from new dts.

Link here: https://github.com/ingamedeo/orangepi3-h6-mainline

This may be a silly question, but I'm new to compiling linux. I'm guessing you're supposed to compile this kernel to add to a new build image for install on the OPI3. Or is this to add to an existing OPI3 image (i.e updating the kernels and uboot on an already installed armbian build). Even some initial guidance for further research on my part would be appreciated.

Link to comment
Share on other sites

6 hours ago, Nick Bishop said:

This may be a silly question, but I'm new to compiling linux. I'm guessing you're supposed to compile this kernel to add to a new build image for install on the OPI3. Or is this to add to an existing OPI3 image (i.e updating the kernels and uboot on an already installed armbian build). Even some initial guidance for further research on my part would be appreciated.

To be honest if the given explanations through the link are not enough you should keep considering PCIe broken for now and wait until somebody upstreams her work.

Link to comment
Share on other sites

hello guys,

Thank you for putting the time and the work on this. You are awesome.

 

I was wondering when do you think this will be available in a stable branch (or unstable for that matter, since AFAIK it is not implemented on any) since I have spent quite some time trying to cross-compile the kernel and install it in my armbian image unsuccesfully. 

 

Thanks a lot!

 

 

Link to comment
Share on other sites

On 8/16/2020 at 4:38 AM, sine said:

I was wondering when do you think this will be available in a stable branch (or unstable for that matter, since AFAIK it is not implemented on any) since I have spent quite some time trying to cross-compile the kernel and install it in my armbian image unsuccesfully. 

 

Probably never - as this is a bit of a security concern... 

 

There's a reason why there are EL levels in ARM-V8A, and bringing in support for things under the kernel are a bit of a concern.

 

If Mainline picks this up, perhaps, but I suggest that this be kept as a science project - most folks don't need PCI-e support on hobby boards

Link to comment
Share on other sites

On 3/28/2020 at 10:39 PM, Icenowy said:

However, the Cortex-A53 cores used by H6 supports virtualization, which can be used to change the order of the address space.

 

Recently, I tried to make use of virtualization to provide linear mapping of PCIe, and I succeed in making an Intel 6205 wireless card working.

 

The hypervisor code is at https://github.com/Icenowy/aw-el2-barebone . It's intended to start before U-Boot, and located at 0x40010000.

 

A U-Boot fork that is patched to load the hypervisor is at https://github.com/Icenowy/u-boot/tree/h6-load-hyp , and a kernel that utilizes the wrapped PCIe (and patched to reserve memory for the hypervisor) is at https://github.com/Icenowy/linux/tree/h6-pcie-wrapped .

 

In order to let the hypervisor start before U-Boot, BL31 needs to be built with `PRELOADED_BL33_BASE=0x40010000` in make parameter -- this will change the EL2 entrypoint to the hypervisor. Mainline ATF from ARM works.

 

As mentioned above - interesting - but outside of wrapping PCI-e addresses - this is a security concern

Link to comment
Share on other sites

  • Igor unpinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines