Add Support for RK3399 (2xA72+4xA53)

[tkaiser]

Can anyone running a recent kernel please provide /proc/cpuinfo contents or even better output from 'sudo armbianmonitor -u' when running an image based on Armbian? I'm interested in differentiation strategies for the two CPU clusters.

[firepower]

RK3399 SDK for Geekbuying MECOOL Development Board RK3399  it is the same as  VideoStrong RK3399

[Jason Fisher]

I'm looking for a RK3399/RK3328 for a battery-powered mobile NAS/music player briefcase build.  I want 4GB RAM for possibly using ZFS and replacing general home NAS functions.

 

https://www.amazon.com/RK3399-Android-BT4-0-USB3-0-TYPE-C/dp/B071G53B1Z

 

$135 with free shipping for 4GB RAM/32GB eMMC RK3399 model -- seems not a bad deal considering accessories.  Powered by 5v 2a but I would replace that with 5v 4a for stability I think.  

 

Is there a reason to pay $199 for the Firefly RK3399 4GB model over this?  

 

I will be initially testing USB3 with a JMicron-based Mediasonic ProBox-type enclosure, which supports eSATA PMP and USB3 JBOD -- I have two 8-bay models that have worked well with ZFS for 8? years: started with MacZFS/Hackintosh, now archlinux, 2x raidz2, 5 Seagate+2 WD+1 HGST drive failure in that time.   Using Linux since 1.2.6 so no issues tinkering/helping where needed ..

Edited September 21, 2017 by Jason Fisher

[tkaiser]

On 31.8.2017 at 9:06 PM, tkaiser said:

Can anyone running a recent kernel please provide /proc/cpuinfo contents or even better output from 'sudo armbianmonitor -u' when running an image based on Armbian? I'm interested in differentiation strategies for the two CPU clusters.

 

Answered by someone else in the meantime: https://www.cnx-software.com/2017/09/25/checking-out-debian-and-linux-sdk-for-videostrong-vs-rd-rk3399-board/#comment-546250

[Stuart Naylor]

On 9/21/2017 at 9:18 PM, Jason Fisher said:

I'm looking for a RK3399/RK3328 for a battery-powered mobile NAS/music player briefcase build.  I want 4GB RAM for possibly using ZFS and replacing general home NAS functions.

I would have a look at a Rock64 or oDroid almost a quarter of the RK3399 price.

It would be great for someone to do a really good RK3399 dev board as it has 4 PCIe 2.1 lanes that seem untapped apart from sole use for LTE.
I think the Firefly has a 2xSata - PCIe optional card dunno about videostrong as the SDK and software offerings are supposedly flakey at best.

As a SoC the RK3399 is pretty amazeballs and sure its price will drop as more qty's are pushed, not sure about the dev boards though and don't think anyone has really cracked it.
There are 2 RK3399 tv-boxes that do have Sata VorkeZ3 & Nagrace NT-V9 that are much cheaper than the dev boards but still pretty dear and best of luck with a linux image for them.
 
 

[tkaiser]

Just some links / thoughts since I don't feel too comfortable jumping in a thread where ZFS on toys is starting to be discussed:

• https://forum.armbian.com/index.php?/topic/3946-rk3399-orange/
• https://forum.armbian.com/index.php?/topic/3953-preview-generate-omv-images-for-sbc-with-armbian/#comment-32340
• When thinking about wasting disks for redundancy at least I would also start to care about data integrity (and buy a HP N54L on eBay instead of playing with large USB attached storage pools, I both don't like bit rot and single points of failure that much)
• When playing with ZFS on the Clearfog Pro in the meantime after multiplying /sys/module/zfs/parameters/zfs_dirty_data_max by 4 I get +100MB/s write and +250MB/s read with a RAIDZ consisting of 4 SSD. Accessing this zpool when using Ethernet in parallel the board freezes, too lazy to look into yet. But at least based on memory requirements setting on at least ZoL 0.7 seems the way to go: https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.0
• If I would want to attach more than a few disks in a reliable way to a somewhat affordable ARM board my choice would be the Helios4 (2GB ECC DRAM, 4 native SATA ports which are all able to be paired with quality Marvell FBS -- FIS-based switching -- SATA port multipliers)
• But if performance is also a concern I would wait a few weeks/months and grab one of the new Denverton mainboards that support QAT (QuickAssist technology) to be paired with ZoL 0.7 or above

 

BTW: I think the probability is high that at least the VideoStrong thingie does not expose PCIe at all and the mPCIe slot there is just useable with those WWAN LTE modems (mPCIe formfactor but only utilizing power and USB data lines on pins 36/38 of the mPCIe connector)

[Stuart Naylor]

1 hour ago, tkaiser said:

buy a HP N54L

http://www.ebay.co.uk/itm/HP-Proliant-N54L-Microserver-/272860806947 2nd user one that will prob accept an offer of £80

[Jason Fisher]

Appreciate the responses.  Going with the Rock64 for now due to the support available.  That N54L looks really interesting.

 

Tuning ARC/cache will help -- I have been successfully dual-booting Windows/Linux with ZFS and arch, using a VM in Windows to access/share ZFS back with good enough performance for my needs (~100MB/sec, Plex server) on a 2.5GB RAM instance after tuning ARC and a few other things, so I think it's going to come down to the stability of the USB3 chipset. (or SATA FIS)  https://wiki.freebsd.org/ZFSTuningGuide

 

The Firefly RK3399 has a PCIe M.2 to SATA board available with two ports on an ASMedia ASM1061 card which I believe supports FIS -- http://shop.t-firefly.com/goods.php?id=52 -- but once you figure additional time needed on the RK3399 to get things going currently, it seems like it's much more expensive than simply $199 vs ~$60..

 

I will be experimenting with a few setups on the storage end, but ultimately this will be a portable NAS/Plex/mobile office server briefcase boombox build using a $20 Lepai amplifier, powered by LiFePO4 batteries (https://www.amazon.com/dp/B01M4P35Z9 ), charged by solar (https://www.amazon.com/dp/B074G1CN6N ) and can be used as speakers/media server for a portable DLP/Android projector.  Moving off-grid/tiny cabin next year and downsizing/combining functionality.  A second briefcase will be an LTE receiver/repeater/WiFi hotspot that can be positioned independently where there is a better line to a tower -- LTE with the Firefly seems interesting but need to confirm that it will work with Verizon and Sprint. 

 

Something like this for the form factor:

 

 

[tkaiser]

37 minutes ago, Jason Fisher said:

The Firefly RK3399 has a PCIe M.2 to SATA board available with two ports on an ASMedia ASM1061 card which I believe supports FIS

 

It seems it's really an ASM1061 (using just a single PCIe 2.x lane) since my ASM1062 is listed as that:

root@clearfogpro:~# lspci
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04)
01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
02:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)

So we have a SoC with PCIe 2.1 x4 in the spec sheet, one manufacturer selling an x1 SATA card and zero performance numbers about this setup. Since I've been fooled by specifiations already way too often (best example: the shitty 'native SATA' performance of Allwinner SATA equipped boards like Banana Pis) and since I doubt a tablet SoC like RK3399 has been optimized for PCIe performance at least I'll stay with the devices that were designed for these use cases and where we know that performance is superiour.  

 

 

[Stuart Naylor]

24 minutes ago, tkaiser said:

fooled by specifiations already way too often (best example: the shitty 'native SATA' performance of Allwinner SATA equipped boards like Banana Pis

You where not fooled for too long its just like the Odroid HC1 where its just a crappy usb3.0 to Sata bridge with banana's being off USB2.0

It is native and actually the RK3399 is interesting if someone actually goes beyond the Google/Rockchip reference design.
The dev boards are practically the same as the chromebook versions.
The reviews on the chromebooks in terms of performance for Arm are extremely good, but you could be right and how optimised those 4 5Gbs lanes are who knows.
Anyone got a firefly and tried the sata expansion board and post any ioperf?

Maybe those lanes are already taken with the raft of connectors on the 'reference' board?

It would be interesting to get some real world figures on this thing.

[zador.blood.stained]

14 minutes ago, Stuart Naylor said:

with banana's being off USB2.0

On A10, A20 and R40 SATA is native, it just has low performance (espacially low write performance).

[tkaiser]

1 hour ago, Stuart Naylor said:

You where not fooled for too long its just like the Odroid HC1 where its just a crappy usb3.0 to Sata bridge with banana's being off USB2.0

 

No, please check https://forum.armbian.com/index.php?/topic/1925-some-storage-benchmarks-on-sbcs/&do=findComment&comment=34192

 

 

USB3 is there on more recent ARM SoCs since they can license USB3 IP blocks now and have not to pay too much for this any more. Same with PCIe, it's there, you can connect stuff to it, it allows the IDH who do the job to be creative and add peripherals they couldn't in the past due to not enough high speed buses but that's it. A TV box SoC like RK3328 shows surprisignly an insanely great USB3 performance (and RK3399 will do the same since [1]) but why should this be the same with PCIe on a tablet SoC like the RK3399?

 

Look at the link above and i.MX6:

• Native SATA II: native transfer rate of 3.0 Gbit/s that, when accounted for the 8b/10b encoding scheme, equals to the maximum uncoded transfer rate of 2.4 Gbit/s (300 MB/s). In reality it's a third we get. But SATA II is an improvement over SATA I (limited to 150 MB/s) since it defined NCQ so workloads with many small disk accesses could benefit greatly from this feature (never tested in detail since i.MX6 is boring slow for my use cases)
• Native Gigabit Ethernet: Standard talks about a 1Gbit/s signal rate and all we get with i.MX6 when using the internal GbE implementation are ~400Mbits/sec at the IP layer and not ~940 MBits/sec as it should be. Only using the single PCIe 2.0 lane and a good PCIe GbE adapter exceeding 900 Mbits/sec is possible (never tested in detail since i.MX6 is boring slow for my use cases)
• Native PCIe 2.x: This standard talks about 5 GT/s per lane again with 8b/10b encoding applied so maximum bandwidth would be 500 MB/s not counting any protocol overhead and funnily leaving out all performance issues that are related to (high) latency issues. The result: lowest performance again (check the Hummingboard numbers).

There's a SoC made for a specific market, there are some lowspeed and some highspeed interfaces connected to and there are some specifications talking about signal or line rates. This is all not related to real-world performance. An EspressoBin with a dual-core SoC running at just 800 MHz will outperform any tablet or phone SoC when it's about network and/or storage. Since 'made for the job'. I'm not that stupid (any more) to expect from an RK3399 with its PCIe 2.x x4 interface anything near the theoretical 2000MB/s (and focusing on bandwidth only is already stupid like hell anyway  ).

 

[1] Talking to an RK engineer few months ago about USB3 issues with ROCK64: 'still confused about this issue, because I test the USB3 drive on my RK3399 SDK (use the same xHCI host controller as RK3328 but different USB3 PHY), and I am surprised to find that it works well on RK3399.'

[Stuart Naylor]

@tkaiser Dunno all bananas & oranges to me and may it was the orangepi's that has a sata that really was just like the HC1 which was just a USB adapter in chip form on board.

I remember your benchmarks and thinking yeap pants for sata.

There is a single PCIe2.1 port on the RK3399 and I presume its a x4 hence the 4 lanes.

The RK3399 does have supposedly a 2.1 x4 and there is only one way we will find out.
"PCIe 2.1 (4 full-duplex lanes )" as the specs quote.

Firefly used a x2 cost? I just dunno but was wondering if anyone had tested. Doh nope they split it into 2 ports one for m.2 and the other for LTE.
So its got about 400Gb/s max raw as its PCIe1.0x2

So maybe your right as either its cost or it just can not do it.

[tkaiser]

9 hours ago, Stuart Naylor said:

Doh nope they split it into 2 ports one for m.2 and the other for LTE.
So its got about 400Gb/s max raw as its PCIe1.0x2

 

Huh? I really don't know what you're talking about (PCIe 1.0? Why? 400Gb/s? What?).

 

If you refer to the Firefly Kickstarter page this is just an insane BS collection. I only know that they use an M.2 B key slot which is able to transport a couple of totally different protocols (see here). M.2 B key can either be used for PCIe 2.x x2 or SATA or USB3 (or maybe it's not 'or' but 'and' instead and the highspeed data lines are different ones allowing more than one such a protocol at the same time). Mentioning this theoretical SATA capability by Firefly marketing is of course misleading since RK3399 has no SATA capabilities and for SATA being usable on the M.2 slot there's missing something: a PCIe SATA controller needing another PCIe lane sitting somewhere on the Firefly around (as usual Kickstarter marketing was successful in fooling a lot of users trying out M.2 SATA which simply can not work on this board).

 

Then there's a slot that looks like mPCIe but the problem with this thing is that it can be used with 4 totally different types of cards:

• mSATA: this allows a directly connected SATA device (not on the Firefly of course)
• mPCIe: a single PCIe 2.x lane exposed
• USB 2: only power and pins 36/38 of the mPCIe connector are used and the device is USB only (but will often be marketed as 'PCIe device' -- see almost all such 'WWAN modems')
• PCIe+USB2: Since PCIe data lines and USB2 data lines are separate (unlike mSATA vs. mPCIe where they are shared) cards could be used that use both PCIe and USB2

What's that thing on the Firefly doing? At least I neither know nor care (but if only 'LTE' is mentioned it could be USB2 only!).

 

RK3399 is a great SoC for many use cases (attach displays, attach cameras, attach USB peripherals and do with it for what it's designed for: consuming media) but unless someone provides real performance numbers when trying to combine the SoC with reasonable storage controllers I think it's better to ignore this potential use case here. And 'providing real performance numbers' is IMO not that easy since needing the necessary equipment and skills (experts in embedded area are usually missing)

 

[chrisf]

According to the Firefly schematic, the M2-B connector on the top has the 4 PCIe lanes, USB 2.0 and I2C connected

The mPCIe connector on the bottom has only USB 2.0 connected

 

[tkaiser]

36 minutes ago, chrisf said:

According to the Firefly schematic, the M2-B connector on the top has the 4 PCIe lanes, USB 2.0 and I2C connected

The mPCIe connector on the bottom has only USB 2.0 connected

 

Interesting, thanks. But doesn't change much wrt 'marketing BS' spread by their Kickstarter campaign since everything connector related there is just misleading (and to make use of x4 PCIe you would need M.2 M key anyway, B is limited to x2). So we have two PCIe lanes wasted on the Firefly, most probably 4 on the VideoStrong, no idea about the Xunlong board (there sits an ASM106x around with just one SATA port connected so it would be somewhat strange to waste 2 lanes with an ASM1062 but who knows? Xunlong provides also a mPCIe slot and since distance is far away from the ASM106x I would believe that's mPCIe with another PCIe lane used) and all those RK3399 TV boxes will ignore PCIe for sure.

 

Maybe the only RK3399 device making full use of all 4 PCIe lanes is Theobroma's RK3399-Q7 (combined with the carrier board that provides even a real PCIe slot). I still doubt that performance/price ratio justifies looking at RK3399 when talking about storage/NAS use cases (at least as long as 'USB3-to-SATA' is not considered to be an option).

[tkaiser]

2 hours ago, tkaiser said:

and to make use of x4 PCIe you would need M.2 M key anyway, B is limited to x2

 

And another nice example where you can look at pictures or inside the 'technical documentation' and get not the slightest idea what's possible:

 

 

The Banana folks let someone design another board for them, this time based on RTD1296 (which should be a way better choice for storage/NAS use cases than RK3399 since 'made for the job').

 

Three times M.2 on this PCB. On the top PCB side there's 2 times M.2 E key (allows for various protocols including PCIe 2.x x2) one time labeled as 'PCIe 2.0' (or on the detail page 'PCIe 2.2' which doesn't exist... but hey, it's SinoVoip who simply don't give a shit about providing correct information) and one time PCIe 1.1/SDIO. What about lanes used? Is it PCIe 2.0 x2 or x1 (hey, just a difference between 500 MB/s vs. 1000 MB/s). On the bottom side that looks like B key so theoretically capable of 'PCIe x2, SATA, USB 2.0 and 3.0, audio, UIM, HSIC, SSIC, I2C and SMBus'. It's labeled 'M.2' which is plain stupid since telling NOTHING but at least SIM slot nearby and the marking suggest that this is USB2 only to insert WWAN modems (which usually appear as a few devices on the USB bus)

[Stuart Naylor]

Yeah they have added a few new boards have you seen this one?
https://www.aliexpress.com/store/product/Newest-arrive-Banana-PI-BPI-R2-Opensource-Router/302756_32823351577.html?spm=2114.12010608.0.0.412779ceyHOvoD
http://www.banana-pi.org/r2.html

https://www.cnx-software.com/2017/01/03/banana-pi-bpi-r2-router-board-powered-by-mediatek-mt7623a-quad-core-processor-comes-with-5-gbe-ports-sata-and-more/

https://www.phoronix.com/scan.php?page=news_item&px=ARM-Hardware-Linux-4.14 do you think we might actually get working software?

https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.14-rc2-Released

Still slightly bemused as the RTD1296 is just a STB with Sata and still struggle to see the differentiation 

[Jason Fisher]

I think my needs/goals with the ROCK64 or an RK3399 would ultimately fit into this list:

 

- Stability

 

- Fast enough storage to saturate gigabit ethernet/WiFi

 

- Two ports with port multiplier support (prefer SATA/SATA via PCI Express, but two USB 3.0 is ~OK)

 

- Low enough latency to satisfy user experience (large git repo, traversing filesystem normally, general responsiveness) -- i.e. prefer to avoid overhead of USB3-to-SATA and feel this affects the experience more than the raw throughput possible

 

- ZFS -- I have tested a dozen PCI/PCI Express SATA/USB3/motherboard controllers and many years of ZFS implementations, good and bad, completely abused these two arrays and would never willingly go back to MDRAID or trust a single drive

 

- ~5 watt or less SBC idle with USB device attached (not counting power of drives/external controller)

 

I am currently at around 160MB/sec average or so on two 8x spinning disk arrays currently connected via an old Silicon Image 3132 PCI Express x1 controller - rated for 2.5 Gbps .. it has been the most stable controller combo for the JMicron USB3/SATA combo port multiplier chipset built into the array chassis.  USB3 (yes, 8 drives over 1x USB 3.0) has actually been more stable over the years depending on the USB3 controller used -- in terms of dropouts and performance consistency, but the latency on file operations for that performance is consistently abysmal..

 

I don't see much of a use case here for the high end of SSD throughput .. unless you are editing 4k video.  RED Raw 4k is 162MB/sec at 30fps.  But I think that will have to wait a generation here.  But of course there are other advantages .. I suppose I would prefer an SSD that was 2x the size and 4x slower max throughput for the power advantages alone.

 

It would be interesting to put an additional (n-tier?) caching file system layer over attached storage that caches to eMMC.

[Jason Fisher]

If we're looking to test controller capabilities only and eliminate bottlenecks .. maybe someone can expense one of these .. http://techreport.com/review/16255/acard-ans-9010-serial-ata-ram-disk

[pbies]

@tkaiser cpuinfo for Orange Pi RK3399:

 

processor	: 0
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 48.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 1
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 48.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 2
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 48.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 3
model name	: ARMv8 Processor rev 4 (v8l)
BogoMIPS	: 48.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 4
model name	: ARMv8 Processor rev 2 (v8l)
BogoMIPS	: 48.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 2

processor	: 5
model name	: ARMv8 Processor rev 2 (v8l)
BogoMIPS	: 48.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 2

Serial		: 0000000000000000

 

[tkaiser]

4 hours ago, pbies said:

cpuinfo for Orange Pi RK3399

 

Thank you but Jean-Luc from cnxsoft answered my question already one year ago: https://forum.armbian.com/topic/4854-add-support-for-rk3399-2xa724xa53/?do=findComment&comment=40088

 

RK3399 is RK3399, the only things that matter wrt board support are $someone providing a proper hardware description (device-tree these days) and in extreme cases special boot BLOBs (for DRAM initialization and such stuff). If this $someone is neither SoC nor board vendor (as in the case of Orange Pi RK3399) unfortunately you're on your own.

 

None of the developers ever got an Orange Pi RK3399 dev sample and in the meantime even some people spread the rumour this board will be discontinued (neither know nor care, I never understood the OPi RK3399 concept)