tkaiser

Latest RK3399 arrival in the lab. For now just some q&d photographs:
 



 
@wtarreau my first 96boards thing so far (just like you I felt the standard being directed towards nowhere given that there's no Ethernet). And guess what: 2 x Ethernet here!
 
A quick preliminary specifications list:
RK3399 (performing identical to any other RK3399 thingy out there as long as no throttling happens) 2 GB DDR3 RAM (in April Vamrs said they will provide 1GB, 2GB and 4GB variants for $99, $129 and $149) Board size is the standard 160×120 mm 96Boards EE form factor. EE = Enterprise Edition, for details download 96Boards-EE-Specification.pdf (1.1MB) Full size x16 PCIe slot as per EE specs (of course only x4 usable since RK3399 only provides 4 lanes at Gen2 speed) Board can be powered with 12V through barrel plug, 4-pin ATX plug or via pin header (Vamrs sent a 12V/4A PSU with the board) Serial console available via Micro USB (there's an onboard FTDI chip) 2 SATA ports + 2 SATA power ports (5V/12V). SATA is provided by a JMS561 USB3 SATA bridge that can operate in some silly RAID modes or PM mode (with spinning rust this chip is totally sufficient -- for SSDs better use NVMe/PCIe) Socketed eMMC and mechanical SD card adapter available (Vamrs sent also a SanDisk 8GB eMMC module as can be seen on the pictures) SIM card slot on the lower PCB side to be combined with an USB based WWAN modem in the mPCIe slot (USB2 only) 1 x SD card slot routed to RK3399, 1 x SD card slot for the BMC (Baseboard Management Controller) Gigabit Ethernet and separate Fast Ethernet port for the BMC Ampak AP6354 (dual-band and dual-antenna WiFi + BT 4.1) USB-C port with USB3 SuperSpeed and DisplayPort available eDP and HDMI 2.0 USB2 on pin headers and 2 type A receptacles all behind an internal USB2 hub USB3 on one pin header and 2 type A receptacles all behind an internal USB3 hub 96boards Low Speed Expansion connector with various interfaces exposed 96boards High Speed Expansion connector with various interfaces exposed (e.g. the 2nd USB2 host port, see diagram below) S/PDIF audio out 'real' on/off switch to cut power. To really power on the board the translucent button next to it needs to be pressed  

Count the patches and search for NEO2 there: https://github.com/armbian/build/tree/master/patch/kernel/sunxi-next

-And I haven't even begun yet.
 
I think it's important to understand the details on how the boot-process works. Once you have this knowledge, you'll know much better how to get things working if something gets messed up.
 
The CPU has a small on-chip boot-ROM. In this boot-ROM, there's code that cannot be altered.
This code is the code that checks the 3 jumper connections; usually these are set to load the boot-loader from SPI NOR-Flash.
If there's something messed up, you'll not get the "Marvell >>" prompt, but instead you'll get a single "> " - still, don't worry if that happens, because there is a way out.
When the boot-loader is 'trusted-firmware'-verified, then it is loaded (usually from SPI NOR-Flash as mentioned earlier, but could also be from SATA, UART or eMMC), and finally this boot-loader is executed. The boot-ROM code has now done its job.
 
The bootloader that was just loaded and executed is usually Uboot (it could be anything; even your own code).
This bootloader is the one that presents you with the "Marvell >>" prompt and allows you to interrupt the boot-process by pressing for instance space or return.
If the bootloader is not interrupted by a keypress within the timeout (usually 2 seconds), then it issues one single command automatically:
run bootcmd
-This 'bootcmd' is nothing but an environment variable. It contains a string that is executed by Uboot's command-interpreter.
You can issue this command to see what's in the variable:
 
printenv bootcmd
 
My 'bootcmd' looks like this:
run boot_armbian
 
-Yours might be different.
So another environment variable is being executed by the command-interpreter.
That environment variable holds instructions on ...
1: Get netboot images - just in case we're netbooting via PXE
2: Setup boot parameters
3: Probe the block devices, in order to find the most likely block-device to boot from (such as MicroSD-card/USB/SATA)
   Probing is usually done by the 'test' command. A boot-interface and boot-device is chosen (those are usually kept in environment variables)
   The boot-interface could for instance be "scsi" if you're booting from SATA or "mmc" if you're booting from a MicroSD card
4: Load the kernel image and emergency image (usually done by the ext4load command)
5: Execute kernel image (done by the 'iboot' command).
 
When step 5 is executed, Uboot finished its task and the kernel takes over.
If something goes wrong, the emergency disk image is brought up.
 
 
... If you at some point are at the "Marvell >>" prompt and want to play around, here's a few things you can try out:
 
If you have a SATA disk connected:
scsi scan; scsi dev 0:1
ext4ls scsi 0:1 /
 
If you have a MicroSD card inserted:
ext4ls mmc 0:1 /
 
If you have a USB block-device attached:
ext4ls usb 0:1 /
ext4ls usb 1:1 /
 
The number before the colon is the device number, the number after the colon is the partition.
-So if you have your rootfs on partition 12 on your SATA drive, then you could for instance ...
ext4ls scsi 0:14 /bin
... to see some of the executable files on that partition.
 
Note: 'scsi scan' probes the SATA interface; you will not be able to do anything useful with the device without issuing that command.
 
You can also set some environment variables if you wish to:
setenv myVariable "echo hello there"
printenv myVariable
echo $myVariable
run myVariable
 
You should be able to execute the boot instructions one-by-one until you reach 'iboot'.
 
Let's assume that you've found out how to make your board boot; you've written down all the commands necessary and tested that they indeed boot if you write them exactly as you have them ready (hopefully you have a terminal with copy-and-paste).
When you've got your commands tested, you can set the environment variables and save those variables to your SPI-Flash using this command:
saveenv
Wait for the prompt to return. Make sure you can type on the command-line (thus you will know that the writing is done).
After that, you can just boot as usual (either by typing 'boot' or 'reset' or by pressing the reset button).
-Do *not* press the reset button while the flash-memory is being written to; that will surely mess up things.
 
At this point, you may have tried some of the above; I expect that you're likely stuck somewhere, if so, please let me know where, so we can get you unstuck.

Nope, lazy mode won (thanks to parted and resize2fs now using the eMMC's full capacity, then did do-release-upgrade to upgrade Vamrs' Xenial image to 18.04 LTS, then quickly checked partition 5 whether latest DT changes are present -- they are).  Then I took two SATA SSDs and one NVMe SSD (in Pine's PCIe to M.2 adapter), threw everything in a corner, connected the stuff and fired it up again:
 

 
The huge fan is there to generate sbc-bench numbers now almost without throttling. Numbers here: http://ix.io/1nVS -- performance (not so) surprisingly identical to all other RK3399 thingies around but DDR3 memory results in slightly lower memory performance (negligible with almost every use case) and of course those RK3399 boards where we allowed 2000/1500 MHz so far show ~5% better CPU performance. The better cpuminer scores compared to RockPro64 with same 4.4. kernel here are just due to testing with Bionic on Ficus (GCC 7.3) vs. Debian Stretch there (GCC 6.3 -- with some tasks simply updating GCC you end up with whopping performance improvements for free)
 
Not able to test the PCIe SSD since 'rockchip-pcie f8000000.pcie: PCIe link training gen1 timeout!' (full dmesg output). I guess there's still some work needed to get DT bits correctly for PCIe? Ok, let's forget about PCIe now (there won't be anything new anyway since PCIe lives inside RK3399 so as long as trace routing on the PCB is ok this RK3399 will perform 100% identical to every other RK3399 thingy out there)
 
Now let's check USB situation. I'm a bit excited to have a look at the USB3 SATA implementation here since on the PCB itself there's a JMS561 controller (USB3-to-SATA bridge with support for 2 SATA ports, providing some silly RAID functionality but also PM mode. PM --> port multiplier). The two SSDs when connected with the JMS561 in PM mode appear as one device on the USB bus:
root@rock960:~# lsusb Bus 004 Device 003: ID 1058:0a10 Western Digital Technologies, Inc. Bus 004 Device 002: ID 05e3:0616 Genesys Logic, Inc. hub Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 003 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 005 Device 003: ID 0403:6010 Future Technology Devices International, Ltd FT2232C Dual USB-UART/FIFO IC Bus 005 Device 002: ID 1a40:0201 Terminus Technology Inc. FE 2.1 7-port Hub Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub root@rock960:~# lsusb -t /: Bus 06.Port 1: Dev 1, Class=root_hub, Driver=ohci-platform/1p, 12M /: Bus 05.Port 1: Dev 1, Class=root_hub, Driver=ehci-platform/1p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/7p, 480M |__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=ftdi_sio, 480M |__ Port 2: Dev 3, If 1, Class=Vendor Specific Class, Driver=ftdi_sio, 480M /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M |__ Port 3: Dev 3, If 0, Class=Mass Storage, Driver=uas, 5000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ohci-platform/1p, 12M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-platform/1p, 480M The JMS561 appears as '1058:0a10 Western Digital Technologies, Inc' device which is an indication that the chip runs with WD branded firmware (I'm not that excited). This chipset can be found in a lot of dual drive external USB3 thingies from various drive manufacturers but I thought we would see it here running with original JMicron firmware There's a Genesys Logic USB3 hub on the board connected to RK3399's USB3 host port (the hub appears twice, once with product ID 0616 -- that's SuperSpeed most people call USB3 -- and once with ID 0610 -- that's Hi-Speed or incorrectly called 'USB2') The JMS561 is behind the internal hub We also have a 7-port Terminus Technology USB2 hub on the PCB One FTDI UART adapter is internally connected to this hub (no idea what's that for -- maybe access to the 'MCU' that can be added to the board?) I then used a 4th SSD in a VIA VL716 enclosure (USB-C) connected to the USB-C port just to end up with the same symptoms I already ran into when testing with USB-C on NanoPC-T4 and RockPro64 a while ago. Tons of error messages in the logs (updated dmesg). In the kernel messages the question 'Maybe the USB cable is bad?' appears and I would believe the kernel is right (I have 2 USB-C cables, the good one from Apple somewhere where I don't find it, the el cheapo providing those messages. I need to buy another one) The good news: there is SuperSpeed (USB3) on the USB-C connector and it's not behind the internal USB3 hub (BTW: Schematics for the board are here: https://dl.vamrs.com/products/ficus/docs/hw/). The bad news: all USB receptacles are behind one of the two internal USB hubs.
 
So let's focus now on the 2 SATA ports provided by the JMS561.
 
1) Samsung EVO840 connected to one of the SATA ports. Testing methodology exactly identical as outlined here: https://forum.armbian.com/topic/8097-nanopi-m4-performance-and-consumption-review/?do=findComment&comment=61783. Only some minor slowdowns compared to NanoPi M4 (there the JMS567 USB-to-SATA bridge also being behind an internal USB3 hub). In other words: When HDDs are attached this is just fine since then the HDD is the bottleneck but neither the JMS561 nor the USB3 connection:
random random kB reclen write rewrite read reread read write 102400 4 23192 26619 20636 20727 20633 26177 102400 16 79214 89381 78784 79102 78889 88974 102400 512 292661 295840 271486 277983 277481 300712 102400 1024 321196 322169 305092 312765 312185 329470 102400 16384 356091 356179 350200 360167 359817 357583 2048000 16384 371761 374096 361612 361590 361724 374401  
2) Another Samsung added to 2nd 'SATA' port (now 2 disks connected to the JMS561). Running two independent iozone tests with 2GB test size at the same time. As expected we now run into 'shared bandwidth' and bus contention issues since both SSD are connected to the same JMS561 that is bottlenecked by the upstream connection to the SoC (one single SuperSpeed line that has to go through the USB3 hub). If we would talk about HDD the below numbers are still excellent since HDD are still the bottleneck with such a '2 disk' setup.
random random kB reclen write rewrite read reread read write PM851 2048000 16384 131096 132075 161149 162122 161282 131273 EVO840 2048000 16384 151916 157295 176928 183874 186000 178285 3) Now adding another Samsung SSD in an JMS567 enclosure to one of the USB3 receptacles (also behind the hub). As expected 'shared bandwidth' and bus contention issues matter even more now with 3 parallel iozone benchmarks running on each SSD in parallel. But the newly added Samsung EVO750 in an external enclosure finished way earlier:
random random kB reclen write rewrite read reread read write PM851 2048000 16384 112897 130875 125697 171855 175661 134626 EVO840 2048000 16384 113541 132846 127664 172124 174311 137836 EVO750 2048000 16384 214842 164890 328764 326065 197740 154398 TL;DR: The 2 available SATA ports provided by a JMS561 in PM fashion are totally sufficient when you want to attach one or two HDDs. In case you need highest random IO or sequential IO performance exceeding the USB3 limit (with this 'behind a hub' topology this means ~365 MB/s) then your only choice is the PCIe slot once it works (either by using a PCIe based SSD there or a SATA or even SAS adapter)
 

Latest RK3399 arrival in the lab. For now just some q&d photographs:
 



 
@wtarreau my first 96boards thing so far (just like you I felt the standard being directed towards nowhere given that there's no Ethernet). And guess what: 2 x Ethernet here!
 
A quick preliminary specifications list:
RK3399 (performing identical to any other RK3399 thingy out there as long as no throttling happens) 2 GB DDR3 RAM (in April Vamrs said they will provide 1GB, 2GB and 4GB variants for $99, $129 and $149) Board size is the standard 160×120 mm 96Boards EE form factor. EE = Enterprise Edition, for details download 96Boards-EE-Specification.pdf (1.1MB) Full size x16 PCIe slot as per EE specs (of course only x4 usable since RK3399 only provides 4 lanes at Gen2 speed) Board can be powered with 12V through barrel plug, 4-pin ATX plug or via pin header (Vamrs sent a 12V/4A PSU with the board) Serial console available via Micro USB (there's an onboard FTDI chip) 2 SATA ports + 2 SATA power ports (5V/12V). SATA is provided by a JMS561 USB3 SATA bridge that can operate in some silly RAID modes or PM mode (with spinning rust this chip is totally sufficient -- for SSDs better use NVMe/PCIe) Socketed eMMC and mechanical SD card adapter available (Vamrs sent also a SanDisk 8GB eMMC module as can be seen on the pictures) SIM card slot on the lower PCB side to be combined with an USB based WWAN modem in the mPCIe slot (USB2 only) 1 x SD card slot routed to RK3399, 1 x SD card slot for the BMC (Baseboard Management Controller) Gigabit Ethernet and separate Fast Ethernet port for the BMC Ampak AP6354 (dual-band and dual-antenna WiFi + BT 4.1) USB-C port with USB3 SuperSpeed and DisplayPort available eDP and HDMI 2.0 USB2 on pin headers and 2 type A receptacles all behind an internal USB2 hub USB3 on one pin header and 2 type A receptacles all behind an internal USB3 hub 96boards Low Speed Expansion connector with various interfaces exposed 96boards High Speed Expansion connector with various interfaces exposed (e.g. the 2nd USB2 host port, see diagram below) S/PDIF audio out 'real' on/off switch to cut power. To really power on the board the translucent button next to it needs to be pressed  

Sure. Just read the review.

Another look at the 2 SATA ports provided by the JMS561.
 
The same IC is inside Hardkernel's Cloudshell 2 thingy where users face serious SMART related issues. So let's do a quick check with an SSD connected to each SATA port:
root@rock960:~# for i in a b ; do smartctl -d sat -x /dev/sd${i}; smartctl -l devstat /dev/sd${i} -d sat,12; echo -e "\n\n\n\n\n" ; done | curl -F 'f:1=<-' ix.io http://ix.io/1o0p root@rock960:~# dmesg | tail -n5 [ 10.090856] rk_gmac-dwmac fe300000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx [ 89.067105] xhci-hcd xhci-hcd.7.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring [ 89.067959] xhci-hcd xhci-hcd.7.auto: @000000007bc32a30 00000000 00000000 1b000000 03078001 [ 89.162393] xhci-hcd xhci-hcd.7.auto: ERROR Transfer event for disabled endpoint or incorrect stream ring [ 89.163274] xhci-hcd xhci-hcd.7.auto: @000000007bc32ac0 00000000 00000000 1b000000 03078001 As http://ix.io/1o0p clearly shows the issue is still there. For the disk connected to the 'SATA1' port only bogus SMART data is returned. Shutting down the board, exchanging SATA cables so that now the EVO840 is connected to 'SATA1' and again: SMART readouts broken for the disk connected to this port: http://ix.io/1o0t
 
I then wanted to create a btrfs mirror to test the same with drives busy but to my surprise Vamrs' kernel has no btrfs support enabled. Then tried an mdraid-1 (which I personally consider the most stupid disk setup ever but hey, it's only about a load test) but also no mdraid support in this kernel
 
Ok, then continuing with individual ext4 filesystems on each SSD, running iozone -e -I -a -s 20000M -r 16384k -i 0 -i 1 -i 2 on each SSD and then again doing the smartctl queries.  No USB bus resets, performance also not affected but of course still bogus SMART readouts for one SSD and the above xhci error messages in the log.
 
So both SATA ports still are fine from a performance point of view when combined with HDDs (since all HDDs are that slow that they are the bottleneck and not the JMS561 provided SATA port) but unfortunately there are functional limitations wrt the disk connected to the 'SATA1' port (no SMART health queries, you can start SMART selftests but since SMART status can not be queried no way to get selftest results without exchanging disks/cables).
 
That being said Ficus with just one HDD connected to the 'SATA2' port is a great combination. I hope Vamrs looks into this, gets in touch with JMicron and checks whether this can be resolved with a firmware update.

Latest RK3399 arrival in the lab. For now just some q&d photographs:
 



 
@wtarreau my first 96boards thing so far (just like you I felt the standard being directed towards nowhere given that there's no Ethernet). And guess what: 2 x Ethernet here!
 
A quick preliminary specifications list:
RK3399 (performing identical to any other RK3399 thingy out there as long as no throttling happens) 2 GB DDR3 RAM (in April Vamrs said they will provide 1GB, 2GB and 4GB variants for $99, $129 and $149) Board size is the standard 160×120 mm 96Boards EE form factor. EE = Enterprise Edition, for details download 96Boards-EE-Specification.pdf (1.1MB) Full size x16 PCIe slot as per EE specs (of course only x4 usable since RK3399 only provides 4 lanes at Gen2 speed) Board can be powered with 12V through barrel plug, 4-pin ATX plug or via pin header (Vamrs sent a 12V/4A PSU with the board) Serial console available via Micro USB (there's an onboard FTDI chip) 2 SATA ports + 2 SATA power ports (5V/12V). SATA is provided by a JMS561 USB3 SATA bridge that can operate in some silly RAID modes or PM mode (with spinning rust this chip is totally sufficient -- for SSDs better use NVMe/PCIe) Socketed eMMC and mechanical SD card adapter available (Vamrs sent also a SanDisk 8GB eMMC module as can be seen on the pictures) SIM card slot on the lower PCB side to be combined with an USB based WWAN modem in the mPCIe slot (USB2 only) 1 x SD card slot routed to RK3399, 1 x SD card slot for the BMC (Baseboard Management Controller) Gigabit Ethernet and separate Fast Ethernet port for the BMC Ampak AP6354 (dual-band and dual-antenna WiFi + BT 4.1) USB-C port with USB3 SuperSpeed and DisplayPort available eDP and HDMI 2.0 USB2 on pin headers and 2 type A receptacles all behind an internal USB2 hub USB3 on one pin header and 2 type A receptacles all behind an internal USB3 hub 96boards Low Speed Expansion connector with various interfaces exposed 96boards High Speed Expansion connector with various interfaces exposed (e.g. the 2nd USB2 host port, see diagram below) S/PDIF audio out 'real' on/off switch to cut power. To really power on the board the translucent button next to it needs to be pressed  

Please be very careful with these wordings. M.2 is not mSATA. M.2 is just a mechanical connector able to transport a bunch of different protocols. In case you have no other device suited for an M.2 SATA SSD you might want to have a look at https://jeyi.aliexpress.com/store/group/USB3-0-USB3-1-Type-C-Series/710516_511630295.html for external USB3 enclosures (to get a bulky but ultra fast 'USB pendrive'). But since a 'generic' 60 GB SSD will perform crappy anyway and you talked about Amazon the best you could do is to return such a thing right now.

Latest RK3399 arrival in the lab. For now just some q&d photographs:
 



 
@wtarreau my first 96boards thing so far (just like you I felt the standard being directed towards nowhere given that there's no Ethernet). And guess what: 2 x Ethernet here!
 
A quick preliminary specifications list:
RK3399 (performing identical to any other RK3399 thingy out there as long as no throttling happens) 2 GB DDR3 RAM (in April Vamrs said they will provide 1GB, 2GB and 4GB variants for $99, $129 and $149) Board size is the standard 160×120 mm 96Boards EE form factor. EE = Enterprise Edition, for details download 96Boards-EE-Specification.pdf (1.1MB) Full size x16 PCIe slot as per EE specs (of course only x4 usable since RK3399 only provides 4 lanes at Gen2 speed) Board can be powered with 12V through barrel plug, 4-pin ATX plug or via pin header (Vamrs sent a 12V/4A PSU with the board) Serial console available via Micro USB (there's an onboard FTDI chip) 2 SATA ports + 2 SATA power ports (5V/12V). SATA is provided by a JMS561 USB3 SATA bridge that can operate in some silly RAID modes or PM mode (with spinning rust this chip is totally sufficient -- for SSDs better use NVMe/PCIe) Socketed eMMC and mechanical SD card adapter available (Vamrs sent also a SanDisk 8GB eMMC module as can be seen on the pictures) SIM card slot on the lower PCB side to be combined with an USB based WWAN modem in the mPCIe slot (USB2 only) 1 x SD card slot routed to RK3399, 1 x SD card slot for the BMC (Baseboard Management Controller) Gigabit Ethernet and separate Fast Ethernet port for the BMC Ampak AP6354 (dual-band and dual-antenna WiFi + BT 4.1) USB-C port with USB3 SuperSpeed and DisplayPort available eDP and HDMI 2.0 USB2 on pin headers and 2 type A receptacles all behind an internal USB2 hub USB3 on one pin header and 2 type A receptacles all behind an internal USB3 hub 96boards Low Speed Expansion connector with various interfaces exposed 96boards High Speed Expansion connector with various interfaces exposed (e.g. the 2nd USB2 host port, see diagram below) S/PDIF audio out 'real' on/off switch to cut power. To really power on the board the translucent button next to it needs to be pressed  

And two quick storage performance updates:
 
1) This is the 8GB eMMC FriendlyELEC sells (write performance limited to 43 MB/s as it's mostly the case with low capacity eMMC, read performance exceeding 130 MB/s, nice random IO values -- compare with our overview):
random random kB reclen write rewrite read reread read write 102400 4 8565 8793 24853 24808 19463 8523 102400 16 25604 25986 66627 66701 56571 24459 102400 512 42217 42326 125788 126508 125959 40394 102400 1024 42762 43178 129692 129726 130452 42636 102400 16384 43914 42877 132828 133254 133417 43012  
2) In the meantime I built also OMV images for NanoPC T4 and NanoPi M4 and did a quick LanTest benchmark with M4 and an externally connected Seagate Barracuda in an USB3 enclosure (not UAS capable -- but this doesn't matter with HDDs)
 
Close to 100 MB/s sequential speed without any more optimizations is simply awesome:

(debug output)

Please be very careful with these wordings. M.2 is not mSATA. M.2 is just a mechanical connector able to transport a bunch of different protocols. In case you have no other device suited for an M.2 SATA SSD you might want to have a look at https://jeyi.aliexpress.com/store/group/USB3-0-USB3-1-Type-C-Series/710516_511630295.html for external USB3 enclosures (to get a bulky but ultra fast 'USB pendrive'). But since a 'generic' 60 GB SSD will perform crappy anyway and you talked about Amazon the best you could do is to return such a thing right now.

Nope, everything as expected. The M4 number in the list https://github.com/ThomasKaiser/sbc-bench/blob/master/Results.md has been made with mainline kernel which shows way higher memory bandwidth on RK3399 (check the other RK3399 devices there). Cpuminer numbers differ due to GCC version (Stretch ships with 6.3, Bionic with 7.3 -- see the first three Rock64 numbers with 1400 MHz in the list -- always Stretch but two times with manually built newer GCC versions which significantly improve cpuminer performance)
 
If you love performance use recent software...

Relevant swapping occurred. Still no idea why so I added monitoring of virtual memory settings few hours ago. I'll look into this within the next months but no high priority since there is detailed monitoring in sbc-bench we know exactly why numbers differ currently between Orange Pi One Plus (1 GB) and PineH64 (more than 1 GB).

Relevant swapping occurred. Still no idea why so I added monitoring of virtual memory settings few hours ago. I'll look into this within the next months but no high priority since there is detailed monitoring in sbc-bench we know exactly why numbers differ currently between Orange Pi One Plus (1 GB) and PineH64 (more than 1 GB).

I did just that:
 

 
The thick thermal pad has been removed and replaced with a 15x15mm copper shim (0.8mm height). The grey stuff around the SoC are leftovers from another thermal pad (came with RockPro64) just to prevent shorting things with the heatsink.
 
I tried to use sbc-bench in thermal testing mode. NanoPi Fire is in upright position (so convection might help a little bit) in a room with 24°C ambient temperature.
sbc-bench.sh -T 80 ; sbc-bench.sh -t 60 Problem N° 1: with stock thermal pad I would have to wait endlessly for the SoC to cool down below 60°C (that what 'sbc-bench.sh -t 60' is supposed to do). The board once hot remained at 62°C to 63°C. I had to temporarely switch cpufreq governor to let the CPU cores clock down from 1400 to 400 MHz to get an idle temperature below 60°C again.
 
Then numbers were as follows:
1400 MHz: 146.17 sec 1300 MHz: 148.42 sec 1200 MHz: 0 sec 1100 MHz: 0 sec 1000 MHz: 0 sec 900 MHz: 0 sec 800 MHz: 0 sec 700 MHz: 0 sec 600 MHz: 0 sec 500 MHz: 0 sec 400 MHz: 5.80 sec (the 6 seconds on 400 MHz are not result of throttling but me manually switching back to performance cpufreq governor once the real benchmark run started). Full output where you can see how long it took to cool down from 80°C to 60°C: https://pastebin.com/raw/1DDt8yGk
 
Now with copper shim instead of thermal pad.
 
Problem N° 2: now it's impossible for the 'pre-heat' run to get above 80°C. I would have to wait endlessly for this so I stopped and ran 'sbc-bench.sh -T 78 ; sbc-bench.sh -t 60' instead to only wait until temperature exceeds 78°C. Full output: https://pastebin.com/raw/a0vmQJ0b
 
No throttling happened and temperature remained below 80°C. Again it's obvious that thermal pads simply suck when it's about efficient heat dissipation. But I've to admit that I've no idea whether the contact area now with my copper shim is sufficient or thermal pads are the recommended way since with my copper shim now only the very small die area and the copper shim have contact while the area around not.
 
@mindee can you please shed some light?
 
Edit: haha, today I realized that I ran with just 4 CPU cores active yesterday when I made my tests due to 'maxcpus=4' set in /boot/armbianEnv.txt (to test for swap efficiency of various approaches). Doesn't change anything wrt thermal conductivity but of course now with all 8 CPU cores active NanoPi Fire3 starts to throttle severely when running heavy stuff but at least with copper shim clockspeeds are much higher compared to thermal pad.

EDIT: This is NOT about a (hidden) backdoor but just a local privileges escalation that affects one single 3.4 kernel variant for 2 Allwinner SoCs. In case you came here through lurid headlines please read this comment here first and help stop spreading further BS like "all Allwinner devices contain a hidden backdoor" -- the code has been found since Allwinner could be convinced to open their Github repos in the past so everyone is able to audit their Android kernel source!
 
 
 
It has been brought to our attention two days ago on 2016-04-30 in linux-sunxi IRC that Allwinner's 3.4 legacy kernel for H3/A83T/H8 contains a nasty local privileges escalation. Any process running with any UID can become root easily by simply doing this:
echo "rootmydevice" > /proc/sunxi_debug/sunxi_debug Combined with networked services that might allow access to /proc this means that this is also a network enabled root exploit. As usual we fixed such an issue within a few hours, on May 1st new Armbian 5.10 images were rolled out and in the meantime the fix is also in Armbian's apt repo so please upgrade now (apt-get update/upgrade or start with a fresh OS image) if you're affected! If you're already running Armbian 5.10 then this vulnerability is already fixed.
 
This security flaw is currently present in every OS image for H3, A83T or H8 devices that rely on Kernel 3.4: this applies to all OS images available for Orange Pis (except Armbian 5.10 available since May 1st), any for FriendlyARM's NanoPi M1, SinoVoip's M2+ and M3, Cuebietech's Cubietruck + and Linksprite's pcDuino8 Uno
 
We informed all vendors where possible also two days ago (FriendlyARM and SinoVoip took notice, the others still ignore the issue) FriendlyARM: https://github.com/friendlyarm/h3_lichee/issues/1 SinoVoip M3: https://github.com/BPI-SINOVOIP/BPI-M3-bsp/issues/10 SinoVoip M2+: https://github.com/BPI-SINOVOIP/BPI-M2P-bsp/issues/1 Cubietruck +: https://github.com/cubieboard/Cubietruck_Plus-kernel-source/issues/2 LinkSprite pcDuino8 Uno: https://github.com/pcduino/pcduino8-uno-kernel/issues/3 Users of any of the available OS images for H3 based Orange Pis are somewhat lost since none of the OS images or kernels used for these are maintained any longer. Loboris disappeared (so the opened issue is more like a reference) and Xunlong themselves do not maintain their software at all (not even possible to open an issue there).   In case some readers here do also have accounts at Orange Pi forums, please spread the word and inform the users that their OS images are exploitable unless they already use Armbian 5.10. Same applies to Banana Pi forums, please spread the word there also that all M3 and their M2+ OS images are affected (I deleted my account there since talking to @sinovoip is like talking to a wall and just makes me sick, but it might be worth the efforts to try to warn their users)

Anyone catching the irony to keep Micro USB for powering? Guess it all depends on your target audience...

Quick summary:
unfortunately all the time throttling happened so results are not comparable other than expected with vm.swappiness=1 less swap activity happened compared to vm.swappiness=0 (while 'everyone on the Internet' will tell you the opposite): vm.swappiness=0 Compression: 2302,2234,2283 Decompression: 5486,5483,5490 Total: 3894,3858,3887 Write: 1168700 Read: 1337688 vm.swappiness=1 Compression: 2338,2260,2261 Decompression: 5506,5480,5512 Total: 3922,3870,3887 Write: 1138240 Read: 941548 vm.swappiness=60 Compression: 2266,2199,2204 Decompression: 5512,5495,5461 Total: 3889,3847,3832 Write: 1385560 Read: 1584220 vm.swappiness=100 Compression: 2261,2190,2200 Decompression: 5421,5485,5436 Total: 3841,3837,3818 Write: 1400808 Read: 1601404  
Still no idea why with with Orange Pi Plus and 1 GB RAM massive swapping occurs while the same benchmark on NanoPi Fire3 also with 1 GB results in low swapping attempts (different kernels, maybe different contents of /proc/sys/vm/*)

Sure thing.  I'll catch up later.

Haha, but now I know. I was an idiot before since it's simply zram/swap as can be easily seen by comparing iostat output from before and after:
before: zram1 0.93 3.69 0.01 1176 4 zram2 0.93 3.69 0.01 1176 4 zram3 0.93 3.69 0.01 1176 4 zram4 0.93 3.69 0.01 1176 4 after: zram1 588.13 1101.08 1251.45 1408792 1601184 zram2 586.62 1094.84 1251.62 1400808 1601404 zram3 582.01 1087.59 1240.44 1391524 1587092 zram4 587.14 1098.00 1250.54 1404848 1600016 That's 5.3GB read and 6.1GB written on the zram devices. I still have no idea why this benchmark on some boards (most probably kernels) runs with 1 GB without swapping but not on others like here:
RAM size: 994 MB, # CPU hardware threads: 4 RAM usage: 882 MB, # Benchmark threads: 4 NanoPi Fire3 also with just 1 GB RAM finishes with only minimal swapping:
RAM size: 990 MB, # CPU hardware threads: 8 RAM usage: 901 MB, # Benchmark threads: 8 Maybe vm.swappiness is the culprit. Can you repeat the bench another three times doing the following prior to each run:
sysctl -w vm.swappiness=0 sysctl -w vm.swappiness=1 sysctl -w vm.swappiness=60  

Exactly same numbers as PineH64 which is not that much of a surprise given same type of memory is used with same settings. Your 7-zip numbers seem to be lower but that's just some background activity trashing the benchmark as the monitoring reveals. If you see %sys and especially %iowait percentage in the monitoring output you know you need to repeat the benchmark and stop as many active processes as possible prior to benchmark execution:
System health while running 7-zip multi core benchmark: Time CPU load %cpu %sys %usr %nice %io %irq Temp 16:15:10: 1800MHz 5.63 23% 1% 18% 0% 3% 0% 25.0°C 16:15:31: 1800MHz 5.05 84% 1% 83% 0% 0% 0% 48.2°C 16:15:51: 1800MHz 4.77 86% 1% 84% 0% 0% 0% 43.1°C 16:16:31: 1800MHz 5.15 88% 15% 53% 0% 19% 0% 39.6°C 16:16:51: 1800MHz 4.94 80% 1% 78% 0% 0% 0% 42.7°C 16:17:11: 1800MHz 4.82 92% 1% 89% 0% 0% 0% 45.0°C 16:17:31: 1800MHz 4.64 87% 1% 85% 0% 0% 0% 41.9°C 16:17:52: 1800MHz 4.74 94% 16% 72% 0% 5% 0% 43.8°C 16:18:13: 1800MHz 4.69 81% 1% 80% 0% 0% 0% 48.6°C 16:18:33: 1800MHz 4.56 86% 1% 84% 0% 0% 0% 39.5°C 16:19:28: 1800MHz 6.93 84% 12% 38% 0% 34% 0% 31.2°C I bet unattended-upgrades was running in the background (you could check /var/log/dpkg.log)

Well, but that's what the extensive monitoring is for: to be able to throw results away quickly (no fire&forget benchmarking since only producing numbers without meaning).
 
The kernel reported high %sys and %io activity starting at the end of the single threaded 7-zip benchmark and that's the only reason your 7-zip numbers are lower than mine made on PineH64. Same H6, same type of memory, same u-boot, same kernel, same settings --> same performance.
 
IMO no more tests with improved cooling needed. The only remaining question is how good heat dissipation of both boards would be with otherwise identical environmental conditions (PineH64's PCB is rather huge and in both boards the copper ground plane is used as 'bottom heatsink' dissipating the heat away. But most probably PineH64 performs way better here with an appropriate heatsink due to larger PCB). But such a test is also pretty useless since results are somewhat predictable (larger PCB wins) and type of heatsink and whether there's some airflow around or not will be the more important factors. If heat dissipation problems are solved both boards will perform absolutely identical.

Anyone catching the irony to keep Micro USB for powering? Guess it all depends on your target audience...

Actually USB was never a good choice for networking especially when using behind a hub. I tested various setups and the only one that could actually work stably is with (only) one RTL8153 connected. If I connect 2 or more, as soon as I start stress testing (full duplex iperf on all connected NIC), the whole internal hub goes down in a few seconds (dmesg shows the internal hub is disconnected, lsusb -t only shows the root hub) and I have to do a USB reset or reboot the device to get that back again. I'm powering the board with official 5V/4A PSU so the board itself should be fine, but there still seems to be a current limit on the internal USB hub.
 
One RTL8153 combined with the internal GbE, however, works smoothly. I even tried to configure LACP with my CRS326 switch.

bonding1=ether1+ether2 (NanoPi M4)
bonding2=ether5+ether6+ether7 (NanoPC T4)
tested with iperf3 -P 2. With layer3+4 hash, two connections can easily use up to 2Gbps bandwidth between two devices.
 
PCIe cards would definitely be better, though I don't know how many people seriously want to use RK3399 devices as a router.