tkaiser Posted June 18, 2016 Posted June 18, 2016 The following is a short overview what you can expect from small and big H3 devices when used as a NAS. I chose the least capable device (OPi Lite for $12: not even Ethernet and just 512MB DRAM) and the best possible (OPi Plus 2E for $35: GBit Ethernet, 3 USB host ports exposed that do not have to share bandwidth, 2GB DRAM). I wanted to test also a H3 device in between with 1GB DRAM but since results are somewhat predictable I dropped the whole idea (the performance bottleneck on all Fast Ethernet equipped devices will be network unless you add the $7.50 for an USB-Ethernet dongle -- see below -- and all other Gbit Ethernet capable H3 devices are not priced competitive) Low end 3 weeks ago I ordered 2 cheap USB3-Ethernet dongles (Realtek RTL8153 based and recommended by @Rodolfo): http://www.ebay.com/itm/141821170951 They arrived in the meantime so I thought: Let's make OPi Lite an Ethernet device. With our current legacy kernel config and network settings you simply connect the adapter and an Ethernet cable, boot and have eth0 up and running (well, this should apply to most available USB-Ethernet adapters since we enabled every device available in kernel config). The dongle according to lsusb: Bus 001 Device 002: ID 0bda:8153 Realtek Semiconductor Corp. Bus 001 Device 002: ID 0bda:8153 Realtek Semiconductor Corp. Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 2.10 bDeviceClass 0 (Defined at Interface level) bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 64 idVendor 0x0bda Realtek Semiconductor Corp. idProduct 0x8153 bcdDevice 30.00 iManufacturer 1 Realtek-SYD iProduct 2 USB 10/100/1000 LAN iSerial 3 00E04C680528 bNumConfigurations 2 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 39 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xa0 (Bus Powered) Remote Wakeup MaxPower 100mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 3 bInterfaceClass 255 Vendor Specific Class bInterfaceSubClass 255 Vendor Specific Subclass bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 2 Transfer Type Bulk Synch Type None Usage Type Data wMaxPacketSize 0x0200 1x 512 bytes bInterval 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x02 EP 2 OUT bmAttributes 2 Transfer Type Bulk Synch Type None Usage Type Data wMaxPacketSize 0x0200 1x 512 bytes bInterval 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x83 EP 3 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0002 1x 2 bytes bInterval 8 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 80 bNumInterfaces 2 bConfigurationValue 2 iConfiguration 0 bmAttributes 0xa0 (Bus Powered) Remote Wakeup MaxPower 100mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 2 Communications bInterfaceSubClass 6 Ethernet Networking bInterfaceProtocol 0 iInterface 5 CDC Communications Control CDC Header: bcdCDC 1.10 CDC Union: bMasterInterface 0 bSlaveInterface 1 CDC Ethernet: iMacAddress 3 00E04C680528 bmEthernetStatistics 0x00000000 wMaxSegmentSize 1514 wNumberMCFilters 0x0000 bNumberPowerFilters 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x83 EP 3 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0010 1x 16 bytes bInterval 8 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 0 bNumEndpoints 0 bInterfaceClass 10 CDC Data bInterfaceSubClass 0 Unused bInterfaceProtocol 0 iInterface 0 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 1 bNumEndpoints 2 bInterfaceClass 10 CDC Data bInterfaceSubClass 0 Unused bInterfaceProtocol 0 iInterface 4 Ethernet Data Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 2 Transfer Type Bulk Synch Type None Usage Type Data wMaxPacketSize 0x0200 1x 512 bytes bInterval 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x02 EP 2 OUT bmAttributes 2 Transfer Type Bulk Synch Type None Usage Type Data wMaxPacketSize 0x0200 1x 512 bytes bInterval 0 Binary Object Store Descriptor: bLength 5 bDescriptorType 15 wTotalLength 22 bNumDeviceCaps 2 USB 2.0 Extension Device Capability: bLength 7 bDescriptorType 16 bDevCapabilityType 2 bmAttributes 0x00000002 Link Power Management (LPM) Supported SuperSpeed USB Device Capability: bLength 10 bDescriptorType 16 bDevCapabilityType 3 bmAttributes 0x02 Latency Tolerance Messages (LTM) Supported wSpeedsSupported 0x000e Device can operate at Full Speed (12Mbps) Device can operate at High Speed (480Mbps) Device can operate at SuperSpeed (5Gbps) bFunctionalitySupport 2 Lowest fully-functional device speed is High Speed (480Mbps) bU1DevExitLat 10 micro seconds bU2DevExitLat 2047 micro seconds Device Status: 0x0000 (Bus Powered) Since I want Lite's both USB host ports for disks, I used the OTG port and a Micro USB to USB adapter: a simple iperf test against a GbE device showed 270/300 Mbits/sec (depending on direction). Power requirements when adding Ethernet using this dongle: Plugging in the dongle without network cable attached: +700mW Connecting network cable to USB dongle (GbE!): another +400mW GbE transmission in one direction (limited to ~300 Mbits/sec): another +800mW So you can calculate with ~2W additional peak consumption per Ethernet adapter (at least 1.1W more if connected to a GbE network -- this is slightly more than the average 0.9W on Gbit Ethernet equipped SBC when the usual RTL8211E PHY establishes a GBit connection) I connected then a 3.5" Seagate Barracuda with external PSU (ext4 since with a 3.4 kernel we can not use more interesting filesystems like btrfs -- iozone shows ~35MB/s in both directions), compiled Netatalk 3.1.18 and tested NAS performance from my MacBook (no further tuning except 'echo performance >/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor' -- without this write performance totally sucks): Read performance is quite ok given that iperf shows just 270-300 Mbits/sec but write performance needs some tuning (not today). By looking at 'iostat 5' output it was obvious that write buffers were flushed only every few seconds so for normal NAS useage with small files the whole problem doesn't exist and it also should be possible to increase performance (not today). Anyway: search the net for correctly measured performance numbers of other SBC used as NAS and you will be already satisfied given that we're talking here about a $12+$7.50 combination High end Orange Pi Plus 2E is -- in my very personal opinion -- the best H3 device available if you think about NAS useage. It is equipped with the maximum amount of DRAM H3 can deal with, has Gbit Ethernet, exposes all 3 USB host ports + 1 OTG and comes with 16GB of pretty fast eMMC. At a competitive price (please keep in mind that you can install the OS on eMMC so you don't have to add the price of an SD card here). You can attach up to 4 USB disks (with mainline kernel and UASP capable enclosures they will show sequential speeds close to 40 MB/s, with legacy kernel it's ~5MB/s less) What you see here is the result of Gbit Ethernet paired with way more RAM and a test data size too small (only 300 MB fit perfectly into memory) so this is the increase in speed you will benefit from in normal NAS situations (dealing with files that do not exceed a few hundred MB in size). In case you try to write/read files larger 1 GB (or use software that often uses sync calls to ensure data is properly written to disk) be prepared that USB 2.0 becomes the bottleneck. In these situations sequential transfer speeds between NAS and clients will drop down to ~32MB/s without further tuning (applies to legacy kernel, for mainline see new post coming in the next days) Anyway: Please keep in mind that these are 'single disk' measurements. You can attach up to 4 disks to an OPi Plus 2E (using individual spindown policies to save energy or RAID modes to improve performance and/or availability), with Armbian defaults at least two of them can be accessed concurrently at full speed (USB2 maxing out at ~35MB/s and GbE being able to exceed 70MB/s easily) and with some tuning that might apply even to 3 disks accessed at the same time. And if I compare these benchmark results based on defaults (burning Armbian to SD card, firing up the NAS software, measuring performance, done) with what had to be done prior to being able to simply use Armbian as 'NAS distro of choice', eg. these one year old results with A20 then choosing OPi Plus 2E is a no-brainer. Regarding OPi Lite (or One or the smaller NanoPi M1) as NAS: This was more proof of concept than a recommendation. Being able to use as much RAM as possible for buffers is something especially a NAS / fileserver benefits from. So choosing a device with only 512MB is not the best idea. 'Native' Gbit Ethernet as present on a few H3 devices also clearly outperforms USB based solutions (iperf throughput with a recent Armbian without any tuning: 680/930 Mbits/sec). And if you add costs for USB-Gbit-Ethernet adapter and SD card the smaller boards aren't priced that competitive any longer. 10
Gravelrash Posted June 18, 2016 Posted June 18, 2016 Thankyou for the above, thats interesting and informative reading, I still favour the A20 boards with there true sata interface
tkaiser Posted June 18, 2016 Author Posted June 18, 2016 I still favour the A20 boards with there true sata interface Me not any more The problem with A20 is that sequential SATA performance is still unbalanced (45/200 MB/s write/read under best conditions -- read as overclocked CPU and DRAM), that the same applies to Gbit Ethernet performance (here the read performance sucks so unfortunately in this direction network is the bottleneck) so that we end up here with the following situation if we're talking about NAS: Client --> NAS: slow SATA write performance is the bottleneck NAS --> Client: slow GbE read performance is the bottleneck With H3 single disk access over USB 2.0 is the bottleneck (accessing data on disk that is not already cached will be limited to ~32 MB/s NAS throughput with legacy kernel, write performance to the NAS depends mostly on available DRAM and buffer settings/useage of the filesharing daemon in question). By switching to mainline kernel soon we're able to get close to 38/39 MB/s when UASP capable disk enclosures are used. And if disk performance really is an issue RAID-0 could be an option. I just created one on 3.4.112 without any tuning using 2 cheap crappy external 2.5" disks that also use crappy USB-to-SATA brudges, just by executing mdadm / mkfs.ext4 without using my brain: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2 random random kB reclen write rewrite read reread read write 102400 4 6529 7440 6561 6320 934 2006 102400 16 15845 16920 16380 16963 2995 8821 102400 512 29908 30385 30048 30786 23447 30493 102400 1024 55131 56964 53047 57078 36459 54178 102400 16384 62542 62603 62128 63343 62864 61185 Now we're able to write/read to/from our H3 NAS already with ~60MB/s. And when using UASP capable disk enclosures and mainline kernel RAID-0 performance will increase even more (+75 MB/s since single disk access gets close to 39MB/s). With mainline kernel we can also use btrfs and transparent filesystem compression so files that are able to be compressed 1:2 will then be read/written with 78MB/s instead of 39MB/s. Also with mainline kernel we can combine 3 USB disks in a way that sequential performance is twice the USB2 bandwidth while getting a fully redundant setup by combining RAID-0 with RAID-1 (mdraid + btrfs' own RAID implementation). This works even with disks of different size since when partitioning intelligently (using stripes of a specific size and combining striping/mirroring) you get a fully redundant disk setup showing doubled performance and 50 percent of the added capacity of all individual disks -- But that's stuff for another article I prepare for being released after the first H3 vanilla Armbian OS images I always talked about sequential transfer speeds above (since that's the most important parameter for normal NAS useage). Let's have a look at random IO: here the current situation clearly sucks with H3 compared to A20. Since with legacy kernel we can only use USB2.0 BOT mode (slower sequential transfer speeds and especially way slower when it's about random IO compared to UAS) while A20 here shows native SATA 2.0 capabilities (native command queueing and stuff like that). Therefore accessing many small files in random fashion on a SSD A20 might be ten times faster than H3 (or even more depending on the USB-to-SATA bridge used between H3 and the disk in an USB disk enclosure). Again: this might change with mainline kernel and an UASP capable disk enclosure since then also random IO over USB gets way faster. So always check which SATA bridge chip is used inside an enclosure. And BTW: That's the really great thing when we're talking about H3 for NAS useage: Since switching to mainline kernel (when UASP capable disk enclosures are in use!) increases disk access performance automagically for free (and when you use Armbian it's just a simple switch to exchange legacy kernel with vanilla). Further readings: http://linux-sunxi.org/Sunxi_devices_as_NAS http://linux-sunxi.org/USB/UAS http://linux-sunxi.org/SATA https://olimex.wordpress.com/2015/07/09/allwinner-did-it-again-new-quad-core-powerful-chip-pin-to-pin-compatible-with-a10-and-a20/#comments 3
tkaiser Posted June 18, 2016 Author Posted June 18, 2016 Almost forgot: The test of the cheap RTL8153 USB Ethernet dongle was also intended to give our H3 users the idea what to expect from spending additional $7.50 regarding network throughput if your H3 device is only 10/100 Mbits/sec capable (with Fast Ethernet you're limited to ~8 MB/s NAS throughput anyway). The most important detail to consider is whether you're dealing with USB ports that have to share bandwidth or not. H3 features 3 USB2.0 host ports and 1 OTG port (that can be switched to host mode and shows just a little performance drop) but unfortunately some devices use an internal USB hub. The performance numbers for OPi Lite above are only valid for other H3 devices where USB ports do not have to share bandwidth. So in case you own a Beelink X2 for example (2 USB host ports available) you're able to increase NAS performance by adding a GbE USB dongle and connecting an USB disk to the other port. If you own an Orange Pi 2 (or '2 mini') that's not true since the 4 USB ports available there have to share bandwidth since they are behind an internal USB hub. The count of USB ports doesn't matter at all. The question is just whether an internal USB hub (shared bandwidth) is used or not. Back to NAS useage again: That means horrible hardware designs like every Raspberry Pi (only one USB 2.0 port available between SoC and the outside) or moronic stuff like Banana Pi M3 (A83T SoC features two host ports but the 'engineers' chose to use just one of them connecting to an internal USB hub so all external USB ports and the ultra-slow GL830 USB-to-SATA bridge have to share bandwidth) will show worse NAS performance compared to an Orange Pi Lite with external Gbit Ethernet adapter 1
tkaiser Posted June 19, 2016 Author Posted June 19, 2016 Another update. Since I already created an 'el cheapo' RAID-0 made of two 2.5" 'USB disks' and therefore eliminated the 'single disk USB2.0 speed limitation' I tested again through the network (again: zero tuning, these are just 'Armbian defaults'): Accessing this RAID-0 through OPi Plus 2E's GBit Ethernet we get 48/63 MB/s (write/read) -- this time I chose 3GB test size so buffers aren't involved that much in this mode: Accessing it through the RTL8153 dongle we get 25/30 MB/s. I tested the same installation then also on Orange Pi PC Plus (1 GByte) and Lite (512 GByte) by swapping the SD card and only relinking script.bin and adjusting /etc/defaults/cpufreq-utils with exactly the same results. In this mode amount of DRAM is not relevant since due to the large test file buffers are immediately flushed to disk and we have the Ethernet dongle as bottleneck (so results with a single disk instead of a RAID-0 will look identical. But in normal NAS use cases the more DRAM will result in better overall performance) Funny: The first time I tried to test the Realtek Ethernet dongle connected to OPi Plus 2E I did what most people (me included) do most of the time when benchmarking: I did it wrong I brought the RTL8153 up as eth1 using an address from within the same net range as eth0 (onboard GbE) and so the kernel decided to do some fancy routing and while trying to use eth1 the packets used eth0 instead. The following sequential transfer speeds are simply impossible with an USB2 device (the RTL8153 results above showing 25/30 MB/s have been done with eth1 being in another net range than onboard eth0 on OPi Plus 2E and PC Plus) 2
Ford Prefect Posted June 20, 2016 Posted June 20, 2016 Since I already created an 'el cheapo' RAID-0 made of two 2.5" 'USB disks' and therefore eliminated the 'single disk USB2.0 speed limitation' Hi nice topic. Is there a tutorial somewhere ? I would like to try that on some Dockstar devices that I have.
tkaiser Posted June 20, 2016 Author Posted June 20, 2016 Is there a tutorial somewhere ? I would like to try that on some Dockstar devices that I have. Regarding RAID-0 with old kernels you have to rely on mdadm (you find gazillions of tutorials for this on the net). With mainline kernel (ready rather sooner than later for H3 devices! ) one could also use btrfs' own RAID-0 implementation (and mix it with RAID-1 -- then things get interesting!). But RAID-0 with old Seagate Dockstars is a waste of efforts anyway since the SoC is too weak and network the bottleneck.
tkaiser Posted June 22, 2016 Author Posted June 22, 2016 Small update using OPi Lite, kernel 4.6-rc1 and still the same USB3-Gbit-Ethernet adapter from first post: That's now 35 MB/s in both directions (I use btrfs on the disk with activated LZO compression) and almost too good to be true. In fact the network connection drops after some time (more related to a specific amount of data that has been transferred) and client and server disconnect. Won't investigate further since onboard network works quite well. Just a small note how to test with mainline kernel. There's still an Armbian 5.07 image available with mostly working Ethernet driver for BPi M2+ http://linux-sunxi.org/Banana_Pi_M2%2B#OS_images This image works out of the box on BPi M2+ or Orange Pi Plus 2E (only one USB port is missing) but since it's ten weeks old and the mainlining efforts for H3/Oranges are still WiP of course a few things do not work: HDMI Wi-Fi/BT, CSI Camera and other exotic stuff THS/throttling (this image uses a fixed cpufreq of 816MHz -- which should be ok for Oranges but not for BPi M2+, NanoPi M1 or Beelink X2 since all these 3 devices overheat a lot) Ethernet driver has some quirks but is already quite useable To use this old Armbian image with any of the H3 devices equipped with Fast Ethernet simply do the following (the example below applies to Orange Pi PC but you should get the idea what to adopt: simply change MyBoard="NanoPi M1" if you want to run it there) MyBoard="Orange Pi PC" MyBoardName="$(echo "${MyBoard}" | tr '[:upper:]' '[:lower:]' | sed -e 's/+/plus/' -e 's/\ //g' -e 's/2mini$/2/g' -e 's/plus2$/plus/g')" cat <<-EOF >/etc/armbian-release # PLEASE DO NOT EDIT THIS FILE BOARD=${MyBoardName} ID="${MyBoard}" VERSION=5.07 LINUXFAMILY=sun8i BRANCH=dev EOF wget -q -O - https://raw.githubusercontent.com/igorpecovnik/lib/master/scripts/armhwinfo >/etc/init.d/armhwinfo wget -q -O - https://raw.githubusercontent.com/igorpecovnik/lib/master/scripts/firstrun >/etc/init.d/firstrun wget -q -O - https://raw.githubusercontent.com/igorpecovnik/lib/master/scripts/armbianmonitor/armbianmonitor >/usr/local/bin/armbianmonitor cd /boot/dtb/ cat sun8i-h3-orangepi-pc.dtb >sun8i-h3-bananapi-m2plus.dtb sed -i "s/bananapim2plus/${MyBoardName}/" /etc/hostname reboot But please note that these images won't be able to upgrade to later official vanilla Armbian image (without hassles) and that you might permanently damage H3 when you run this image on any non Orange H3 device since they overheat too easy. But for some compatibility testing it should be quite ok.
tkaiser Posted June 23, 2016 Author Posted June 23, 2016 Another update: since I was testing an A20 board today in preparation of Armbian 5.15 release, SATA was mentioned in this thread and I found 2.5" disks I gave it a try and compared NAS performance on an A20 board using SATA, old/slow USB mass storage mode, faster "USB attached SCSI protocol" mode (UAS) and also a RAID-0 between SATA/UAS (to eliminate storage bottlenecks and make the network the limiting factor) I used a pcDuino3 Nano running 4.6.2, at 960MHz clockspeed (performance governor) using 2 identical SpinPoint M7 (same firmware, same useage profile). Single drive tests done with the same drive to show the differences protocol (SATA, UAS, USB Mass Storage) and the bridge chip in the enclosure might make. Drive data according to smartmontools (both USB-to-SATA bridges I used are fully SAT compatible and allow to query SMART data and trigger SMART self tests): Model Family: SAMSUNG SpinPoint M7 Device Model: SAMSUNG HM500JI Firmware Version: 2AC101C4 User Capacity: 500,107,862,016 bytes [500 GB] Sector Size: 512 bytes logical/physical Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 6 SATA Version is: SATA 2.6, 3.0 Gb/s SMART support is: Available - device has SMART capability. SMART support is: Enabled The iozone tests used these 3 sets (the first set is our usual test setup to get especially random IO performance, the two last test runs use twice the size of physical DRAM to show real sequential disk throughput without caches/buffers involved): iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2 iozone -a -g 2000m -s 2000m -i 0 -i 1 -r 4K iozone -a -g 2000m -s 2000m -i 0 -i 1 -r 1024K Onboard SATA: random random kB reclen write rewrite read reread read write 102400 4 6187 6403 10898 11470 497 5613 102400 16 12819 12422 22983 23358 1921 11974 102400 512 25472 25326 44996 44854 23549 25337 102400 1024 26429 26344 45306 46847 28518 26395 102400 16384 27194 27369 71150 72926 68996 27630 2048000 4 40796 39704 79686 78330 2048000 1024 40106 39322 79919 77556 USB/UAS: lsusb: 152d:3562 JMicron Technology Corp. / JMicron USA Technology Corp. JMS567 SATA 6Gb/s bridge dmesg: usbcore: registered new interface driver uas random random kB reclen write rewrite read reread read write 102400 4 4709 4825 6402 6599 496 4483 102400 16 12489 12306 15237 15500 1843 12022 102400 512 28286 28193 28422 29075 20617 28278 102400 1024 27861 28110 29240 29541 24940 27897 102400 16384 32088 32230 35944 36109 36408 31965 2048000 4 37538 37318 37074 37188 2048000 1024 37868 36838 37218 37293 USB/BOT (Mass storage): lsusb: 04fc:0c15 Sunplus Technology Co., Ltd SPIF215A SATA bridge dmesg: usbcore: registered new interface driver usb-storage random random kB reclen write rewrite read reread read write 102400 4 3646 3663 4810 5090 484 3485 102400 16 10371 9723 13667 13167 1880 9858 102400 512 26678 26648 26981 27092 19296 26626 102400 1024 27536 27472 27326 27577 21148 27504 102400 16384 34559 34533 32692 33114 33020 34524 2048000 4 32443 31353 32570 32777 2048000 1024 32223 31745 32575 32726 RAID-0 made of one SpinPoint M7 attached to SATA and the other to the UAS capable JMS567 USB enclosure I used btrfs and enabled also LZO compression so iozone results (especially sequential) are pretty worthless. It's just about eliminating storage bottlenecks to get the idea how an A20 based NAS with GbE behaves compared to an 'USB only' NAS as above. mkfs.btrfs -m raid1 -d raid0 /dev/sda1 /dev/sdb1 mount -o compress=lzo /dev/sda1 /mnt/sda/ random random kB reclen write rewrite read reread read write 102400 4 5607 5644 8357 8595 568 5203 102400 16 13339 12873 19534 19906 2026 12318 102400 512 49861 49044 50417 52949 31396 49289 102400 1024 53645 52701 56170 57347 43232 53006 102400 16384 64269 66759 62455 64459 61649 65130 2048000 4 86293 70816 65671 65579 2048000 1024 110122 96309 154940 153801 Simple conclusion USB when used on a device that is 'USB Attached SCSI' capable is not that bad (applies to A20, H3 and most likely also A64 when used with mainline kernel). But it depends also on the USB-to-SATA bridge used in disk enclosures or onboard (unfortunately on all sunxi boards with an USB-to-SATA bridge the horribly slow GL830 is used that's also not UAS capable, so bad news for owners of Orange Pi Plus, Plus 2, Banana Pi M3 or Cubietruck Plus). The Ethernet implementation in the newer Allwinner SoCs also is not that bad as the one used in A20 (here maximum network speeds differs a lot between several GbE capable sunxi boards for still unknown reasons). So when we're able to use mainline kernel on H3 devices performance will further improve (see also the measurements with the USB-GbE dongle above). At least for me SATA isn't a requirement any longer. But in case Olimex is right and Allwinner releases a pin compatible quad core A20 successor later this year it might get interesting again Some background information: http://linux-sunxi.org/USB/UAS http://linux-sunxi.org/SATA http://linux-sunxi.org/Sunxi_devices_as_NAS 1
tkaiser Posted June 24, 2016 Author Posted June 24, 2016 A last update when to choose better a different device. As said in the last post there is some hope that Allwinner will release a cheap quad core A20 successor that will be again SATA capable. A20's SATA implementation is NQC ready so especially random IO benefits from combination of SATA and a fast SSD. Unfortunately I only have a rather slow SSD lying around to test with Model Family: Samsung based SSDs Device Model: Samsung SSD 840 EVO 120GB Firmware Version: EXT0BB0Q User Capacity: 120,034,123,776 bytes [120 GB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4c SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Jun 24 14:17:04 2016 UTC SMART support is: Available - device has SMART capability. SMART support is: Enabled But here we go: still testing with A20 based pcDuino3 Nano running at 960 MHz CPU, 408 MHz DRAM, performance governor and 4.6.2 kernel: SSD connected to SATA (A20 is known for slow sequential write performance while reads can exceed 200MB/s): random random kB reclen write rewrite read reread read write 102400 4 15763 20117 42851 43604 27816 19619 102400 16 27910 31112 106510 107041 78693 31509 102400 512 38375 38832 206048 206948 203762 38885 102400 1024 38733 39123 211012 212564 210230 39032 102400 16384 39378 39348 225381 227301 227189 39331 2048000 16384 39464 39501 160257 170876 As a reference some overclocking results from a year ago (same SSD, same board): http://forum.lemaker.org/forum.php?mod=redirect&goto=findpost&ptid=12167&pid=66512 Same SSD, same setup, but now UAS used: 152d:3562 JMicron Technology Corp. / JMicron USA Technology Corp. JMS567 SATA 6Gb/s bridge /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-platform/1p, 480M |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 480M random random kB reclen write rewrite read reread read write 102400 4 7124 7901 8340 9410 7959 7926 102400 16 18734 21132 21238 21222 18935 21015 102400 512 35315 36161 36435 36605 37525 35993 102400 1024 34190 34881 38668 38587 38342 34920 102400 16384 36943 37217 39472 39546 39564 37158 2048000 16384 37426 37861 36412 36480 USB Mass storage: 04fc:0c15 Sunplus Technology Co., Ltd SPIF215A SATA bridge /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-platform/1p, 480M |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 480M random random kB reclen write rewrite read reread read write 102400 4 5206 6333 6367 6386 6368 6376 102400 16 15873 17997 18252 18194 18280 18118 102400 512 33996 34267 35473 35673 35653 34103 102400 1024 34269 34386 36021 35950 36105 34527 102400 16384 36207 36364 37482 37502 37381 36399 2048000 16384 34864 35305 31494 31466 Conclusion: Many small file accesses and especially random IO (database server for example): Better choose a slower SoC with the faster storage implementation! 1
Ford Prefect Posted June 25, 2016 Posted June 25, 2016 But RAID-0 with old Seagate Dockstars is a waste of efforts anyway since the SoC is too weak and network the bottleneck. Yes you are right I just made a test and the CPU is the bottleneck . Now that my switch is a Gb one I only get 17% network but 95% CPU . At least the Gigabit port of the Dockstar is not completely useless. On the contrary on my OPI PC the 100Mb network is the bottleneck . I will get some USB3 dongles they seem to be useful even on USB2 ports.
hojnikb Posted June 28, 2016 Posted June 28, 2016 Yeah, but dockstar was pretty great at the time. In 2009 you could get a Gigabit ethernet, 4 usb ports and very good linux support for around 20$. I had it up until a few months ago, when i finally switched over to a cheap ITX setup. E1 amd cpu, itx case. Hits 110MB/s with ease and it was pretty cheap as well (board itself was around 40€)
Set3 Posted July 22, 2016 Posted July 22, 2016 Hi guys, Impressive indeed, I am now using Orange Pi 2E Plus latest Armbian jessie server image + upgrade, all defaults, no effect by governor thing, raid0 using mdadm, simple SAMBA share, windows 10 client (old duo core), 2 old 2TB Seagate barracuda disks, cheap USB to SATA adapters Although SAMBA is not famous for high performance, it seems this gives similar results to the fancy apple server thingy above. 1.8 GB video file 1 disk : W 32.7 MBps / R 54.4 MBps 2 disks raid0 : W 41.8 MBps / R 52.8 MBps 33GB video files to see if it "holds" the speed ( takes around 10 -15 mins) 2 disks raid0 : W 40 MBps / 56MBps From 1 disk to 2 disks in RAID0, write speed increases 28%, Read speed does hardly improve, but as 56MBps = at least 448Mbps on Ethernet, that is already half my Ethernet bandwidth, not bad. Can my PC / switches even handle more :-) ? Hmm this limit is also approx the USB limit, so is that a limit of any H3 port anyway, also for Ethernet ? I just don't seem to get hd-idle to spin drives down yet, to investigate further. Thanks tkaiser for triggering this , great job guys !
tkaiser Posted July 23, 2016 Author Posted July 23, 2016 Hmm this limit is also approx the USB limit, so is that a limit of any H3 port anyway, also for Ethernet ? Nope, Ethernet is connected through RGMII and not USB and does not have to share bandwidth. USB disk transfers with legacy kernel max out at 35MB/s (depending on the bridge chip used inside the drive's enclosure -- see some performance numbers). 2 disks combined as RAID-0 on the Plus 2E (where no USB pors have to share bandwidth since no internal USB hub is in use!) should double performance. As written long time ago... always test individually too: http://linux-sunxi.org/Sunxi_devices_as_NAS#Use_the_right_tools So have a look at the first link for the iozone calls and measure directly on the board without network/Samba involved. And then you could also do a quick test how fast network is using iperf (Jperf.exe on Windows). Regarding disks not spinning down. Maybe there's a reason (that's the reason we ship with iotop and iostat). But some disks really ignore sleeping settings and since we're talking about USB disks again the used USB-to-SATA bridge might make the difference (only if the bridge supports SCSI / ATA Translation (SAT) then Linux tools get direct access to drive features like this) I would check whether you can send the disks to sleep with hdparm (hdparm -Y /dev/sda -- check with -C afterwards), install smartmontools and check disk using 'smartctl -a' and in case the disks do support being sent to sleep investigating with iotop/iostat whether activity happens or trying out scripted approaches like the one below (when you google for 'wwn-0x5000cca222d23f5f ' you get the source) or something like this https://github.com/lynix/hdd-spindown.sh/blob/master/hdd-spindown.sh #! /bin/bash # Check for idle disks and spin them down # run this from cron every 30 mins # */30 * * * * /usr/local/bin/spindown.sh # -- craiger, parts taken from Zack Reed # space separated list of disk wwn names to monitor. # These should be found in /dev/disk/by-id: DISKNAMES="wwn-0x5000cca222d23f5f " export PATH=$PATH:/sbin:/usr/sbin # Create a file on the ramdisk and cycle it to test for disk activity ( if [ ! -f /dev/shm/1 ] ; then touch /dev/shm/1 /dev/shm/2; fi ; mv /dev/shm/1 /dev/shm/2; cat /proc/diskstats > /dev/shm/1 ) >/dev/null 2>&1 date # convert disk wwn id's into sdX names because kernel developers # think they're so clever and randomize sd* names at boot, while most # userspace tools assume they're static, including /proc/diskstats for id in $DISKNAMES do SDLIST="$SDLIST `readlink /dev/disk/by-id/$id | cut -d/ -f3`" done # Loop through all disks and spin them down if idle. for disk in $SDLIST do # Check if disk exists if [ -e /dev/$disk ]; then # Check if disk is currently spinning if [ "$(hdparm -C /dev/$disk | grep state)" = " drive state is: active/idle" ]; then #Check if disk has been accessed since last time if [ "$(diff /dev/shm/1 /dev/shm/2 | grep $disk )" = "" ]; then echo "Drive $disk is spun up and idle. Spinning down now." hdparm -y /dev/$disk else echo "Drive $disk is spun up and being accessed. No change." fi else echo "Drive $disk is already spun down. No change." fi else echo "Drive $disk not found." #will never happen fi done BTW: Since we have to move a few customers to SMB (Apple tells devs since years that their old/own protocol AFP is deprecated in favour of SMB and added some proprietary extensions to the SMB protocol) I will test Samba within the next few weeks intensively. Will be using Samba 4.3 and above since this versions contains the Apple adoptions and when I'm at it I'll also have a look at performance tweaks. The servers we're talking about are a 'bit' larger than an Orange Pi (Oracle, Dell, HP servers running Solaris x86, FreeBSD or Linux) but I will give this a try on SBCs too. WIll be interesting how the kind of storage implementation matters (native SATA an A20 vs. USB Mass Storage vs. USB Attached SCSI possible with A20/H3 and mainline kernel) 1
Marcos A. S. Posted July 23, 2016 Posted July 23, 2016 Hi @tkaiser! I'm trying to use an OrangePi One as my DIY Mini NAS server but I'm getting *very* low performance while using external USB devices. Is the OrangePi one worse than the OrangePi Lite for this task? I'm seeing 1.2MBps at most with my setup. Am I doing something wrong? Thanks!
mikell Posted July 25, 2016 Posted July 25, 2016 I was curious about the hard drive speed between a pre-built NAS and a H3 NAS I was unable to load Iozone on the Lenovo NAS Equipment: NAS1 Orange PI PC 4 port hub, USB 2.0, 5V/3A powered Inateck FE2005 2.5†HDD enclosure, JMS567 chipset Seagate, 2.5â€, ST1000LM024, 1T 5400rpm NAS2 Lenovo Iomega EZ Media armv5tel Debian 7 wheezy (Original OS) Seagate 3.5“ 1T, 7200 rpm Results: Orange Pi PC hdparm -t --direct --offset 500 /dev/sda Timing O_DIRECT disk reads (offset 500 GB): 102 MB in 3.01 seconds = 33.88 MB/sec Lenovo hdparm -t --direct --offset 500 /dev/sda Timing O_DIRECT disk reads (offset 500 GB): 308 MB in 3.00 seconds = 102.60 MB/sec Dmesg reported several errors with the HDD(/dev/sda) when it was connected directly to the on-board usb. I solved it by passing through a powered usb hub. There appears to be current limits to the on-board USB. 1
tkaiser Posted July 25, 2016 Author Posted July 25, 2016 Is the OrangePi one worse than the OrangePi Lite for this task? Nope, both are H3 devices, obviously the Lite needs an external Ethernet solution (I used a rather fast USB3 GbE dongle) so unless you do not improve network situation (OPi One has just Fast Ethernet like most H3 devices) you can't get the transfer speeds I achieved with Lite + external Ethernet dongle. The post you answered to contained this link: http://linux-sunxi.org/Sunxi_devices_as_NAS#Use_the_right_tools So why not following the advices there to identify the bottleneck? Speaking about '1.2MBps' without telling what this number should be (local throughput test using tool X, network+storage benchmark using tool Y? 'Real world' throughput shown by Windows Explorer when using SMB? armv5tel Sounds like Marvell Kirkwood 88F6281 as used also in older QNAPs and other NAS boxes. Has native SATA with ok-ish performance (given that most HDDs show rather low transfer rates when they are full -- see below for the relationship between used capacity and performance) but GbE performance somewhat sucks. So in case you do the obvious (use Orange Pi Plus 2E instead of Orange Pi PC or add at least an USB3/GbE capable USB dongle) Orange Pi PC might outperform the Kirkwood based NAS if you add GbE networking. Your performance numbers are not really NAS related (that's always storage + network performance) and unfortunately of no great use. Using hdparm for benchmarking purposes is always wrong unless you ensure that you execute it 20 times and use an average value. A 'benchmark' that runs just 3 seconds will be heavily influenced by background HDD activity. And while hdparm tries to flush buffers before starting the 'benchmark', starting even only heavy write activity when hdparm already runs invalidates the read results hdparm shows (tested that many times, unfortunately forum search still sucks somehow so I can't reference numbers). Then HDDs show the following 'real world' performance behaviour: When they're nearly empty all read and write operations happen on the outer tracks where performance is normally twice that good as on the inner tracks. This is called ZBR: https://en.wikipedia.org/wiki/Zone_bit_recording So every HDD will show lower performance when it will be used as intended (filled with data) since all modern drives use a ZBR variant and fragmentation adds to the problem. Most if not all hdparm/dd benchmarks that can be found on the net ignore this so every time you find hdparm/dd numbers without telling something about the test setup simply forget about these numbers since they're 100 percent meaningless.
mdel Posted July 31, 2016 Posted July 31, 2016 @tkaiser have you located cheap ebay/ali UASP enclosures (2.5 or 3.5) ? i'm not sure how to properly check that in linux or if my 990fx chipset has uasp support but the three recent usb 3 enclosures i bought don't seem to be uasp compatible. Unfortunately i don't have a windows host at hand to double check that but i will next week. thx
tkaiser Posted July 31, 2016 Author Posted July 31, 2016 have you located cheap ebay/ali UASP enclosures (2.5 or 3.5) ? Nope, not even tried. The data my disks have to store has some value therefor I'm not following the 'buy cheap, buy twice' route here. I only order enclosures where the chipset is known (and would return an enclosure that comes with a different bridge chip), the last times I got some enclosures from CSL via Amazon (seems to be re-branded Inateck stuff, at the time I bought the enclosures they were cheaper than Inateck) So I would simply search for the chipsets on Amazon, eg. JMS567 and then test using an Armbian image with vanilla kernel and an A20 or H3 board. Attach the USB disk, then check dmesg output for 'uas': dmesg | egrep -i "usb|uas"
mdel Posted August 1, 2016 Posted August 1, 2016 okay i've also found that Inatek/CSL/MantisTek enclosure on amazon (same prices around 10e), i've ordered one to do some tests. Actually i got an Orico 3588US3 (3.5") from gearbest which is listed on amazon as an UASP enclosure, but it loads with "usb-storage" driver on my current kernel. So i'm not quite sure my 3.13.0 kernel even has uas drivers, i see no mention of CONFIG_USB_UAS in the config (ubuntu 14.04). And no "uas" module in my /lib/modules/, unless it was built in of course but usb-storage is not so i was expecting an uas driver somewhere.
tkaiser Posted August 1, 2016 Author Posted August 1, 2016 So i'm not quite sure my 3.13.0 kernel even has uas drivers, i see no mention of CONFIG_USB_UAS in the config (ubuntu 14.04). And no "uas" module in my /lib/modules/, unless it was built in of course but usb-storage is not so i was expecting an uas driver somewhere. Well, you seem to be talking about a x86 PC here? The USB controller used must support UAS as well and you find UAS support only on more recent USB3 controllers. USB Attached SCSI is an optional feature released at the same time as USB3 but can be combined with USB2 as well. But the only 2 USB2.0 host controller implementations capable of speaking UAS I know of are Allwinner's A20 and H3 (used with mainline kernel).
mdel Posted August 1, 2016 Posted August 1, 2016 yes sorry i was talking about my desktop amd64 kernel, i'm testing the enclosures there before moving one to my A20 or H3 boards for further tests. i'll continue investigating my desktop UAS support.
mdel Posted August 8, 2016 Posted August 8, 2016 i'm wondering what kind of mainline images are you using on your h3 devices ? I haven't seen any on the armbian download pages, are there some beta images i could test ? thx
tkaiser Posted August 8, 2016 Author Posted August 8, 2016 i'm wondering what kind of mainline images are you using on your h3 devices ? Explained in post #8 above Might work or not for you but IIRC you have a Beelink X2 (prone to overheating) so you should be careful not to overload the system later since no THS (throttling) is implemented in mainline kernel for H3 yet!
mdel Posted August 8, 2016 Posted August 8, 2016 thx sorry i missed that i'll test on my opi pc (with smallish heatsink) well now that you say i could kill my x2, i must say i'll probably be tempted to test it on it too =)
fr00t Posted August 8, 2016 Posted August 8, 2016 Orange Pi Plus 2E is -- in my very personal opinion -- the best H3 device available if you think about NAS useage. I am glad to read this as I recently purchased one for this very purpose. Unfortunately I discovered (from your review) too late that "sata" really means sata-over-usb, because my main criteria was "cheapest given gigabit ethernet + either SATA or usb 3.0". I envisioned attaching it to drives over sata for a home NAS solution with good throughput. Am I misunderstanding what is possible, or do you just have to different class of boards that aren't going to be bottlenecked by USB 2.0? (e.g. ODROID xu4 for instance would have significantly more throughput?) What is the benefit of gigabit ethernet when you are so bottlenecked by USB 2.0?
tkaiser Posted August 8, 2016 Author Posted August 8, 2016 "cheapest given gigabit ethernet + either SATA or usb 3.0" Then you should search whether you get a SheevaPlug on eBay (but don't be surprised that performance will suck since having Gbit Ethernet and SATA does not mean that full speed is possible -- the SoC is too weak). Another attempt would be to read through http://linux-sunxi.org/Sunxi_devices_as_NAS And if you want an ARM device that can fully saturate up to 3 SATA 3.0 channels while also being able to saturate several GbE links then the Clearfog Base would be an option (or cheap x86 designs )
DavidMM Posted August 10, 2016 Posted August 10, 2016 Hi How many 2.5 HDD does the OPi Plus 2E support? (using a 3 Amp power source) I have a banana pi (2 Amp power source) with two HDDs: one 2.5" and the other 3.5" and can't add a third 2.5" disk. In the long run I will like to migrate all the disk to 2.5" HDD so there is no additional power bricks.
tkaiser Posted August 10, 2016 Author Posted August 10, 2016 How many 2.5 HDD does the OPi Plus 2E support? (using a 3 Amp power source) You're talking about letting OPi Plus 2E power the disks? And you're talking about modern 2.5 SATA notebook HDDs (in opposite to some 2.5" server disks that need pretty much power all the time)? Normal notebook disks consume not that much even in active state. The problem is spin-up since then they might consume 1W peak consumption (check datasheets but don't trust them too much). I managed to let 3 disks (2 HDD, 1 SSD) be powered through an OPi PC maybe half a year ago. But only if I added them one after the other to delay consumption peaks. Letting all three attached and starting the board didn't work. Some disks support delayed/controlled spin-up (Staggered Spin-up: SSU and Power-Up In Standby: PUIS) but since we're talking about USB here it's then also a matter which USB-to-SATA bridge is used inside the enclosure (has to support SCSI / ATA Translation) and how kernel/drivers behave so in other words: better forget about it. If you're lucky it might work but it's better to expect the opposite. With your Banana Pi (A20 based) it's most probably undervoltage you're running into unless you use the SATA power port to power the Banana (and using one external PSU to power both SATA disk and board). But that's a different story BTW: Using an insufficient power supply to power a couple of disks is the best way to constantly run into weird problems.
CampGareth Posted August 10, 2016 Posted August 10, 2016 I'm planning to go down this route too with a couple of OPi Plus 2Es en-route from china now. Since I'm powering both the boards and the drives with a desktop PC PSU I didn't want to use HDD enclosures. Thankfully I've found USB 3.0 to eSATA adapters that're pretty much perfect for the job. The NAS aspect is a little different though, I use Ceph at home which is a distributed storage system so individual node performance doesn't matter much so long as there are a lot of them. I've been having problems with it crashing on my previous A20 powered board but I'll figure that out soon.
Recommended Posts