research H3 devices as NAS
6 6

48 posts in this topic

The following is a short overview what you can expect from small and big H3 devices when used as a NAS. I chose the least capable device (OPi Lite for $12: not even Ethernet and just 512MB DRAM) and the best possible (OPi Plus 2E for $35: GBit Ethernet, 3 USB host ports exposed that do not have to share bandwidth, 2GB DRAM).

 
OPi_Lite_RTL8153.jpg
 
I wanted to test also a H3 device in between with 1GB DRAM but since results are somewhat predictable I dropped the whole idea (the performance bottleneck on all Fast Ethernet equipped devices will be network unless you add the $7.50 for an USB-Ethernet dongle -- see below -- and all other Gbit Ethernet capable H3 devices are not priced competitive)
 
Low end
 
3 weeks ago I ordered 2 cheap USB3-Ethernet dongles (Realtek RTL8153 based and recommended by @Rodolfo): http://www.ebay.com/itm/141821170951
 
They arrived in the meantime so I thought: Let's make OPi Lite an Ethernet device. With our current legacy kernel config and network settings you simply connect the adapter and an Ethernet cable, boot and have eth0 up and running (well, this should apply to most available USB-Ethernet adapters since we enabled every device available in kernel config). The dongle according to lsusb:
Bus 001 Device 002: ID 0bda:8153 Realtek Semiconductor Corp.
 

 

Bus 001 Device 002: ID 0bda:8153 Realtek Semiconductor Corp. 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0bda Realtek Semiconductor Corp.
  idProduct          0x8153 
  bcdDevice           30.00
  iManufacturer           1 Realtek-SYD
  iProduct                2 USB 10/100/1000 LAN
  iSerial                 3 00E04C680528
  bNumConfigurations      2
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           39
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xa0
      (Bus Powered)
      Remote Wakeup
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           3
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass    255 Vendor Specific Subclass
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0002  1x 2 bytes
        bInterval               8
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           80
    bNumInterfaces          2
    bConfigurationValue     2
    iConfiguration          0 
    bmAttributes         0xa0
      (Bus Powered)
      Remote Wakeup
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         2 Communications
      bInterfaceSubClass      6 Ethernet Networking
      bInterfaceProtocol      0 
      iInterface              5 CDC Communications Control
      CDC Header:
        bcdCDC               1.10
      CDC Union:
        bMasterInterface        0
        bSlaveInterface         1 
      CDC Ethernet:
        iMacAddress                      3 00E04C680528
        bmEthernetStatistics    0x00000000
        wMaxSegmentSize               1514
        wNumberMCFilters            0x0000
        bNumberPowerFilters              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0010  1x 16 bytes
        bInterval               8
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass        10 CDC Data
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0 
      iInterface              0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       1
      bNumEndpoints           2
      bInterfaceClass        10 CDC Data
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0 
      iInterface              4 Ethernet Data
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength           22
  bNumDeviceCaps          2
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000002
      Link Power Management (LPM) Supported
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x02
      Latency Tolerance Messages (LTM) Supported
    wSpeedsSupported   0x000e
      Device can operate at Full Speed (12Mbps)
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   2
      Lowest fully-functional device speed is High Speed (480Mbps)
    bU1DevExitLat          10 micro seconds
    bU2DevExitLat        2047 micro seconds
Device Status:     0x0000
  (Bus Powered)
 
 
Since I want Lite's both USB host ports for disks, I used the OTG port and a Micro USB to USB adapter: a simple iperf test against a GbE device showed 270/300 Mbits/sec (depending on direction).
 
Power requirements when adding Ethernet using this dongle:
  • Plugging in the dongle without network cable attached: +700mW
  • Connecting network cable to USB dongle (GbE!): another +400mW
  • GbE transmission in one direction (limited to ~300 Mbits/sec): another +800mW
So you can calculate with ~2W additional peak consumption per Ethernet adapter (at least 1.1W more if connected to a GbE network -- this is slightly more than the average 0.9W on Gbit Ethernet equipped SBC when the usual RTL8211E PHY establishes a GBit connection)
 
I connected then a 3.5" Seagate Barracuda with external PSU (ext4 since with a 3.4 kernel we can not use more interesting filesystems like btrfs -- iozone shows ~35MB/s in both directions), compiled Netatalk 3.1.18 and tested NAS performance from my MacBook (no further tuning except 'echo performance >/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor' -- without this write performance totally sucks):
 
Bildschirmfoto%202016-06-17%20um%2012.13
 
Read performance is quite ok given that iperf shows just 270-300 Mbits/sec but write performance needs some tuning (not today). By looking at 'iostat 5' output it was obvious that write buffers were flushed only every few seconds so for normal NAS useage with small files the whole problem doesn't exist and it also should be possible to increase performance (not today). Anyway: search the net for correctly measured performance numbers of other SBC used as NAS and you will be already satisfied given that we're talking here about a $12+$7.50 combination :)
 
High end
 
Orange Pi Plus 2E is -- in my very personal opinion -- the best H3 device available if you think about NAS useage. It is equipped with the maximum amount of DRAM H3 can deal with, has Gbit Ethernet, exposes all 3 USB host ports + 1 OTG and comes with 16GB of pretty fast eMMC. At a competitive price (please keep in mind that you can install the OS on eMMC so you don't have to add the price of an SD card here).
 
You can attach up to 4 USB disks (with mainline kernel and UASP capable enclosures they will show sequential speeds close to 40 MB/s on the host ports and +35 MB/s on the OTG port -- with legacy kernel it's ~5MB/s less)
 
Bildschirmfoto%202016-06-17%20um%2016.48
 
What you see here is the result of Gbit Ethernet paired with way more RAM and a test data size too small (only 300 MB fit perfectly into memory) so this is the increase in speed you will benefit from in normal NAS situations (dealing with files that do not exceed a few hundred MB in size). In case you try to write/read files larger 1 GB (or use software that often uses sync calls to ensure data is properly written to disk) be prepared that USB 2.0 becomes the bottleneck. In these situations sequential transfer speeds between NAS and clients will drop down to ~32MB/s without further tuning (applies to legacy kernel, for mainline see new post coming in the next days)
 
Anyway: Please keep in mind that these are 'single disk' measurements. You can attach up to 4 disks to an OPi Plus 2E (using individual spindown policies to save energy or RAID modes to improve performance and/or availability), with Armbian defaults at least two of them can be accessed concurrently at full speed (USB2 maxing out at ~35MB/s and GbE being able to exceed 70MB/s easily) and with some tuning that might apply even to 3 disks accessed at the same time.
 
And if I compare these benchmark results based on defaults (burning Armbian to SD card, firing up the NAS software, measuring performance, done) with what had to be done prior to being able to simply use Armbian as 'NAS distro of choice', eg. these one year old results with A20 then choosing OPi Plus 2E is a no-brainer.
 
Regarding OPi Lite (or One or the smaller NanoPi M1) as NAS: This was more proof of concept than a recommendation. Being able to use as much RAM as possible for buffers is something especially a NAS / fileserver benefits from. So choosing a device with only 512MB is not the best idea. 'Native' Gbit Ethernet as present on a few H3 devices also clearly outperforms USB based solutions (iperf throughput with a recent Armbian without any tuning: 680/930 Mbits/sec). And if you add costs for USB-Gbit-Ethernet adapter and SD card the smaller boards aren't priced that competitive any longer.
manuti, Gravelrash, gnasch and 7 others like this

Share this post


Link to post
Share on other sites

Thankyou for the above, thats interesting and informative reading,

 

I still favour the A20 boards with there true sata interface

Share this post


Link to post
Share on other sites

I still favour the A20 boards with there true sata interface

 

Me not any more :)

 

The problem with A20 is that sequential SATA performance is still unbalanced (45/200 MB/s write/read under best conditions -- read as overclocked CPU and DRAM), that the same applies to Gbit Ethernet performance (here the read performance sucks so unfortunately in this direction network is the bottleneck) so that we end up here with the following situation if we're talking about NAS:

  • Client --> NAS: slow SATA write performance is the bottleneck
  • NAS --> Client: slow GbE read performance is the bottleneck

With H3 single disk access over USB 2.0 is the bottleneck (accessing data on disk that is not already cached will be limited to ~32 MB/s NAS throughput with legacy kernel, write performance to the NAS depends mostly on available DRAM and buffer settings/useage of the filesharing daemon in question). By switching to mainline kernel soon we're able to get close to 38/39 MB/s when UASP capable disk enclosures are used. And if disk performance really is an issue RAID-0 could be an option. I just created one on 3.4.112 without any tuning using 2 cheap crappy external 2.5" disks that also use crappy USB-to-SATA brudges, just by executing mdadm / mkfs.ext4 without using my brain:

iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4     6529     7440     6561     6320      934     2006
          102400      16    15845    16920    16380    16963     2995     8821
          102400     512    29908    30385    30048    30786    23447    30493
          102400    1024    55131    56964    53047    57078    36459    54178
          102400   16384    62542    62603    62128    63343    62864    61185

Now we're able to write/read to/from our H3 NAS already with ~60MB/s. And when using UASP capable disk enclosures and mainline kernel RAID-0 performance will increase even more (+75 MB/s since single disk access gets close to 39MB/s).

 

With mainline kernel we can also use btrfs and transparent filesystem compression so files that are able to be compressed 1:2 will then be read/written with 78MB/s instead of 39MB/s. Also with mainline kernel we can combine 3 USB disks in a way that sequential performance is twice the USB2 bandwidth while getting a fully redundant setup by combining RAID-0 with RAID-1 (mdraid + btrfs' own RAID implementation). This works even with disks of different size since when partitioning intelligently (using stripes of a specific size and combining striping/mirroring) you get a fully redundant disk setup showing doubled performance and 50 percent of the added capacity of all individual disks -- But that's stuff for another article I prepare for being released after the first H3 vanilla Armbian OS images :)

 

I always talked about sequential transfer speeds above (since that's the most important parameter for normal NAS useage). Let's have a look at random IO: here the current situation clearly sucks with H3 compared to A20. Since with legacy kernel we can only use USB2.0 BOT mode (slower sequential transfer speeds and especially way slower when it's about random IO compared to UAS) while A20 here shows native SATA 2.0 capabilities (native command queueing and stuff like that). Therefore accessing many small files in random fashion on a SSD A20 might be ten times faster than H3 (or even more depending on the USB-to-SATA bridge used between H3 and the disk in an USB disk enclosure).

 

Again: this might change with mainline kernel and an UASP capable disk enclosure since then also random IO over USB gets way faster. So always check which SATA bridge chip is used inside an enclosure. And BTW: That's the really great thing when we're talking about H3 for NAS useage: Since switching to mainline kernel (when UASP capable disk enclosures are in use!) increases disk access performance automagically for free (and when you use Armbian it's just a simple switch to exchange legacy kernel with vanilla).

 

Further readings:

Share this post


Link to post
Share on other sites

Almost forgot: The test of the cheap RTL8153 USB Ethernet dongle was also intended to give our H3 users the idea what to expect from spending additional $7.50 regarding network throughput if your H3 device is only 10/100 Mbits/sec capable (with Fast Ethernet you're limited to ~8 MB/s NAS throughput anyway).

 

The most important detail to consider is whether you're dealing with USB ports that have to share bandwidth or not. H3 features 3 USB2.0 host ports and 1 OTG port (that can be switched to host mode and shows just a little performance drop) but unfortunately some devices use an internal USB hub.

 

The performance numbers for OPi Lite above are only valid for other H3 devices where USB ports do not have to share bandwidth. So in case you own a Beelink X2 for example (2 USB host ports available) you're able to increase NAS performance by adding a GbE USB dongle and connecting an USB disk to the other port. If you own an Orange Pi 2 (or '2 mini') that's not true since the 4 USB ports available there have to share bandwidth since they are behind an internal USB hub.

 

The count of USB ports doesn't matter at all. The question is just whether an internal USB hub (shared bandwidth) is used or not.

 

Back to NAS useage again: That means horrible hardware designs like every Raspberry Pi (only one USB 2.0 port available between SoC and the outside) or moronic stuff like Banana Pi M3 (A83T SoC features two host ports but the 'engineers' chose to use just one of them connecting to an internal USB hub so all external USB ports and the ultra-slow GL830 USB-to-SATA bridge have to share bandwidth) will show worse NAS performance compared to an Orange Pi Lite with external Gbit Ethernet adapter :)

wildcat_paris likes this

Share this post


Link to post
Share on other sites

Another update. Since I already created an 'el cheapo' RAID-0 made of two 2.5" 'USB disks' and therefore eliminated the 'single disk USB2.0 speed limitation' I tested again through the network (again: zero tuning, these are just 'Armbian defaults'):

 

Accessing this RAID-0 through OPi Plus 2E's GBit Ethernet we get 48/63 MB/s (write/read) -- this time I chose 3GB test size so buffers aren't involved that much in this mode: 

 

Bildschirmfoto%202016-06-18%20um%2021.14

 

Accessing it through the RTL8153 dongle we get 25/30 MB/s. I tested the same installation then also on Orange Pi PC Plus (1 GByte) and Lite (512 GByte) by swapping the SD card and only relinking script.bin and adjusting /etc/defaults/cpufreq-utils with exactly the same results. In this mode amount of DRAM is not relevant since due to the large test file buffers are immediately flushed to disk and we have the Ethernet dongle as bottleneck (so results with a single disk instead of a RAID-0 will look identical. But in normal NAS use cases the more DRAM will result in better overall performance)

 

 

 

Bildschirmfoto%202016-06-19%20um%2011.21

 

Bildschirmfoto%202016-06-19%20um%2011.50

 

 

Bildschirmfoto%202016-06-19%20um%2012.28

 

 

 

Funny: The first time I tried to test the Realtek Ethernet dongle connected to OPi Plus 2E I did what most people (me included) do most of the time when benchmarking: I did it wrong :)

 

I brought the RTL8153 up as eth1 using an address from within the same net range as eth0 (onboard GbE) and so the kernel decided to do some fancy routing and while trying to use eth1 the packets used eth0 instead. The following sequential transfer speeds are simply impossible with an USB2 device :)

 

 

 

Bildschirmfoto%202016-06-19%20um%2010.54

 

 

 

(the RTL8153 results above showing 25/30 MB/s have been done with eth1 being in another net range than onboard eth0 on OPi Plus 2E and PC Plus)

slinde and wildcat_paris like this

Share this post


Link to post
Share on other sites

 Since I already created an 'el cheapo' RAID-0 made of two 2.5" 'USB disks' and therefore eliminated the 'single disk USB2.0 speed limitation'

Hi nice topic.

Is there a tutorial somewhere ?

I would like to try that on some Dockstar devices that I have.

Share this post


Link to post
Share on other sites

Is there a tutorial somewhere ?

I would like to try that on some Dockstar devices that I have.

 

Regarding RAID-0 with old kernels you have to rely on mdadm (you find gazillions of tutorials for this on the net). With mainline kernel (ready rather sooner than later for H3 devices! :) ) one could also use btrfs' own RAID-0 implementation (and mix it with RAID-1 -- then things get interesting!).

 

But RAID-0 with old Seagate Dockstars is a waste of efforts anyway since the SoC is too weak and network the bottleneck.

Share this post


Link to post
Share on other sites

Small update using OPi Lite, kernel 4.6-rc1 and still the same USB3-Gbit-Ethernet adapter from first post:

Bildschirmfoto%202016-06-22%20um%2015.05

 

That's now 35 MB/s in both directions (I use btrfs on the disk with activated LZO compression) and almost too good to be true. In fact the network connection drops after some time (more related to a specific amount of data that has been transferred) and client and server disconnect. Won't investigate further since onboard network works quite well.

 

Just a small note how to test with mainline kernel. There's still an Armbian 5.07 image available with mostly working Ethernet driver for BPi M2+  http://linux-sunxi.org/Banana_Pi_M2%2B#OS_images

 

This image works out of the box on BPi M2+ or Orange Pi Plus 2E (only one USB port is missing) but since it's ten weeks old and the mainlining efforts for H3/Oranges are still WiP of course a few things do not work:

  • HDMI
  • Wi-Fi/BT, CSI Camera and other exotic stuff
  • THS/throttling (this image uses a fixed cpufreq of 816MHz -- which should be ok for Oranges but not for BPi M2+, NanoPi M1 or Beelink X2 since all these 3 devices overheat a lot)
  • Ethernet driver has some quirks but is already quite useable

To use this old Armbian image with any of the H3 devices equipped with Fast Ethernet simply do the following (the example below applies to Orange Pi PC but you should get the idea what to adopt: simply change MyBoard="NanoPi M1" if you want to run it there)

MyBoard="Orange Pi PC"
MyBoardName="$(echo "${MyBoard}" | tr '[:upper:]' '[:lower:]' | sed -e 's/+/plus/' -e 's/\ //g' -e 's/2mini$/2/g' -e 's/plus2$/plus/g')"
cat <<-EOF >/etc/armbian-release
# PLEASE DO NOT EDIT THIS FILE
BOARD=${MyBoardName}
ID="${MyBoard}"
VERSION=5.07
LINUXFAMILY=sun8i
BRANCH=dev
EOF
wget -q -O - https://raw.githubusercontent.com/igorpecovnik/lib/master/scripts/armhwinfo >/etc/init.d/armhwinfo
wget -q -O - https://raw.githubusercontent.com/igorpecovnik/lib/master/scripts/firstrun >/etc/init.d/firstrun
wget -q -O - https://raw.githubusercontent.com/igorpecovnik/lib/master/scripts/armbianmonitor/armbianmonitor >/usr/local/bin/armbianmonitor
cd /boot/dtb/
cat sun8i-h3-orangepi-pc.dtb >sun8i-h3-bananapi-m2plus.dtb
sed -i "s/bananapim2plus/${MyBoardName}/" /etc/hostname
reboot

But please note that these images won't be able to upgrade to later official vanilla Armbian image (without hassles) and that you might permanently damage H3 when you run this image on any non Orange H3 device since they overheat too easy. But for some compatibility testing it should be quite ok.

Share this post


Link to post
Share on other sites

Another update: since I was testing an A20 board today in preparation of Armbian 5.15 release, SATA was mentioned in this thread and I found 2.5" disks I gave it a try and compared NAS performance on an A20 board using SATA, old/slow USB mass storage mode, faster "USB attached SCSI protocol" mode (UAS) and also a RAID-0 between SATA/UAS (to eliminate storage bottlenecks and make the network the limiting factor)

 
I used a pcDuino3 Nano running 4.6.2, at 960MHz clockspeed (performance governor) using 2 identical SpinPoint M7 (same firmware, same useage profile). Single drive tests done with the same drive to show the differences protocol (SATA, UAS, USB Mass Storage) and the bridge chip in the enclosure might make. Drive data according to smartmontools (both USB-to-SATA bridges I used are fully SAT compatible and allow to query SMART data and trigger SMART self tests):
Model Family:     SAMSUNG SpinPoint M7
Device Model:     SAMSUNG HM500JI
Firmware Version: 2AC101C4
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Size:      512 bytes logical/physical
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 2.6, 3.0 Gb/s
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

The iozone tests used these 3 sets (the first set is our usual test setup to get especially random IO performance, the two last test runs use twice the size of physical DRAM to show real sequential disk throughput without caches/buffers involved):

iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
iozone -a -g 2000m -s 2000m -i 0 -i 1 -r 4K
iozone -a -g 2000m -s 2000m -i 0 -i 1 -r 1024K

Onboard SATA:

                                                               random   random
              kB  reclen    write  rewrite    read    reread     read    write
          102400       4     6187     6403    10898    11470      497     5613
          102400      16    12819    12422    22983    23358     1921    11974
          102400     512    25472    25326    44996    44854    23549    25337
          102400    1024    26429    26344    45306    46847    28518    26395
          102400   16384    27194    27369    71150    72926    68996    27630
         2048000       4    40796    39704    79686    78330
         2048000    1024    40106    39322    79919    77556

Bildschirmfoto%202016-06-23%20um%2016.16
Bildschirmfoto%202016-06-23%20um%2016.04

 

 
USB/UAS:
  • lsusb: 152d:3562 JMicron Technology Corp. / JMicron USA Technology Corp. JMS567 SATA 6Gb/s bridge
  • dmesg: usbcore: registered new interface driver uas
                                                               random   random
              kB  reclen    write  rewrite    read    reread     read    write
          102400       4     4709     4825     6402     6599      496     4483
          102400      16    12489    12306    15237    15500     1843    12022
          102400     512    28286    28193    28422    29075    20617    28278
          102400    1024    27861    28110    29240    29541    24940    27897
          102400   16384    32088    32230    35944    36109    36408    31965
         2048000       4    37538    37318    37074    37188
         2048000    1024    37868    36838    37218    37293

Bildschirmfoto%202016-06-23%20um%2017.30
Bildschirmfoto%202016-06-23%20um%2019.27

 

 
USB/BOT (Mass storage):
  • lsusb: 04fc:0c15 Sunplus Technology Co., Ltd SPIF215A SATA bridge
  • dmesg: usbcore: registered new interface driver usb-storage
                                                               random   random
              kB  reclen    write  rewrite    read    reread     read    write
          102400       4     3646     3663     4810     5090      484     3485
          102400      16    10371     9723    13667    13167     1880     9858
          102400     512    26678    26648    26981    27092    19296    26626
          102400    1024    27536    27472    27326    27577    21148    27504
          102400   16384    34559    34533    32692    33114    33020    34524
         2048000       4    32443    31353    32570    32777
         2048000    1024    32223    31745    32575    32726

Bildschirmfoto%202016-06-23%20um%2020.14
Bildschirmfoto%202016-06-23%20um%2020.08

 

         
RAID-0 made of one SpinPoint M7 attached to SATA and the other to the UAS capable JMS567 USB enclosure
 
I used btrfs and enabled also LZO compression so iozone results (especially sequential) are pretty worthless. It's just about eliminating storage bottlenecks to get the idea how an A20 based NAS with GbE behaves compared to an 'USB only' NAS as above.
mkfs.btrfs -m raid1 -d raid0 /dev/sda1 /dev/sdb1
mount -o compress=lzo /dev/sda1 /mnt/sda/
                                                               random   random
              kB  reclen    write  rewrite    read    reread     read    write
          102400       4     5607     5644     8357     8595      568     5203
          102400      16    13339    12873    19534    19906     2026    12318
          102400     512    49861    49044    50417    52949    31396    49289
          102400    1024    53645    52701    56170    57347    43232    53006
          102400   16384    64269    66759    62455    64459    61649    65130
         2048000       4    86293    70816    65671    65579
         2048000    1024   110122    96309   154940   153801

Bildschirmfoto%202016-06-23%20um%2020.46
Bildschirmfoto%202016-06-23%20um%2020.59

 

 
Simple conclusion
 
USB when used on a device that is 'USB Attached SCSI' capable is not that bad (applies to A20, H3 and most likely also A64 when used with mainline kernel). But it depends also on the USB-to-SATA bridge used in disk enclosures or onboard (unfortunately on all sunxi boards with an USB-to-SATA bridge the horribly slow GL830 is used that's also not UAS capable, so bad news for owners of Orange Pi Plus, Plus 2, Banana Pi M3 or Cubietruck Plus).
 
The Ethernet implementation in the newer Allwinner SoCs also is not that bad as the one used in A20 (here maximum network speeds differs a lot between several GbE capable sunxi boards for still unknown reasons). So when we're able to use mainline kernel on H3 devices performance will further improve (see also the measurements with the USB-GbE dongle above). At least for me SATA isn't a requirement any longer. But in case Olimex is right and Allwinner releases a pin compatible quad core A20 successor later this year it might get interesting again :)
 
Some background information:
wildcat_paris likes this

Share this post


Link to post
Share on other sites

A last update when to choose better a different device. As said in the last post there is some hope that Allwinner will release a cheap quad core A20 successor that will be again SATA capable. A20's SATA implementation is NQC ready so especially random IO benefits from combination of SATA and a fast SSD. Unfortunately I only have a rather slow SSD lying around to test with

Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 840 EVO 120GB
Firmware Version: EXT0BB0Q
User Capacity:    120,034,123,776 bytes [120 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Jun 24 14:17:04 2016 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

But here we go: still testing with A20 based pcDuino3 Nano running at 960 MHz CPU, 408 MHz DRAM, performance governor and 4.6.2 kernel:

 

SSD connected to SATA (A20 is known for slow sequential write performance while reads can exceed 200MB/s):
                                                               random   random
              kB  reclen    write  rewrite    read    reread     read    write
          102400       4    15763    20117    42851    43604    27816    19619
          102400      16    27910    31112   106510   107041    78693    31509
          102400     512    38375    38832   206048   206948   203762    38885
          102400    1024    38733    39123   211012   212564   210230    39032
          102400   16384    39378    39348   225381   227301   227189    39331
         2048000   16384    39464    39501   160257   170876

As a reference some overclocking results from a year ago (same SSD, same board): http://forum.lemaker.org/forum.php?mod=redirect&goto=findpost&ptid=12167&pid=66512

 
Same SSD, same setup, but now UAS used:
152d:3562 JMicron Technology Corp. / JMicron USA Technology Corp. JMS567 SATA 6Gb/s bridge
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-platform/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 480M
                                                               random   random
              kB  reclen    write  rewrite    read    reread     read    write
          102400       4     7124     7901     8340     9410     7959     7926
          102400      16    18734    21132    21238    21222    18935    21015
          102400     512    35315    36161    36435    36605    37525    35993
          102400    1024    34190    34881    38668    38587    38342    34920
          102400   16384    36943    37217    39472    39546    39564    37158
         2048000   16384    37426    37861    36412    36480

USB Mass storage:

04fc:0c15 Sunplus Technology Co., Ltd SPIF215A SATA bridge
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-platform/1p, 480M
    |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 480M
                                                               random   random
              kB  reclen    write  rewrite    read    reread     read    write
          102400       4     5206     6333     6367     6386     6368     6376
          102400      16    15873    17997    18252    18194    18280    18118
          102400     512    33996    34267    35473    35673    35653    34103
          102400    1024    34269    34386    36021    35950    36105    34527
          102400   16384    36207    36364    37482    37502    37381    36399
         2048000   16384    34864    35305    31494    31466

Conclusion:

 

Many small file accesses and especially random IO (database server for example): Better choose a slower SoC with the faster storage implementation! :)

wildcat_paris likes this

Share this post


Link to post
Share on other sites

But RAID-0 with old Seagate Dockstars is a waste of efforts anyway since the SoC is too weak and network the bottleneck.

Yes you are right I just made a test and the CPU is the bottleneck .

Now that my switch is a Gb one I only get 17% network but 95% CPU .

At least the Gigabit port of the Dockstar is not completely useless.

On the contrary on my OPI PC the 100Mb network  is the bottleneck .

I will get some USB3 dongles they seem to be useful even on USB2 ports.

Share this post


Link to post
Share on other sites

Yeah, but dockstar was pretty great at the time. In 2009 you could get a Gigabit ethernet, 4 usb ports and very good linux support for around 20$.

 

I had it up until a few months ago, when i finally switched over to a cheap ITX setup. E1 amd cpu, itx case. Hits 110MB/s with ease and it was pretty cheap as well (board itself was around 40€)

Share this post


Link to post
Share on other sites

Hi guys,

 

Impressive indeed, I am now using Orange Pi 2E Plus latest Armbian jessie server image + upgrade, all defaults, no effect by governor thing, raid0 using mdadm, simple SAMBA share, windows 10 client (old duo core), 2 old 2TB Seagate barracuda disks, cheap USB to SATA adapters

 

Although SAMBA is not famous for high performance, it seems this gives similar results to the fancy apple server thingy above.

 

1.8 GB video file

    1 disk            : W 32.7 MBps / R 54.4 MBps

    2 disks raid0 : W 41.8 MBps / R 52.8 MBps

 

33GB video files to see if it "holds" the speed ( takes around 10 -15 mins)

    2 disks raid0 : W 40 MBps / 56MBps

 

From 1 disk to 2 disks in RAID0, write speed increases 28%, Read speed does hardly improve, but as 56MBps = at least 448Mbps on Ethernet, that is already half my Ethernet bandwidth, not bad.

Can my PC / switches even handle more :-) ?

 

Hmm this limit is also approx the USB limit, so is that a limit of any H3 port anyway, also for Ethernet ?

 

I just don't seem to get hd-idle to spin drives down yet, to investigate further.

 

Thanks tkaiser for triggering this  , great job guys !

Share this post


Link to post
Share on other sites

Hmm this limit is also approx the USB limit, so is that a limit of any H3 port anyway, also for Ethernet ?

 

Nope, Ethernet is connected through RGMII and not USB and does not have to share bandwidth. USB disk transfers with legacy kernel max out at 35MB/s (depending on the bridge chip used inside the drive's enclosure -- see some performance numbers). 2 disks combined as RAID-0 on the Plus 2E (where no USB pors have to share bandwidth since no internal USB hub is in use!) should double performance. 

 

As written long time ago... always test individually too: http://linux-sunxi.org/Sunxi_devices_as_NAS#Use_the_right_tools

 

So have a look at the first link for the iozone calls and measure directly on the board without network/Samba involved. And then you could also do a quick test how fast network is using iperf (Jperf.exe on Windows).

 

Regarding disks not spinning down. Maybe there's a reason (that's the reason we ship with iotop and iostat). But some disks really ignore sleeping settings and since we're talking about USB disks again the used USB-to-SATA bridge might make the difference (only if the bridge supports SCSI / ATA Translation (SAT) then Linux tools get direct access to drive features like this)

 

I would check whether you can send the disks to sleep with hdparm (hdparm -Y /dev/sda -- check with -C afterwards), install smartmontools and check disk using 'smartctl -a' and in case the disks do support being sent to sleep investigating with iotop/iostat whether activity happens or trying out scripted approaches like the one below (when you google for 'wwn-0x5000cca222d23f5f ' you get the source) or something like this https://github.com/lynix/hdd-spindown.sh/blob/master/hdd-spindown.sh

 

 

 

#! /bin/bash
# Check for idle disks and spin them down
# run this from cron every 30 mins
#     */30 * * * * /usr/local/bin/spindown.sh
#  -- craiger, parts taken from Zack Reed

# space separated list of disk wwn names to monitor.
# These should be found in /dev/disk/by-id:
DISKNAMES="wwn-0x5000cca222d23f5f "

export PATH=$PATH:/sbin:/usr/sbin

# Create a file on the ramdisk and cycle it to test for disk activity
( if [ ! -f /dev/shm/1 ] ; then touch /dev/shm/1 /dev/shm/2; fi ; mv /dev/shm/1 /dev/shm/2; cat /proc/diskstats > /dev/shm/1 ) >/dev/null 2>&1

date 

# convert disk wwn id's into sdX names because kernel developers
# think they're so clever and randomize sd* names at boot, while most 
# userspace tools assume they're static, including /proc/diskstats
for id in $DISKNAMES 
do
  SDLIST="$SDLIST `readlink /dev/disk/by-id/$id | cut -d/ -f3`"
done

# Loop through all disks and spin them down if idle.
for disk in $SDLIST
do
  # Check if disk exists
  if [ -e /dev/$disk ]; then
     # Check if disk is currently spinning
     if [ "$(hdparm -C /dev/$disk | grep state)" = " drive state is:  active/idle" ]; then
         #Check if disk has been accessed since last time
         if [ "$(diff /dev/shm/1 /dev/shm/2 | grep $disk )" =  "" ]; then                        
            echo "Drive $disk is spun up and idle.  Spinning down now." 
            hdparm -y /dev/$disk 
         else 
            echo "Drive $disk is spun up and being accessed.  No change." 
         fi
     else
        echo "Drive $disk is already spun down.  No change." 
     fi
  else
     echo "Drive $disk not found."   #will never happen
  fi
done 

 

 

 

BTW: Since we have to move a few customers to SMB (Apple tells devs since years that their old/own protocol AFP is deprecated in favour of SMB and added some proprietary extensions to the SMB protocol) I will test Samba within the next few weeks intensively. Will be using Samba 4.3 and above since this versions contains the Apple adoptions and when I'm at it I'll also have a look at performance tweaks. The servers we're talking about are a 'bit' larger than an Orange Pi (Oracle, Dell, HP servers running Solaris x86, FreeBSD or Linux) but I will give this a try on SBCs too. WIll be interesting how the kind of storage implementation matters (native SATA an A20 vs. USB Mass Storage vs. USB Attached SCSI possible with A20/H3 and mainline kernel)

Marcos A. S. likes this

Share this post


Link to post
Share on other sites

Hi @tkaiser!

 

I'm trying to use an OrangePi One as my DIY Mini NAS server but I'm getting *very* low performance while using external USB devices. 

Is the OrangePi one worse than the OrangePi Lite for this task?

 

I'm seeing 1.2MBps at most with my setup. Am I doing something wrong? :(

 

Thanks!

Share this post


Link to post
Share on other sites

I was curious about the hard drive speed between a pre-built NAS and a H3 NAS

I was unable to load Iozone on the Lenovo NAS

 

Equipment:

NAS1

Orange PI PC

4 port hub, USB 2.0, 5V/3A powered

Inateck FE2005 2.5†HDD enclosure, JMS567 chipset

Seagate, 2.5â€, ST1000LM024, 1T 5400rpm

 

NAS2

Lenovo Iomega EZ Media

armv5tel

Debian 7 wheezy (Original OS)

Seagate 3.5“ 1T, 7200 rpm

 

Results:

Orange Pi PC

 hdparm -t --direct --offset 500 /dev/sda

 Timing O_DIRECT disk reads (offset 500 GB): 102 MB in  3.01 seconds =  33.88 MB/sec

 

Lenovo

hdparm -t --direct --offset 500 /dev/sda

 Timing O_DIRECT disk reads (offset 500 GB): 308 MB in  3.00 seconds = 102.60 MB/sec

 

 

Dmesg reported several errors with the HDD(/dev/sda) when it was connected directly to the on-board usb.  I solved it by passing through a powered usb hub.  There appears to be current limits to the on-board USB.

Marcos A. S. likes this

Share this post


Link to post
Share on other sites

Is the OrangePi one worse than the OrangePi Lite for this task?

 

Nope, both are H3 devices, obviously the Lite needs an external Ethernet solution (I used a rather fast USB3 GbE dongle) so unless you do not improve network situation (OPi One has just Fast Ethernet like most H3 devices) you can't get the transfer speeds I achieved with Lite + external Ethernet dongle. The post you answered to contained this link: http://linux-sunxi.org/Sunxi_devices_as_NAS#Use_the_right_tools

 

So why not following the advices there to identify the bottleneck? Speaking about '1.2MBps' without telling what this number should be (local throughput test using tool X, network+storage benchmark using tool Y? 'Real world' throughput shown by Windows Explorer when using SMB? 

 

armv5tel

 

Sounds like Marvell Kirkwood 88F6281 as used also in older QNAPs and other NAS boxes. Has native SATA with ok-ish performance (given that most HDDs show rather low transfer rates when they are full -- see below for the relationship between used capacity and performance) but GbE performance somewhat sucks. So in case you do the obvious (use Orange Pi Plus 2E instead of Orange Pi PC or add at least an USB3/GbE capable USB dongle) Orange Pi PC might outperform the Kirkwood based NAS if you add GbE networking.

 

Your performance numbers are not really NAS related (that's always storage + network performance) and unfortunately of no great use. Using hdparm for benchmarking purposes is always wrong unless you ensure that you execute it 20 times and use an average value. A 'benchmark' that runs just 3 seconds will be heavily influenced by background HDD activity. And while hdparm tries to flush buffers before starting the 'benchmark', starting even only heavy write activity when hdparm already runs invalidates the read results hdparm shows (tested that many times, unfortunately forum search still sucks somehow so I can't reference numbers).

 

Then HDDs show the following 'real world' performance behaviour: When they're nearly empty all read and write operations happen on the outer tracks where performance is normally twice that good as on the inner tracks. This is called ZBR: https://en.wikipedia.org/wiki/Zone_bit_recording

 

So every HDD will show lower performance when it will be used as intended (filled with data) since all modern drives use a ZBR variant and fragmentation adds to the problem. Most if not all hdparm/dd benchmarks that can be found on the net ignore this so every time you find hdparm/dd numbers without telling something about the test setup simply forget about these numbers since they're 100 percent meaningless. :)

Share this post


Link to post
Share on other sites

@tkaiser

 

have you located cheap ebay/ali UASP enclosures  (2.5 or 3.5) ?

 

i'm not sure how to properly check that in linux or if my 990fx chipset has uasp support but the three recent usb 3 enclosures i bought don't seem to be uasp compatible.

 

Unfortunately i don't have a windows host at hand to double check that but i will next week.

thx

 

 

Share this post


Link to post
Share on other sites

have you located cheap ebay/ali UASP enclosures  (2.5 or 3.5) ?

 

Nope, not even tried. The data my disks have to store has some value therefor I'm not following the 'buy cheap, buy twice' route here. I only order enclosures where the chipset is known (and would return an enclosure that comes with a different bridge chip), the last times I got some enclosures from CSL via Amazon (seems to be re-branded Inateck stuff, at the time I bought the enclosures they were cheaper than Inateck)

 

So I would simply search for the chipsets on Amazon, eg. JMS567 and then test using an Armbian image with vanilla kernel and an A20 or H3 board. Attach the USB disk, then check dmesg output for 'uas':

dmesg | egrep -i "usb|uas"

Share this post


Link to post
Share on other sites

okay i've also found that Inatek/CSL/MantisTek enclosure on amazon (same prices around 10e), i've ordered one to do some tests.

 

Actually i got an Orico 3588US3 (3.5") from gearbest which is listed on amazon as an UASP enclosure, but it loads with "usb-storage" driver on my current kernel.

 

So i'm not quite sure my 3.13.0 kernel even has uas drivers, i see no mention of CONFIG_USB_UAS in the config (ubuntu 14.04).

And no "uas" module in my /lib/modules/, unless it was built in of course but usb-storage is not so i was expecting an uas driver somewhere.

Share this post


Link to post
Share on other sites

So i'm not quite sure my 3.13.0 kernel even has uas drivers, i see no mention of CONFIG_USB_UAS in the config (ubuntu 14.04).

And no "uas" module in my /lib/modules/, unless it was built in of course but usb-storage is not so i was expecting an uas driver somewhere.

 

Well, you seem to be talking about a x86 PC here? The USB controller used must support UAS as well and you find UAS support only on more recent USB3 controllers. USB Attached SCSI is an optional feature released at the same time as USB3 but can be combined with USB2 as well. But the only 2 USB2.0 host controller implementations capable of speaking UAS I know of are Allwinner's A20 and H3 (used with mainline kernel).

Share this post


Link to post
Share on other sites

yes sorry i was talking about my desktop amd64 kernel, i'm testing the enclosures there before moving one to my A20 or H3 boards for further tests.

 

i'll continue investigating my desktop UAS support.

Share this post


Link to post
Share on other sites

i'm wondering what kind of mainline images are you using on your h3 devices  ?

I haven't seen any on the armbian download pages, are there some beta images i could test ?

thx

Share this post


Link to post
Share on other sites

i'm wondering what kind of mainline images are you using on your h3 devices  ?

 

Explained in post #8 above ;)

 

Might work or not for you but IIRC you have a Beelink X2 (prone to overheating) so you should be careful not to overload the system later since no THS (throttling) is implemented in mainline kernel for H3 yet!

Share this post


Link to post
Share on other sites

thx sorry i missed that

 

i'll test on my opi pc (with smallish heatsink)

well now that you say i could kill my x2,  i must say i'll probably be tempted to test it on it too =)

Share this post


Link to post
Share on other sites
 Orange Pi Plus 2E is -- in my very personal opinion -- the best H3 device available if you think about NAS useage.
 

 

 

 I am glad to read this as I recently purchased one for this very purpose.
 
Unfortunately I discovered (from your review) too late that "sata" really means sata-over-usb, because my main criteria was "cheapest given gigabit ethernet + either SATA or usb 3.0". I envisioned attaching it to drives over sata for a home NAS solution with good throughput. Am I misunderstanding what is possible, or do you just have to different class of boards that aren't going to be bottlenecked by USB 2.0? (e.g. ODROID xu4 for instance would have significantly more throughput?)
 
What is the benefit of gigabit ethernet when you are so bottlenecked by USB 2.0?

Share this post


Link to post
Share on other sites

"cheapest given gigabit ethernet + either SATA or usb 3.0"

 

Then you should search whether you get a SheevaPlug on eBay (but don't be surprised that performance will suck since having Gbit Ethernet and SATA does not mean that full speed is possible -- the SoC is too weak). Another attempt would be to read through http://linux-sunxi.org/Sunxi_devices_as_NAS

 

And if you want an ARM device that can fully saturate up to 3 SATA 3.0 channels while also being able to saturate several GbE links then the Clearfog Base would be an option (or cheap x86 designs ;) )

Share this post


Link to post
Share on other sites

Hi

 

How many 2.5 HDD does the OPi Plus 2E support? (using a 3 Amp power source)

 

I have a banana pi (2 Amp power source) with two HDDs: one 2.5" and the other 3.5" and can't add a third 2.5" disk.

 

In the long run I will like to migrate all the disk to 2.5" HDD so there is no additional power bricks.  

Share this post


Link to post
Share on other sites

How many 2.5 HDD does the OPi Plus 2E support? (using a 3 Amp power source)

 

You're talking about letting OPi Plus 2E power the disks? And you're talking about modern 2.5 SATA notebook HDDs (in opposite to some 2.5" server disks that need pretty much power all the time)? :)

 

Normal notebook disks consume not that much even in active state. The problem is spin-up since then they might consume 1W peak consumption (check datasheets but don't trust them too much). I managed to let 3 disks (2 HDD, 1 SSD) be powered through an OPi PC maybe half a year ago. But only if I added them one after the other to delay consumption peaks. Letting all three attached and starting the board didn't work.

 

Some disks support delayed/controlled spin-up (Staggered Spin-up: SSU and Power-Up In Standby: PUIS) but since we're talking about USB here it's then also a matter which USB-to-SATA bridge is used inside the enclosure (has to support SCSI / ATA Translation) and how kernel/drivers behave so in other words: better forget about it. If you're lucky it might work but it's better to expect the opposite.

 

With your Banana Pi (A20 based) it's most probably undervoltage you're running into unless you use the SATA power port to power the Banana (and using one external PSU to power both SATA disk and board). But that's a different story :)

 

BTW: Using an insufficient power supply to power a couple of disks is the best way to constantly run into weird problems.

Share this post


Link to post
Share on other sites

I'm planning to go down this route too with a couple of OPi Plus 2Es en-route from china now. Since I'm powering both the boards and the drives with a desktop PC PSU I didn't want to use HDD enclosures. Thankfully I've found USB 3.0 to eSATA adapters that're pretty much perfect for the job. 

The NAS aspect is a little different though, I use Ceph at home which is a distributed storage system so individual node performance doesn't matter much so long as there are a lot of them. I've been having problems with it crashing on my previous A20 powered board but I'll figure that out soon.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
6 6

  • Support the project

    We need your help to stay focused on the project.

    Choose the amount and currency you would like to donate in below.