Best Chipset for NAS?

[cmirra]

Hi All,

 

Which SOC gives best performance today for SATA I/O?

According to the Sunxi wiki: http://linux-sunxi.org/Sunxi_devices_as_NAS

"Since not every A20 based sunxi device uses GMAC networking (or even Ethernet at all) the following devices are the best choices: A20-OLinuXino-Lime2, Banana Pi, Banana Pro or Banana Pi M1+, Cubietruck, Hummingbird A20, Lamobo R1, Orange Pi, Orange Pi Mini or pcDuino3 Nano (Lite)."

I've actually got a cubietruck+ here (that uses a terrible USB to sata bridge) but it's a terrible choice. Hoping you guys (or tkaiser ) can steer me in the right direction. I could run out and order a banana pro, but I feel like A20 has been around a while and maybe there's something better today? Clearfog is looking nice, but I guess I'd need a pci-e sata card.

 

This isn't 'really' a NAS -- The actual use case is as follows:

 

 

 

As you can see, the inbound data will come from an OTG connection running g_mass_storage gadget to appear as usb disk to host pc. This is essentially a raid1 array that is decrypted via gpio keypad, and made available as a usb drive to another system. Hardware accel for the encryption would of course be welcomed.
 

I'll need something that doesn't use a usb->sata bridge, because usb will likely be tied up transfering inbound data -- and even if it wasn't they seem terribly slow. A good support community would of course be nice too -- The H8/a83T just doesn't seem to have it yet.

 

Any input would be appreciated.

 

Thanks all,

CMirra

[tkaiser]

Well, since this is really not NAS but USB gadget instead you will most probably be limited by slow g_mass_storage mode. The CT+ you chose is not that brain-dead like a Banana Pi M3 (there USB hub and SATA bridge share the same USB host port, on the CT+ the hub is connected to the SoC's HSIC port and the SATA bridge uses an own USB host port) but it's the wrong device for that since USB ports have to share bandwidth.

 

If I were you I would choose a quad-core device with 2 or 3 real USB host ports that has no thermal problems (Orange Pi PC, PC Plus or Lite). Crypto acceleration with H3 seems to not perform 'that good' but maybe H3 is beefy enough for the job.

 

BTW: I don't know whether it's possible to use Marvell's CESA crypto engine right now in Linux. But at least you can use mechanical adapters to convert M.2 and MiniPCIe slots of the Clearfog into normal SATA ports (needs some u-boot voodoo to configure the SERDES lanes to not carry PCIe but SATA instead but this information is available already)

 

Regarding 'community' around H8/A83T unfortunately you're right. There is none (just a bunch of clueless customers that still don't want to understand that they bought paperweights) and OS software/support situation is really horrible. Just as an example: https://github.com/cubieboard/Cubietruck_Plus-kernel-source/issues/2#issuecomment-244859306

[cmirra]

Thanks a ton for the info. Not just here, but everywhere. Doing homework on all this for the last few weeks and your posts in particular came up often.

 

What kind of transfer speeds would be safe to expect with the setup you've suggested? Would there be a huge difference between doing this on an H3 and an A20? A20 seems to have tons more support at the moment. I'm shooting for 35-60MB/s if possible.

 

I'll start looking at the Orange Pi as its price point is a lot easier to play with than the clearfog.

[tkaiser]

What kind of transfer speeds would be safe to expect with the setup you've suggested?

 

http://forum.armbian.com/index.php/topic/1089-usbootpi/

 

Most probably you will be bottlenecked by the USB 2.0 connection and need CPU cycles for encryption. Therefore I would assume H3 will be faster overall even if there's no SATA.

[cmirra]

Here are of the results of using g_mass_storage module on jessie w/orangepi pc hardware. Data is coming from OTG and being written to usb3 drive (on usb2.0 bus) without encryption or raid.

 

 

'top' shows that im only using 1-2% of the cpu, so I'd imagine I have room for decryption and raid overhead.

 

Going to rebuild the kernel a few more times over the next week and try to increase the speed. Not sure where the bottleneck is but I know there are lots of usb modes/flags that I'm not modifying yet. I'm running full desktop version of debian, my next step will be scaling down to just the console version next.

 

For anyone reading this down the road, I had lots of issues getting g_mass_storage to work. Turned out adding "stall=0" to the modprobe string (as mentioned in other posts) made it run just fine.

[tkaiser]

I'm running full desktop version of debian, my next step will be scaling down to just the console version next.

 

Often recommended but absolutely useless in 2016. This was a nice trick 25 years ago when Unix machines had a single 50 MHz CPU core and ran with 8MB of RAM. If you want to improve performance then simply switch to true headless mode that means on your OS image deactivating GPU/HDMI with 'h3consumption' tool (and since then running a desktop is pretty useless, doing an 'systemctl nodm disable' might be an idea).

 

Switching off GPU/HDMI slightly increases memory bandwidth which might help a litte with USB throughput. Then you have to keep in mind that SD card access is limited to ~23MB/sec anyway so without using an USB based RAID-0 and performance governor I would not even start to further tune USB settings (if you then found the best combination switch back to interactive and RAID-1 again -- for whatever reasons RAID should be useful).

 

And a last check would be memory reservations. When you run true headless you don't need any reservations which gives you the full 1 GB RAM (might speed up operations over time if Linux can make use of more FS cache). I would check /boot/boot.cmd and kernel settings (and have to admit that I don't know in detail where to adjust what -- please keep us informed about progress being made)

[tkaiser]

BTW: Since it seems you bought already an CT+ and this 'famous' HDD-RAID subboard (a combination that shows worst storage performance ever) would you mind providing performance numbers to this thread: http://forum.armbian.com/index.php/topic/1925-some-storage-benchmarks-on-sbcs/

 

Simply set up a 'hardware RAID-0' there, create an ext4 FS with default settings and then run the following after doing a cd to the RAID-0:

iozone -e -I -a -s 1M -r 1k -r 2k -r 4k -i 0 -i 1 -i 2
iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
iozone -a -g 4000m -s 4000m -i 0 -i 1 -r 4K -r 1024K

Sequential performance will be bottlenecked by crappy GL830 USB-to-SATA bridge but I wonder how much random IO will suffer with such an insane setup. IMO it's important to provide such numbers to show how much a device that is considered faster (octa-core! Still SATA!) sucks in reality. In case you find the time for this test repeating it with Orange Pi PC running with mainline kernel, btrfs raid-0 and UASP capable disk enclosures would be great (since the cheap and 'slow' H3 device will outperform the overprized CT+ and it's weird RAID card by magnitudes)

[hojnikb]

Wait for boards, that use this chipset.

http://www.cnx-software.com/2016/09/12/hisilicon-hi3798m-v200-soc-to-deliver-cheaper-4k-tv-boxes-with-sata-gigabit-ethernet-andor-usb-3-0/

 

gigabit ethernet and sata/usb3

[cmirra]

BTW: Since it seems you bought already an CT+ and this 'famous' HDD-RAID subboard (a combination that shows worst storage performance ever) would you mind providing performance numbers to this thread: http://forum.armbian.com/index.php/topic/1925-some-storage-benchmarks-on-sbcs/

 

CT+ Raid board just arrived in mail yesterday, so I spent a few hours last night with it to try to get your stats.

 

Unfortunately, I was unable to make it work with any of their images -- and I couldn't get their toolchain working properly (even in a 12.02 64 vm as recommended), even after a few tries. I'll likely try it on the OPi PC hardware tonight to verify it works if possible, then hammer away on my CT+ paperweight again

 

I spoke with support about the board a week or so ago (before I ordered it) and was told it will also work with any computer via usb as well, but I couldn't get my desktop to detect any change in USB devices or raid controllers. Not even a usb sound, but I suspect it may only work via usb/pc in some, but not all of it's modes.

 

Speaking of the raid modes the board supports -- FYI: Modes are not configurable by software, or even jumper -- but by soldering resistors to fairly tiny points on the board. At the moment, I wouldn't recommend this to anyone.

 

Still looking at the clearfog pro for my application, will let you guys know when/if I pull the trigger. In the meantime, I'm still open for suggestions, though I'm mostly happy with the pricepoint on clearfog. You get what you pay for and it's not much more expensive than these 'toy' boards we play with.

 

@hojnikb -- Thanks for the suggestion. Will check it out.

 

Update: Posted stats to the above mentioned thread.

[Nova]

Cmirra do you have a raid module from cubietruck?

 

Please say from the community a new impression for this module please.

[Jens Bauer]

I don't know if this is useful at all, but I'd like to toss it in anyway.

 

For a long time, I've been interested in having a Western Digital My Cloud EX4100 (the one with the Armada chipset).

Marvell's Armada is definitely something.

If I could choose what I'd like on my board, it'd be an Armada 8040 plus a 88SE9230 (4 ch. HyperDuo RAID controller).

-The 88SE9230 requires PCIe, and unfortunately very few SoCs have PCIe. some of the i.MX have PCIe, some of TI's serious SoCs also; many of Marvell's have, Samsung's have, but very few (if any) of the chinese SoCs have PCIe.

(I'm sorry about those I forgot or don't know about.)

 

I've also had my eyes on a ClearFog for a long time - I might end up getting one.

[tkaiser]

I've also had my eyes on a ClearFog for a long time - I might end up getting one.

 

Have you already seen this low-end Marvell announcement (now on Kickstarter): http://www.cnx-software.com/2016/09/23/marvell-espressobin-board-with-gigabit-ethernet-sata-pcie-and-usb-3-0-to-launch-for-39-and-up-crowdfunding/

 

I would prefer this since I would never ever implement SOHO/home RAID (again). ZFS+raidz2 is ok for more than 2 disks, everything else stinks (especially all those cheap HW RAID chips) if you don't have every SPoF lying around (since otherwise the R in RAID becomes a joke)

[tkaiser]

LOL, and there is the MACCHIATOBin: http://www.cnx-software.com/2016/10/11/solidrun-macchiatobin-is-another-marvell-armada-8040-networking-mini-itx-board/

[Jens Bauer]

I would prefer this since I would never ever implement SOHO/home RAID (again). ZFS+raidz2 is ok for more than 2 disks, everything else stinks (especially all those cheap HW RAID chips) if you don't have every SPoF lying around (since otherwise the R in RAID becomes a joke)

 

Is anything wrong with the Marvell 88SE9230 HyperDuo RAID controller ?

-I know that it's easier to "recover" a smashed RAID when using a software RAID, however, the hardware RAID is much quicker; especially the 88SE9230, since if it detects a SSD, it uses the SSD as a temporary storage.

I've noticed that Western Digital recommends Marvell chipsets.

 

In fact, a router+firewall should not run under an operating system at all; it should be stripped down to its simplest form: network only.

That would make it less likely that bugs are present, it would be possible to avoid memory fragmentation completely (if the memory handling is written properly), no accidental runaway-syslog, no background processes running by accident, etc.

 

But my main reason for wanting a HyperDuo RAID system is for other development.

 

I really think that the MACCHIATOBin looks interesting. It might cost almost 10 times as much as the ESPRESSOBin, but the price is only 2/3 of the Sinclair ZX Spectrum when it first arrived in Denmark in 1983.

(Yes, it was from 1982, but we didn't see it until 1983).

One could use the MACCHIATOBin as a desktop computer if adding a graphics card (unlike most chinese Cortex-A, the 8040 does not have built-in HDMI support).

 

Thank you for pointing me to those yummy-boards. :9

[tkaiser]

Is anything wrong with the Marvell 88SE9230 HyperDuo RAID controller ?

 

Sure, the same as with every other hardware RAID controller. It's useless to waste time on trying to explain the problem and it's easy to learn what's wrong with that approach as soon as the controller stops working. You have then 2 or more disks with data on it you can't access any more. Home/SOHO RAID simply doesn't deliver the R in RAID unless you have spart parts lying around and a lot of experiences with exactly that RAID card you use (BTDT and given up on RAID hardware completely almost 15 years ago -- we contract with other IT houses for that who are happy dealing with catastrophes)

 

Also it's 2016 and not 1996 any more. We have modern FS with nice features and unless I would need business continuity (eliminating every single point of failure which is not possible with small crappy SOHO RAID) if I want to throw in more disks than necessary then I'll use them for backup (in fact I do regular snapshots on a ZFS mirror and then send them incrementally to totally different locations using 'zfs send/receive' -- same can we do on Linux with btrfs now)