Have "Supported" boards been "Torture-Tested" for storage/disk-IO?

[esbeeb]

Use case: simple NAS

Boards to consider: anything with at least UASP, better yet, USB 3.0 or SATA.  But stability (under heavy disk I/O and load average) matters over speed!  Also, Gigabit Ethernet is a must.

 

For all the SBC's which are listed as Supported, to what degree can I trust that I won't get kernel freezes, or filesystem errors, when disk I/O gets thick and furious?  I completely understand if the disk I/O slows down to a crawl, as the load average gets high, but I'm wanting to feel assured that a board marked as Supported won't "flake out" on me, potentially destroying all my files, should I decide to use it for a simple NAS.

 

Here is how I would personally try to torture-test a board, for example:

- Be watching kernel messages. 

- Also be watching htop, to see the load average.

- Share out a folder (on an externally attached UASP USB hard drive adapter, like the one @tkaiser always uses, which has an SSD drive connected) over SAMBA, having been formatted with an EXT4 filesystem.  Then from a remote machine, copy a huge number (say, 20 GB worth) of tiny files (like a large ebook collection from Calibre, ie. the "Calibre Library" folder) into the NAS into that SAMBA share.

- Have the root filesystem on an eMMC on the NAS (or good SD card with fast random I/O).  Then run a command like "sudo updatedb" to create lots of disk IO on the root filesystem.

- Just to be bad, have a USB stick with a VFAT filesystem on it, and some files which aren't important.  Connect that stick also, then while doing some file operations on it, just rip the stick out, to create some USB-related kernel errors (knowing fully well it will corrupt that VFAT filesystem).  Does this affect the SAMBA share over on the other USB-attached drive?

 

How would you do a torture test of storage I/O? 

 

Can anyone recommend a "Supported" Armbian board (as above, ie. having at least UASP, better yet, USB 3.0 or SATA) with a recent kernel, which would stand up to such a torture test, without freezing up/flaking out/destroying the filesystems on the root or external EXT4 drive?

 

I would want this sort of assurance before trusting any "Supported" board for being suitable for a simple NAS, having rock solid stability for disk I/O.

[hjc]

I personally never tried to rely on USB for stability, thus never put any important files on a USB drive. I do attach some USB HDDs on SBC and share them using Samba, but they are only used to store music & video files.

Now my (not so) "simple" NAS is a NanoPC T4 combined with 1TB NVMe SSD. Have Samba and Owncloud server syncing ~30k files on several PCs, never seen an error.

[tkaiser]

23 hours ago, esbeeb said:

I would want this sort of assurance before trusting any "stable" board for being suitable for a simple NAS, having rock solid stability for disk I/O.

 

Since I know you're coming from the worst SBC for the NAS use case possible (something called Raspberry Pi) let me elaborate why this device is so lousy and why (former) RPi users think 'if it's running with ARM it's unstable as hell' (which is the reason why I call the RPi a shame for the whole ARM ecosystem since you can not convince professional users to stop wasting energy with huge x86 server boxes and switch to energy efficient ARM designs since they all had already experiences with these crappy RPi toys and think the RPi experience would apply to all ARM boards)

 

The Raspberry Pi hardware is a total fail if you want to do anything related to storage or network with this thing.

• The 'engineers' chose the most crappy way possible to power the board (Micro USB) which in itself is responsible for a huge amount of instability reports. Use an average charger and an average MicroUSB cable and some increased load will result in your RPi toy freezing or shutting down. The problem is usually called 'voltage drop' and affects connected peripherals as well (a host powered 2.5" disk usually doesn't like it when available voltage drops below 4.75V)
• All RPi are based on a SoC which is no real ARM SoC but a VideoCore IV with some crappily integrated ARM cores. This also affects USB functionality since the single USB controller is part of the VideoCore and it required huge amounts of fancy workarounds to get USB working in Linux 'reliable' or at least at all
• Since there's no network and only one USB2 USB OTG port on all those RPi with network and USB-A receptacles everything is behind an internal USB hub that also integrates Ethernet. You can't implement it 'better' if your goal is 'as unrealible as possible'. But that's all what the RPi Trading guys had since they chose the wrong SoC in the first place (most probably since Mr. Upton worked in the BroadCom division that was responsible for these obsolete VideoCore IV SoCs back then)
• Can it get worse? Sure, since there are not only voltage drops leading to problems but also undercurrent is a typical phenomenon with Raspberries. There are current limiters in place which cut power to here and there if this or that happens. See the great example of their new PoE HAT which is such a bizarre fail from an electronics point of view that I miss words.
• Since we're already at QA: remember how they decided to implement Gigabit Ethernet on their latest RPi 3 B+? By not testing anything prior to product launch and later immediately blaming the supplier they chose instead of admitting that the designer is to blame for choosing inappropriate components that then receive zero testing?

What can we learn from this RPi fail?

• Reliably powering the device(s) is the most important part. MicroUSB is crap for this purpose since it encourages users to run into undervoltage and undercurrent problems. With USB-C it starts to be the same depending on the implementation. 5V and high currents --> voltage drop resulting in unreliable operation
• USB storage is unreliable by definition. USB3 and especially the USB3-A receptacle made this even worse. If you love cable and contact problems, only then USB3 with USB3-A receptacles is for you. Adding to this the USB3/XHCI implementations in most ARM SoCs are somewhat crippled and there exists a bunch of problematic USB gear out there that causes trouble (e.g. Seagate USB3 disk enclosures -- this problem is solved in Armbian but x86 Linux users suffer from the same shitshow again and again --> recent example)

So what to choose if you're after reliable NAS operation?

• Avoid stuff that wants to combine high currents and low DC-IN voltages. Boards with 12V or wide range input are more reliable
• Avoid USB, especially USB3 and there especially storage interconnects using crappy USB3-A (due to ultra tiny SuperSpeed contacts inside jack and receptacle)
• If you want to choose a SoC that's only capable of USB then at least avoid cable/contact issues (see how ODROID HC1 and HC2 solved this problem)
• If you want to rely on USB storage avoid hubs in between host and disks (I've tested just recently one board that at least with only one disk behind the hub operates well, all other tests in the past were simply horrible)

Once you followed this list you don't need to worry about stability. Linux is stable, it runs on tons of servers. But those things use appropriately sized and stable PSUs and not crappy phone chargers that were bought as cheap as possible and are connected with tiny wires. They use storage implementations that do not suck (no USB and especially no crappy connectors involved and no complex topology like USB hubs in between).

[esbeeb]

36 minutes ago, hjc said:

I personally never tried to rely on USB for stability, thus never put any important files on a USB drive. I do attach some USB HDDs on SBC and share them using Samba, but they are only used to store music & video files.

Now my (not so) "simple" NAS is a NanoPC T4 combined with 1TB NVMe SSD. Have Samba and Owncloud server syncing ~30k files on several PCs, never seen an error.

A vote of confidence from a reputable poster like you means a lot to me.  I too feel a bit leery about sharing out files over a USB-attached drive, however since many of these SBC's don't have true SATA controllers like a PC would (but merely put the SATA behind USB), then you often sort of can't escape sharing out files over a (very-large-capacity) USB-attached drive, wouldn't you agree?

[esbeeb]

38 minutes ago, tkaiser said:

So what to choose if you're after reliable NAS operation?

• Avoid stuff that wants to combine high currents and low DC-IN voltages. Boards with 12V or wide range input are more reliable
• Avoid USB, especially USB3 and there especially storage interconnects using the crappy USB3-A method (due to ultra tiny SuperSpeed contacts inside jack and receptacle)
• If you want to choose a SoC that's only capable of USB then at least avoid cable/contact issues (see ODROID HC1 and HC2)
• If you want to rely on USB storage avoid hubs in between host and disks (I've tested just recently one board that at least with only one disk behind the hub operates well, all other tests in the past were simply horrible)

Once you followed this list you don't need to worry about stability. Linux is stable, it runs on tons of servers. But those things use appropriately sized and stable PSUs and not crappy phone chargers that were bought as cheap as possible and connected with tiny wires. They use storage implementations that do not suck (no USB and especially no crappy connectors involved and no complex topology like hubs in between).

OK, this is super juicy and helpful.  Having followed this advice, what then are we left to choose from, for boards (if any), which are Armbian-"Supported"?  That is to say, which "Supported" Armbian boards have true SATA (not behind USB), and have Gigabit Ethernet? 

 

Umm, I see the Helios4 board is marked "Supported", but they are only taking "pre-orders" to buy one of these, and it looks like you might need to live in Europe to get one, not sure if they ship to North America or Asia.  I also see the Espressobin is currently "Supported", but I've heard about stability issues on that board.  Does anyone know if the SATA on the Espressobin is "behind" USB? (I'm guessing no.)

 

To me, those are the only boards I'd want to consider for a NAS. @hjc

suggests the NanoPC T4, but that's not yet a "Supported" board.  Also, the NanoPi M4 which you linked to, also is not yet a "Supported" board, @tkaiser

@hjc

[hjc]

1 hour ago, esbeeb said:

To me, those are the only boards I'd want to consider for a NAS. @hjc

suggests the NanoPC T4, but that's not yet a "Supported" board. 

Actually the 4.4 kernel is quite stable, however Armbian development on this board is still in early stages, so it's marked as WIP. It means many optimizations may have not been applied yet (e.g. interrupts), and many configurations may be changed later (e.g. board family change requires a manual upgrade). If you are an experienced Linux user, this may not be a problem.

 

There are a few other boards with fast native SATA or capable of PCIe SATA supported by Armbian, e.g. Helios4, EspressoBin, Clearfog Pro.

[tkaiser]

On 9/16/2018 at 9:34 AM, esbeeb said:

That is to say, which "Supported" Armbian boards have true SATA (not behind USB), and have Gigabit Ethernet?

 

USB in an 'controlled environment' isn't necessarily a bad thing. Take a look at ODROID HC1 and HC2. Both networking and SATA is USB3 based but no cable/contact hassles any more since the chips are on the PCB and with HC2 also powering problems are gone (due to 12V powering).

 

The boards I personally consider for NAS and small server use cases are

• Olimex Lime2 (slow but the 'true UPS mode' is a big plus -- just add a battery and your whole server with connected 2.5" SATA disk runs for hours even if disconnected from mains)
• NanoPi NEO2 or Orange Pi Zero Plus are both fine if you don't need high performance and use the vendor's NAS add-ons (then you get no USB contact/cable hassles and powering also is no problem)
• ODROID HC1 or HC2
• EspressoBin (though still not totally convinced since at least my board shows an idle consumption that is IMO way too high)
• Clearfogs Base/Pro and Helios4
• RockPro64
• NanoPC-T4
• NanoPi M4 (especially if @mindee starts to evaluate a 'NAS HAT' allowing to power the board + 4 3.5" disks with a 12V PSU and there's a 4 port SATA PCIe controller on this thing)
• NanoPi NEO4 (more 'pocket server' than NAS of course)

[TonyMac32]

5 hours ago, tkaiser said:

NanoPi M4 (especially if @mindee starts to evaluate a 'NAS HAT' allowing to power the board + 4 3.5" disks with a 12V PSU and there's a 4 port SATA PCIe controller on this thing)

Uhhh, give me a few days.  :-)  Of course that is one hell of a wishlist you've got there. 

[tkaiser]

8 hours ago, TonyMac32 said:

10 hours ago, tkaiser said:

NanoPi M4 (especially if @mindee starts to evaluate a 'NAS HAT' allowing to power the board + 4 3.5" disks with a 12V PSU and there's a 4 port SATA PCIe controller on this thing)

Uhhh, give me a few days.  :-)  Of course that is one hell of a wishlist you've got there. 

 

Why? In fact currently only boards with mPCIe would qualify (there you can add a SATA mPCIe card which will always be bottlenecked by the single PCIe lane) or maybe M.2 key M. I found such a M.2 key M SATA card but there are three issues: stuff on the bottom PCB side making it impossible to insert it into a M.2 SSD slot due to height constrains, those SATA cards are shorter than 80mm and then still powering attached disks is needed.

 

So as already suggested: Such a HAT for the M4 would contain the PCIe attached SATA controller and 12V/5V DC-DC circuitry. And currently NanoPi M4 is the only board where something like this could work since

• SoC fast enough (hopefully also with HW accelerated transcoding support soon)
• PCIe x2 available on pin header
• SoC on the correct side of the PCB so no heat issues unlike on all the other RK3399 boards

It's not only about 'true SATA' (as many call it -- though native SATA like on EspressoBin, Clearfogs and Helios4 is something different than PCIe attached SATA here) but the main point is if attached drives can be powered reliably. 2 x 3.5" with RK3399 is already covered by RockPro64 (there is a 5V/12V power header on the board for disks), NanoPi M4 could make this possible for up to 4 3.5" HDD.

[TonyMac32]

10 minutes ago, tkaiser said:

but the main point is if attached drives can be powered reliably.

That was most of the point.  4x drives is getting to serious supply territory, I would guess 60 Watts or so (total, not just drives) without actually doing any math (so margin of error probably decently high).  Without driving significant cost I wonder about all of that fitting on a hat comfortably.  (I'd also love to see it, do not misunderstand)

[tkaiser]

20 minutes ago, TonyMac32 said:

4x drives is getting to serious supply territory, I would guess 60 Watts or so (total, not just drives) without actually doing any math (so margin of error probably decently high).  Without driving significant cost I wonder about all of that fitting on a hat comfortably.  (I'd also love to see it, do not misunderstand)

 

Well, since I've not the slightest idea about electronics (software guy) I never thought about that. But my naive assumptions as follows:

• 12V is 'pass thru' since this is something only the spinning motors on 3.5" HDD need. There's a high peak consumption when drives spin-up, once the platters rotate this current on 12V rail can be pretty low
• 12V --> 5V needed for the board itself (15W?) and for the HDD's 5V input (logic and stepper motors to move the heads).

No idea what's the highest 5V current specified by any 3.5" HDD datasheet but what I looked into not exceeded 0.7A. So most probably talking about 30W for DC-DC circuitry? The Helios4 ships with an 12V/8A PSU, DC-DC conversion happens on the mainboard and at least I'm not aware of any power related issues yet.

[TonyMac32]

8 hours ago, tkaiser said:

No idea what's the highest 5V current specified by any 3.5" HDD datasheet 

Well, the hat would have to account for both 3.5 and 2.5, since either could be plugged in.  The 2.5 SSD I have in my desk wants 1.6 Amps at 5V, so 8 watts.  4 of those is now demanding 32 Watts themselves, then you have to consider efficiency, so 40 watts isn't an insane number.  

 

8 hours ago, tkaiser said:

12V is 'pass thru' since this is something only the spinning motors on 3.5" HDD need.

Yes, certainly.

 

 

[chrisf]

14 hours ago, TonyMac32 said:

4x drives is getting to serious supply territory, I would guess 60 Watts or so 

You'd want drive spin-up sequencing for 2 or more 3.5" drives running from a 60W PSU. You'd want at least 30W per drive for spin-up. You'd need to do that for initial power up and when waking from sleep.

 

15 drives peaking at 420W: https://www.45drives.com/wiki/index.php?title=Start_up_Power_Draw

Those were 5,900rpm Seagate drives, with a startup rating of 2A.

[TonyMac32]

7 hours ago, chrisf said:

You'd want drive spin-up sequencing for 2 or more 3.5" drives running from a 60W PSU

Right, spinning disks weren't on my mind at the time.  Timing the power sequence seems like a pain.

[esbeeb]

On 9/17/2018 at 3:37 AM, tkaiser said:

NanoPi NEO2 or Orange Pi Zero Plus are both fine if you don't need high performance and use the vendor's NAS add-ons (then you get no USB contact/cable hassles and powering also is no problem)

Thanks so much.  Knowing those boards have NAS-related HATs makes a huge impact on my decision-making.  When viewing the pages for the NanoPi NEO2 or  Orange Pi Zero Plus , there was no obvious way of knowing that they had custom NAS HATs made by the vendors.  Not to complain, but these decision-swaying NAS HATs were not listed as some kind of tag (you know, in the blue boxes under "Specifications", saying something like "NAS-HAT"), nor were these HATs listed in the "Tested 3rd Party Hardware" section.

 

The fact that these HATs take care of the powering and cabling of the attached drives matters a lot to me as well.  I could mess around with soldering irons, hobby glue guns, etc, but I'd far rather not.  I wish these SBC Vendors wouldn't be so excruciatingly cheap about shaving a few dollars off the pricetag (with such cost-cutting measures, but then making a hassle like this for the end user to solve on their own).  There's a big difference between "Cheap and Cheery", and "Cheap and Grumpy", IMHO.  For example, I'm feeling disappointed that the Espressobin and the Helios4 leaves it up to the end user to make/buy their own molex-to-SATA power cable (are these cables even for sale anywhere?).

 

Just a possible Wish item... please mention these NAS HATs somehow on those Armbian pages: for the NanoPi NEO2 or  Orange Pi Zero.

 

PS: You Armbian guys are awesome, please keep up the good work.

 

PPS: You're going to hate this, @tkaiser, but I file the Raspberry Pi under "Cheap and Cheery".  I'm also trying to find out which NAS-friendly boards I'd also file under "Cheap and Cheery" here.  So far, for example, I'm really liking the NanoPi NEO2 and Orange Pi Zero (plus NAS HATs), because the H5 SoC's are well-supported in the mainline kernel, and are currently Armbian-"Supported".

[Igor]

30 minutes ago, esbeeb said:

please mention these NAS HATs somehow on those Armbian pages

 

Nice idea, but until I am the only one maintaining whole data on the website, there is less than zero possibility for this kind of expansion.

 

Maintainers. Maintainers. Maintainers. Then we can talk about.

[devman]

19 hours ago, esbeeb said:

For example, I'm feeling disappointed that the Espressobin and the Helios4 leave it up to the end user to make their own molex-to-SATA power cable (are these cables even for sale anywhere?).

 

Umm.. the Helios4 kits (full or just the board) come with all the necessary cables.  SATA power, SATA data and AC adapter.   The only thing you need to source yourself is the drives.

[TonyMac32]

https://www.newegg.com/Product/Product.aspx?Item=N82E16812200061

 

[esbeeb]

4 hours ago, devman said:

 

Umm.. the Helios4 kits (full or just the board) come with all the necessary cables.  SATA power, SATA data and AC adapter.   The only thing you need to source yourself is the drives.

Thanks!!