Research PC Engines APU boards as distributed storage nodes

[legogris]

I'm currently on the hunt for the cheapest decently reliable board to use for building a glusterfs cluster. Requirements are simple: GbE, SATA port, arm64, preferably low power.

While helios64 is obviously the way to go for a NAS when it lands, it's totally overkill for a setup like this and I just realized that ODROID HC2 won't fly due to being 32-bit only.

Most other 64-bit boards require expansion boards for SATA, which drags up the total cost.

Finally I realized that the PC Engines APU boards fit all criteria, and come with what I understand to be performant GX-412TC CPUs.

This one, for example, for 86EUR (maybe a bit on the pricey side, but I haven't found better actually): https://www.pcengines.ch/apu2d0.htm

 

 

These boards are generally only used as router boards, and I am actually already using one as a router (which prevents me from doing tests myself right now unfortunately :P)

 

Does anyone have any idea of if this is a Bad Idea or worth a shot? Looking at the specs I don't see an obvious reason why this wouldn't work.

 

EDIT: While I guess an interesting topic to use these boards for other purposes, I realized now that the Espressobin looks more suitable at smaller size and lower cost at ~60EUR + shipping, so I think that's the best bet for now.

 

EDIT2: Seems like GlusterFS has higher requirements than I thought and performs poorly with Gluster; this user got a measly 35Mbps (albeit with 5 disks): https://www.reddit.com/r/DataHoarder/comments/92zwct/espressobin_5drive_glusterfs_build_follow_up/. Redhat recommends 16GB minimum but surely it could be reasonable for home use with less... https://access.redhat.com/articles/66206

The APU has 2 instead of 4 cores and comes with in a 4GB version over the Espressobin's 2GB, so maybe it's the better alternative after all.

 

Edited February 6, 2020 by legogris

[Heisath]

Hi,

 

I recently built a NAS using the APU2E4 (https://pcengines.ch/apu2e4.htm) which gives you 4 cores (I think all APU2 have 4 cores so smaller versions should work just fine), and 4GB RAM. 3 Gbit LAN, 1 SATA, 2xmPCIe (which give you up to 8 SATA), and mSATA (you could use a breakout board or some actual mSATA drive for ssd / os). It is working pretty good. Installation was a breeze, stock debian with mainline kernel works nicely. Thanks to being x64 and not some ARM all hardware worked out of the box.

 

Performance using ext4 harddisk drives and LAN peaked at about 120MB/s write (which is the max of the drives I assume). I did not test any cluster fs.

 

Remember that you might need to build your own cooling solution, the boards ship only with some "heatspreader" (aka alu plate) which normally transfers heat to their enclosure. 

 

Regards,

count-doku

[TRS-80]

Allow me to interject...

 

OP did not mention this as part of his criteria, however as the token bearded Free Software advocating hippie around here , I will weigh in from that standpoint...

 

I looked at these for possible router and/or NAS applications, but when I realized they were Jaguar based, they fell off my radar (my understanding is this is within the time frame where the Intel IME bullshit began).

 

However on this board I am not sure how this would be implemented, if at all. I could not find any further info on this, but I did not dig very deep, either (I basically stopped digging once I read "Jaguar").

 

If that is not the case, I would surely be more much interested in this platform, as OP (and others) found out the hardware specs look quite nice. As I have been looking for some time for NAS (and more recently, router) devices.

Edited February 6, 2020 by TRS-80
clarify

[Heisath]

@TRS-80you did realize those are AMD processors, right? So there's probably no Intel ME in there

 

In case you were worrying because AMD processors might contain the same, well true. Regarding the firmware for the board (which is coreboot) you can check the firmware https://github.com/pcengines/coreboot/ maybe this helps?

 

 

[TRS-80]

Yes of course, AMD equivalent of Intel's IME bullshit (which is explained further down in my link above).

 

Thanks for the link. I had a look around over there once before, but being a low (to middle at best) level wizard I couldn't seem to figure it out. I had considered emailing them at some point to ask, maybe I should.

 

If you follow the "coreboot" link on for ex. the apu2 product page it takes you to PC Engines Github IO page where they explain that their firmware is actually maintained by a third party (3mdeb) and they also go on about signing keys, warrant canaries, open source and such (which I found interesting and perhaps, hopeful).

 

But that was about as far as I got, so far...

 

EDIT: OK, you have now motivated me to reach out to them. I will report back what I learn...

Edited February 6, 2020 by TRS-80
clarify, add last bit

[Daniel Denson]

I'm biased, I actually run a PC Engines distributor... (embeddor) 

 

All APU2/3/4 run the same quad 1Ghz AMD Jaguar CPU.  They have zero 'IME' or similar options and run coreboot open source bios.

 

These units can support SATA controllers on the mPCIe which means you can potentially put 13 SATA drives on or more if you get fancy.

 

These units absolutely smash an esspressobin.  They are not in the same league.  I've build a couple of hobby CEPH clusters on 2C4 units.

 

They have good intel nics (seems a strange mix of an AMD SoC and Intel nic but I'm not complaining)

[TRS-80]

44 minutes ago, Daniel Denson said:

I'm biased, I actually run a PC Engines distributor... (embeddor) 

 

All APU2/3/4 run the same quad 1Ghz AMD Jaguar CPU.  They have zero 'IME' or similar options and run coreboot open source bios.

 

Thanks for the feedback.

 

I did end up emailing PC Engines directly yesterday, and I will update this thread if and when I receive a reply.

 

In the meantime, I would like to point out that using coreboot in and of itself is no guarantee that there are no blobs (or anything else). Coreboot itself is open source of course, however it can be used to load whatever payload(s) one wants, including blobs, etc. Now, I am not suggesting that is what PC Engines are doing. Just pointing out that this is exactly why Libreboot project was created in first place (which can be seen as a subset of coreboot, featuring strictly Free Software, the same way that the GNU list of fully free/libre distributions can be seen as a subset of all the GNU/Linux distributions in general).

 

But this is something I have been trying to puzzle out, as their firmware developers seem to be very keen on the ideals of Free Software, etc. (if you read their bios on GitHub for example). And PC Engines certainly don't make any positive mention of it one way or the other (at least anywhere that I could find).

 

My understanding was that this (AMD equivalent of IME, I don't remember offhand what it's called) was mandatory board/firmware level and required to initialize the hardware of certain CPUs after a certain point in time (which would include Jaguar). But maybe I am mistaken in all or some of that.

 

I am certainly keen to find out, and hope that you are right, as you say it is really nice hardware (which have come up on my radar several times now because of that).

[legogris]

Thanks for the input, guys! I'll order another board and run some benchmarks (and sorry for getting x86 in here xD)

 

@Daniel Denson What case do you use for this with disk(s)?