Jump to content

Surveying the hardware landscape 2019 and beyond, with an eye toward freedom (headless server)


Recommended Posts

Posted (edited)

Greetings everyone,

 

A couple years ago, I did my research and followed the recommendation here and on linux-sunxi to purchase a Cubietruck and have had excellent results.

 

I use my small low power GNU/Linux sever for all manner of things including self hosted "cloud" (contact, calendar, file synchronization), XMPP messaging server, media server, etc...  It has been such a success in fact that we are putting more and more of our valuable personal files on there like pictures, etc. and that has me thinking more and more about reliability, backup, etc...

 

I became very interested in ZFS but my understanding is that it requires 64-bit, but the state of 64-bit ARM devices seems... well, not quite ready for prime time just yet?

 

I have spoken to some friends and family about my self-hosted solution, and a few of them are interested now in doing something similar. So now I am thinking along the lines of purchasing a second (and/or third...) Cubietruck and putting together a sort of distributed cluster of little servers at different locations where we each back up one another's data.

 

So I guess my question is, should I pull the trigger now and purchase additional Cubietruck(s), or just sit tight and wait a little longer until 64-bit ARM matures? Or perhaps there is some other option I am not aware of (hardware recommendation)?

 

My primary concerns are privacy, including keeping my own data on devices I physically control or have access to. Also the whole state of affairs with blob bootloaders is very troubling to me, but it has been difficult for me to find specific info on this, at least with regard to specific ARM based hardware. Maybe I am not looking in the right place(s)? Any pointers about where to find such information would also be greatly appreciated.

 

To give you an idea of where I am coming from, I spent years and hundreds of dollars acquiring a number of KGPE-D16 motherboards and related hardware (ECC RAM, etc.) to run Libreboot and ZFS, only to measure the power consumption and realize that at hundreds of watts it was totally unfit for the purpose of a 24/7/365 server. I just don't want to make any more really bad mistakes like that. I know there are some very knowledgeable people here, and I am hoping some of you can contribute to a discussion that would get me (and others who think similarly) looking in the right direction.

 

TRS-80

 

P.S. - I just want to thank everyone here who is doing such a fine job. The developers as well as those who help in answering questions in the forums, etc. I am very short on time these days and cannot help out in that way myself currently, however I did make a small monthly financial commitment in the form of a membership. It is not a lot but is the least I can do. I feel that those of you who spend your valuable time on development, support, etc. should not have to come out of pocket for server costs and other small expenditures. I know that I personally greatly appreciate your work, I'm sure others do as well, even if they don't say so as often as they should. Cheers!

Edited by TRS-80
typo
Posted
8 minutes ago, TRS-80 said:

but the state of 64-bit ARM devices seems... well, not quite ready for prime time just yet?

 

8 minutes ago, TRS-80 said:

until 64-bit ARM matures

There are many 64bits SoC which are already matures, such A64 or H5, just to name a few ...

Posted (edited)

Thanks for the suggestion, @martinayotte, I am reading up on those architectures now.

 

A quick read of https://linux-sunxi.org/A64 seems to indicate only 1x USB port, and no SATA, which does not seem suitable for a multi-disk ZFS server, even though it is 64-bit.

 

https://linux-sunxi.org/H5 however, with 3x USB seem possibly more suitable. I need to read more.

 

I suppose when I said that, I was referring to the RK3399 boards which seem to be much more powerful, more RAM, and more options as far as places to attach multiple disks (some even with PCIe slots!) which would make them pretty much ideal as far as headless ZFS servers are concerned. Unfortunately, Linux support on those still seems very WIP, for the time being...

 

I keep going back and forth on USB vs SATA interface to attach the hard drives, I seem to recall @tkaiser saying somewhere that speed is pretty much the same nowadays (assuming USB3?) and then from a tinfoil hat perspective I have also read that USB better isolates the hard drive firmware from direct memory access?

 

TRS-80

Edited by TRS-80
clarify
Posted
6 hours ago, TRS-80 said:

I was referring to the RK3399 boards which seem to be much more powerful

 

I would at least take a look at the H6 at this point, support is coming along quite nicely.

 

Also, I own a CoCo 2 and a Model 100, so, tip of the hat to the name.  :lol:

Posted (edited)
On 6/3/2019 at 2:33 AM, TonyMac32 said:

Also, I own a CoCo 2 and a Model 100, so, tip of the hat to the name.  :lol:

 

Wow, you still hanging on to those fossils?! XD  A year or two ago, I finally convinced the missus to get rid of several old consoles she had kept for years and I replaced them all (and more) with an emulator on our front end RPi in the living room (which also runs Kodi, etc.).  But then again we are very limited on space in our current setup. I know there are a few things I would be hoarding more of, if I had the room... XD

 

I programmed my first TRS-80 in BASIC in elementary school, which was one of my first experiences with computers, hence the moniker. I drifted away from my natural interest in tech for many years (building up a business, etc.) and so when I started getting back to my roots a few years ago, it seemed to be a good choice for an online handle. :)

Edited by TRS-80
typos
Posted
10 hours ago, TRS-80 said:

Wow, you still hanging on to those fossils?!

 

Hahaha, well, they are small (not like I have a full TRS-80 desktop setup anywhere) and I like the hardware moreso than the software really.  I've played with putting the systems on FPGA's, etc, but it's just not the same thing.  With my ESP32 board I'm hoping to build a sufficiently "Classic" computer design for my son to learn on, of course these days Python is the closest thing to BASIC without really digging for code...

Posted (edited)

I finally got a chance to look at all the H5 boards, and every one on that list is only USB2. Taking a look at the main H5 page at linux-sunxi, it does not seem to outright state anywhere that the SoC only supports USB2, although that does appear to be the case based on all the boards I looked at.

 

I suppose that puts me right back to where I am with the Cubietruck (I/O wise) except with a 64-bit CPU. Which I am not sure is compelling enough to make that leap, just yet...

 

But wait, there's more! :)

 

Another development has happened recently. The missus has become interested in putting up a couple cameras at home. Of course wanting to do everything using only F/LOSS/H* (as much as possible), I did a little research and learned that there is some open source alternative firmware available for inexpensive Xiaomi IP cameras which was pretty exciting and new information (to me at least). And of course I have known about ZoneMinder already for a long time.

 

I have looked into cameras on and off over the years, and I know that ZoneMinder requires some CPU grunt to sort through the camera feeds looking for changes. Of course now that I have become spoiled with the low power usage and low cost of all these great SBCs that are coming available, I wondered if there were any that could handle such a task. Lo and behold, after a little research I came across this blog post about putting ZoneMinder on an ODROID-XU4.

 

After reading up on the ODROID-XU4, and pricing them, I realized that it is a hell of a lot more CPU grunt than the Cubietruck, and also 2x USB3 for slightly less cost. Here is a comparison of the specs relevant to my use case:

 

Board

  1. SoC
  2. CPU(s)
  3. RAM
  4. on board storage
  5. SATA
  6. USB
  7. Ethernet
  8. Price

ODROID-XU4

  1. Exynos 5422 Octa big.LITTLE ARM
  2. Cortex-A15 @ 2.0 GHz quad-core and Cortex-A7 quad-core
  3. 2 GB LPDDR3 RAM at 933
  4. 8 GB EMMC $12.90
  5. NO (available as add on)
  6. 1 × USB 2.0 A Host, 2 x USB 3.0 Host
  7. 10/100/1000
  8. $70.96

Cubietruck (aka Cubieboard3)

  1. Allwinner A20
  2. ARM Cortex-A7 @ 1 GHz dual-core
  3. 2 GiB DDR3 @ 480 MHz
  4. 8 GB NAND flash
  5. 1x
  6. 2x USB A host (2.0?)
  7. 10/100/1000 RTL8211E
  8. $75 (eBay)

 

So at this point I'm pretty excited about the ODROID-XU4 for the following reasons:

  • it should be able to run ZoneMinder with a few cameras (which is enough for me)
  • USB3 instead of USB2
  • a lot more CPU grunt
  • actually cheaper than Cubietruck!

It's still 32-bit, of course, but for my current direction getting into ZoneMinder, I am thinking this will be my next purchase. Where data integrity is not as important on stored (somewhat temporary) videos. Therefore no need for ZFS, and hence no need for 64-bit. And then this will give me something to work on while I wait for the more powerful 64-bit stuff (like Rockchip 3399) to mature. And then put all my "important" stuff on a ZFS system on that.

 

On 6/3/2019 at 2:33 AM, TonyMac32 said:

 

I would at least take a look at the H6 at this point, support is coming along quite nicely.

 

 

Oh yeah, and I still need to look into H6. Cursory look shows USB3 support (but only 1 port?!) and what the linux-sunxi H6 wiki page says about the PCIe implementation being broken does not sound good either. :( Thanks for the suggestion though @TonyMac32, at first glance I actually missed what you were saying, only one character difference from H5. :) I know you are more involved in the development, and may have more current information than what is presented there on that wiki?

 

* Free/Libre and Open Source Software (and Hardware)

Edited by TRS-80
typo
Posted

I've lapsed pretty badly here, only keeping up with a few pull requests while I launch an ESP32 board (or try to anyway), so my H6 knowledge is not all that great.

The Odroid-N2 is an interesting board as well, for what it's worth.

I use an XU4 as my media server/file server. It is capable of transcoding a video stream in software, although that is not incredibly necessary at this point with Linux support of the video hardware.

Sent from my Pixel using Tapatalk

Posted
41 minutes ago, TRS-80 said:

It's still 32-bit, of course, but for my current direction getting into ZoneMinder, I am thinking this will be my next purchase. Where data integrity is not as important on stored (somewhat temporary) videos.



I ran zoneminder for about a year or so.  Its neat, but its architecture is slow and outdated.   I had to build from a feature branch in order to actually store h264 streams rather than mpeg.     Personally I'd steer away from it, and focus on cameras with their own motion detection, video shipping, and a decent api.  (Amcrest falls in that category)   

Posted (edited)
On 7/11/2019 at 12:18 AM, lanefu said:

I ran zoneminder for about a year or so.  Its neat, but its architecture is slow and outdated.   I had to build from a feature branch in order to actually store h264 streams rather than mpeg.     Personally I'd steer away from it, and focus on cameras with their own motion detection, video shipping, and a decent api.  (Amcrest falls in that category)   

 

Thanks for the input, but proprietary software is a non-starter for me. So much so in fact, that I purchase hardware nowadays based on how free/open the driver support (and company/community around it in general) are. In fact, I never even considered purchasing any IP cameras until I came across those ones with the Dafang hacks.

 

I did hear the Zoneminder guy on some Linux podcast a few months back, and it sounded like some more development has been going on of late. Well, I certainly hope, anyway. I have yet to touch it, but I will be knee deep in it here soon enough! :D

 

So, I've got (2x) the fixed (Xiaofang) ones and (1x) PTZ (Dafang) on the way from AliExpress as we speak. The former were only lke $20 something each, and the latter ~$30(!). I can remember what seems like only a few short years ago PTZ cameras being like $500!

 

Anyway, back to topic, I came across this post which nicely summarizes sort of where we are today with regard to SBCs and particularly how open they are which leads directly to how well they are supported by community / Linux which means a good, long term, and stable experience for end users.

 

For now I am going to pull trigger on (1-2) ODROID-XU4, one for dedicated ZoneMinder instance, and the other one for a spare / development / testing platform. I've also ordered a 16 port (8 PoE) gigabit switch, 1000' box of Cat5e (only $38 shipped on eBay, surprisingly!) and some ends, crimping tool, etc... so yeah we will see how it goes!

Edited by TRS-80
add paragraph about ZM development
Posted (edited)

@lanefu,

 

I would add, that a lot of those features (on board video shipping, motion detection) actually seem to be supported in the Dafang hacks software. I think I read somewhere that the dev who wrote it actually worked on their firmware at some point (not sure if true)? Anyway, check out the links for more info. And I'll try and remember to report back how it all goes (although it will likely be some months from now at the rate my work schedule is going).

Edited by TRS-80
accidentally hit post prematurely
Posted
On 8/23/2019 at 10:21 AM, TRS-80 said:

So, I've got (2x) the fixed (Xiaofang) ones and (1x) PTZ (Dafang) on the way from AliExpress as we speak. The former were only lke $20 something each, and the latter ~$30(!). I can remember what seems like only a few short years ago PTZ cameras being like $500!

 

One of the side benefits of China becoming a surveillance society - Government needs lots of cameras to keep an eye on the folks over in Xinjiang....

Posted
On 6/2/2019 at 11:41 AM, TRS-80 said:

I became very interested in ZFS but my understanding is that it requires 64-bit, but the state of 64-bit ARM devices seems... well, not quite ready for prime time just yet?

 

Without getting into the file system wars - might consider BTRFS, which is much more GPL friendly...

 

One of the challenges with the low memory platforms is that ZFS is fairly RAM and compute intense for what it does - that being said, for the right purpose, it's a good file system to look at.

Posted
On 10/14/2019 at 10:10 PM, sfx2000 said:

 

Without getting into the file system wars - might consider BTRFS, which is much more GPL friendly...

 

One of the challenges with the low memory platforms is that ZFS is fairly RAM and compute intense for what it does - that being said, for the right purpose, it's a good file system to look at.

 

Agreed, don't want to get into file system war either. But reasoned discussion is good. :)

 

When I researched into that (maybe a year or more ago) at the time it seemed that ZFS was more stable / production ready. I read some quite disconcerting things about BTRFS at the time (data loss, etc.). Also it seemed as the ZFS on Linux project had tons of big backing behind it and development going on.

 

Just so you know, I am much more of a "Free Software" than an "open source" guy, personally (as I think you can tell by this post, and others I have made). As a matter of fact now I am feeling a little bit guilty for not promoting the Free Software option. But when the whole point of such systems is file system reliability, the sorts of things I was reading about BTRFS were non-starters for me. Maybe things have changed since then, and I should have another look. I actually would prefer that to be the case. I would even be willing to contribute financially to the development of the Free Software option, until it gets to the point of stability (as such work is beyond my skill level).

 

As far as memory, I think most of that is mainly a requirement when doing the more complicated forms of spreading out your arrays over disks (may not be using correct terminology here, it's been a while). Since I plan on doing simple mirrored pools I think a lot of the memory requirements go down. Also, that is actually the recommended way straight from one of the developers as I recall.

 

Again, all of this info current as of 1+ (maybe even 2+(?)) year(s) ago, and I certainly will brush up and get up to date before implementing anything. Well, maybe I should make time for that, as in my mind I am still waiting on 64-bit ARM boards (with appropriate memory and connectivity for multiple disks) to become stable in Armbian. Maybe BTRFS does not require 64-bit? Perhaps I should revisit this...

Posted

The HCx's are XU4's purpose built for server work. Nice hardware. I use an XU4 now, considering the N2 once I verify the patch I saw actually fixes USB3.

Sent from my Pixel using Tapatalk

Posted
4 hours ago, guidol said:
On 8/23/2019 at 5:21 PM, TRS-80 said:

For now I am going to pull trigger on (1-2) ODROID-XU4

 

I love the XU4. I used it for a long time. Good GPU drivers, no VPU drivers but great CPU. It also was a hell to work with. The thing overheats very quickly, it isn't very stable. And many boards have passed it in every way. But not one all at once.
It was by far the top for a long time. But it's getting old.
The most versatile board in my eyes is the NanoPi M4. It can do everything. But it isn't the most powerful.  
If you want good cpu perfomance the Odroid N2 is by far a better bet. But that's got its issues. And little I/O options. Only 1 USB3 controller for all 4 ports, and they perform bad. Things you don't have with the RK3399's.  But biggest advantage for N2 is you'll never need a fan to cool it. 
Also the Khadas VIM3 is an awesome machine. It's got the most powerful CPU there is. The Odroid N2 + 4 x 400Mhz more on A73. Tho more expensive, and only the pro is advisable. I've got the 2GB/16GB version. SD card reader is mighty slow. eMCC too small to do something. Only 1 USB3 port for an ssd.  Works ok, but it's not perfect at all.
For that the N2 has a better option with eMMC modules up to 128GB. 
The NanoPi M4V2 now is my main machine. Don't use any pc(except the atomic pi) anymore. I just need a second desk to put all my boards and more displays. Instead of one power hungry machine, I use many very power efficient ones for my different tasks. But the XU4 has been down for a long time. With the above reasons. (and I've got too many other boards as well)

Posted

I liquid cool my XU4 if I had the HCx it would probably be better off, but nowhere near as cool. ;-)

The N2 has some patches to fix the USB3 to my understanding. I haven't tried them just yet, or perhaps Igor has integrated it already...

The Renegade Elite has an add-on board for nvme and a wifi, and a nice case with giant heatsink that should dissipate far better than the M4 (but is generally larger).

All that said, the XU4 and relatives are still king for all-around performance and software support. I have the big heatsink one set up for retro gaming, if(once) the RK3399 or s922 are finally properly supported (looking at you Rockchip, shame on you for your RAM bullshit, and Amlogic, one word, "blobs").

Sent from my Pixel using Tapatalk

Posted

Thanks everyone for contributing info. It can be hard to know all these peculiarities about each board, unless you own all of them.

 

I wanted to update this topic with some things I learned between then and now.

 

Recently the Helios64 Announcement got me back into thinking about this. I think that board is the closest thing to what I have been looking for, probably most importantly the first ARM board yet (that I am aware of, that is not something designed for server rooms) to be even talking about ECC memory. Which of course would make the perfect board for a ZFS based NAS (if/when the ECC version actually comes to market - but it seems they are actually getting very close - maybe within the next year?).

 

After reading all the relevant materials (full Armbian post, full product announcement, and all comments of both) I only became even more excited. There was one last bit only though that gave me pause. But because it is something I have been wondering about all along, I don't think this question is particular to the Helios64. And that question is:

  • Exactly how much non-free / binary blobs are required to run these boards?
  • (and) Where can one look to find such information?

I tried digging around the Armbian source code, but I couldn't seem to figure it out.

 

I suppose it depends a lot on the board, and whether you are using (most especially) video, Wi-Fi, Bluetooth, etc. as those seem to be the most problematic areas, in terms of binary blobs.

 

In the case of Helios64 (and I quoted this also over in that thread) their software dev gave a really good answer about binary blobs (which I appreciated, since such info can be hard to find, at least for me):

 

Quote

Aditya Prayoga Admin -> David Pottage • 16 days ago

We will use kernel based on mainline kernel. RK3399 is supported on Armbian CURRENT branch that use LK 5.4. We will also submit the device tree to mainline kernel.

Regarding binary blobs, there are only 2 in use, the pre-bootloader (before u-boot stage) which is pretty common in most SoC, and the DisplayPort driver.

 

(emphasis mine)

 

So I have asked gprovost for some clarification on that (for Helios64) but I wanted to get more of a general answer over here. If some of the Devs might humor me. :)

Posted

For RK3399 IIRC it should be possible to build u-boot without a blob, of course I haven't looked at the new Helios offering (DDR4 + Rockchip is a PITA)

Sent from my Pixel using Tapatalk

Posted
24 minutes ago, TonyMac32 said:

(DDR4 + Rockchip is a PITA)

it got slightly better.. E.g. pinebook uses the lpddr4 driver from u-boot let's see if we can/should move some other too..

 

On 1/24/2020 at 6:30 PM, TRS-80 said:

I tried digging around the Armbian source code, but I couldn't seem to figure it out.

most blobs are used to boot may it be SPL or whatever.. It's then up to you to decide that you trust that they only do what they should.. e.g. we had this discussion for amlogic devices where the blobs did much more than just bringing the SoC in a state where u-boot linux can fully control iirc linux had never fully control over CPU clockspeed.. No idea if this changed in between I'm not interested in amlogic at all atm.

Some sort of a bootrom will always be there and mostly if not ever this isn't opensource nor upgradable/replaceable. Same counts for every wifi chip as well.. some pieces of 'firmware' will always be there.. otherwise there you go: https://www.cnx-software.com/2019/12/16/openwifi-is-an-open-source-linux-wifi-stack-running-on-fpga-hardware/

 

IMO arm was never about being as much as possible opensource, if you look for that maybe you find your luck with a risc-V SoC, but even there it's likely that 'full free' isn't the goal of most SoC makers build something based on it. Same counts then for displays etc. They may or may not contain some sort of firmware which is closed source and can't be replaced.

Posted
3 hours ago, TonyMac32 said:

of course I haven't looked at the new Helios offering (DDR4 + Rockchip is a PITA)

 

It gets better. For the ECC (which I am waiting for) apparently has the error correction built in on module, not in controller.

 

Of course I don't know a lot, maybe that's normal? Not sure how works on SoC as compared to traditional PC with where memory controller is located.

 

At any rate,, I hope adding ECC doesn't make for even more PITA.

Posted

I mean, as long as we're on ARM CPUs, we're not going to be technically free whatever we do.. But I'm all for going as far in that direction as possible and reasonable.

 

I actually got the above bootloader running on an RPi3B+ with Debian 10, which prompted my inclination to get this into armbian.

 

Maybe this is off topic, but seeing as you did some research, what's your opinion on a good board that's as free as possible and with high performance in terms of CPU/memory/IO? (I don't care much about graphics, tinkering a cluster for Docker workloads). Active PoE is obviously a huge bonus, but after ROCK dropping it from their v3 I've abandoned the hopes of seeing that anytime soon so it'd have to be splitters in the foreseeable future at least (unless there's some gem I've missed)

Posted (edited)

Note: I split @legogris post immediately above off from another topic to bring it over here where it's more relevant.

 

@legogris,

 

I am actually still researching that, believe it or not. It's an ongoing task, perhaps never ending... :)

 

For the CPU type workloads, ODROID-XU4 certainly meets the bill (alternatives are also discussed at length above), but I am not certain about from a Freedom perspective. Although do note @TonyMac32 comment above:

 

On 1/25/2020 at 5:28 PM, TonyMac32 said:

For RK3399 IIRC it should be possible to build u-boot without a blob

 

If it is all about compute as you say, last I checked ODROID-MC1 were still on a big sale, in fact this may actually be perfect for what you are looking for. They use at least similar (same?) Samsung Exynos5422 big-LITTLE octa-core CPUS as ODROID-XU4 and so maybe Freedom issues are same as above(?).

 

If like me you are concerned about such things (seems you are) it would probably be worth your time to read the thread.

 

And finally, getting around to @chwe comments I missed earlier,

 

On 1/25/2020 at 6:18 PM, chwe said:

IMO arm was never about being as much as possible opensource, if you look for that maybe you find your luck with a risc-V SoC, but even there it's likely that 'full free' isn't the goal of most SoC makers build something based on it.

 

Yes I follow RISC-V with great interest. You may be right about mfrs, but we may end up with some Free stuff anyway, either on accident, on purpose, or even eventually later on as a result of hacker efforts...

 

And I agree with you about ARM. In fact some things I have read lead me to believe the exact opposite of this. Which makes sense as they are essentially an "intellectual property" holding company. But I think @legogris summarizes my view nicely:

 

46 minutes ago, legogris said:

I'm all for going as far in that direction as possible

 

Which reminds me of something else I read recently in the FOSDEM video boxes post, which in turn leads to some info about Luc Verhaegen, specifically:

 

Quote

The lima driver inspired a few like-minded individuals to tackle similarly daunting tasks and now most major ARM GPUs have open source graphics drivers in various stages of completeness.

 

So, to me, it's all about continuing to push in a direction. Remember that some of these things were absolutely unheard of just a few short years ago...

 

Edited by TRS-80
clarify post moving
Posted (edited)

@chwe added some relevant info in another post, I wanted to copy here for reference:

 

40 minutes ago, chwe said:

 

Thanks, @chwe! :)

Edited by TRS-80
clean up moved post / quote
Posted

So my takeaway after reading up a bit more. For CPU/IO loads, Khadas VIM3 is a beast and looks unparalleled today, but is quite expensive and includes an NPU which you may or may not need. VIM3L is a bit lower specced but can easily be extended to get PoE, which is nice. ODROID N2 comes close, but is physically large. NanoPi M4V2 looks to be the most reasonable option on the market today - if only it came in a 4GB RAM version...

 

I feel that each of these boards is so close but each is missing one single thing that ends up making it a significant compromise.

 

XU4 is quite low-specced compared to each of these, and similarly priced to the NanoPi M4.

 

FWIW, linuxgizmos just compiled a decently comprehensive spreadsheet that is pretty up to date: http://linuxgizmos.com/ringing-in-the-new-year-with-136-open-spec-linux-sbcs-under-200/

 

Also, @NicoD has a lot of good stuff on his youtube channel: https://www.youtube.com/watch?v=2Yu8Rj7o9lk

 

For now I think I will hold off any new purchases and fall back to NanoPi M4V2 in absence of new models or price drops in the coming couple of months.

 

 

Guest
This topic is now closed to further replies.
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines