Review Quick review of NanoPi Fire3

[tkaiser]

This little and inexpensive ($35) board is fully compatible to discontinued NanoPi M3. From a software point of view both boards are identical (though Wi-Fi is missing on the Fire3) and as such identical OS images can be used for both boards. The good news: compared to the last time I looked at the M3 kernel support has improved a lot. Back then we had to run a smelly 3.4.39 (32-bit only) while we can now run mainline on it.

 

I gave it a try using our Armbian Stretch nightly running with 4.14.40 (full armbianmonitor -u output) and did a couple of tests. The Samsung/Nexell S5P6818 SoC consists of 8 A53 cores running at up to 1.4GHz with default settings (can be slightly overclocked up to 1.6 GHz according to Willy Tarreau -- see the reviews at the product page). All cores behave like one big cluster (so adjusting /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq affects all 8 CPU cores at once, this is no artificial pseudo big.LITTLE as Amlogic does it with their S912).

 

Now with a recent 64-bit kernel with the CPU cores brought up in ARMv8 mode we can also make use of ARMv8 Crypto Extensions making the S5P6818 to one of the most powerful boards when it's about AES crypto and stuff can run on all 8 cores in parallel (33 times faster than a RPi 3 and still 28 times faster than a RPi 3+). While the octa-core config sounds interesting for CPU intensive workloads one should keep in mind that this board has only 1 GB DRAM which is simply not enough for many such workloads (or you would need to make massive use of zram instead which performs better than swap on slow storage but of course bottlenecks a lot).

 

Talking about a somewhat powerful CPU we also have to talk about temperatures and consumption. The board is always part of a kit so FriendlyELEC sells it together with a heatsink and a high quality Micro USB cable that solves a lot of the Micro USB related powering problems. In idle I measured 2.6W with Armbian (PSU included at the wall) while Willy Tarreau reported 'It consumes 400mA/5V in idle, and 1.2A/5V under openssl RSA with 8 cores at 1.6 GHz' (most probably using FriendlyELEC's OS image). So I tested for consumption with worst case workloads and used cpuburn-a53 for this. Since I knew from last time when I tested that the board deadlocks when starting cpuburn-a53 at 1.4 GHz I increased max cpufreq in steps (consumption always with PSU included):

1000 MHz: 9.3W
1100 MHz: 10.7W
1200 MHz: 12.2W
1300 MHz: 13.8W
1400 MHz: 14.7W

After a short time with cpuburn-a53 running at 1.4 GHz the board deadlocked again which is not a surprise since my PSU is rated for 2A (10W) and Micro USB itself is only rated for 1.8A (9W). As usual with FriendlyELEC boards there's a 4 pin header for serial debug console that can also be used to power the board more reliably so even demanding tasks that increase consumption to 15W are possible when powering through the header.

 

Talking about temperatures: After applying the heatsink I did a simple compile test (Arm Compute Library) with -j8 and after a minute the board did an emergency shutdown since CPU temperature reached 100°C. So all following tests were done with a huge fan blowing air over the heatsink laterally. IMO for demanding tasks improved airflow / heat dissipation is a must and the small heatsink simply not sufficient.

 

What other hardware is there? The usual 40 pin GPIO header, Gigabit Ethernet (with Armbian's settings and mainline kernel we're talking about iperf3 numbers of ~925 Mbits/sec in both directions), a mini HDMI port (most probably supported by Armbian), a camera and a LCD port (both not supported by Armbian as far as I know), the Micro USB OTG port and a single USB2 type A port. USB Attached SCSI (UAS) is yet not available with mainline kernel so storage performance is a little lower as it could be. This is a Samsung EVO840 in an ASM1153 enclosure connected to the USB2 port:

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4     7544     9889    10098    10117     7996     9770
          102400      16    16143    20165    20365    20260    19858    20172
          102400     512    33138    33659    33120    33373    33417    33321
          102400    1024    33511    33663    33429    34020    34119    33521
          102400   16384    32731    34012    36483    36750    36674    34291

I also did a quick test as NAS and got numbers as expected: everything a little bit slower compared to those USB2 platforms that can make use of UAS -- so if NAS is the use case some of the cheaper Allwinner boards with Gigabit Ethernet are a better idea.

 

As usual FriendlyELEC did a great job documenting the board: http://wiki.friendlyarm.com/wiki/index.php/NanoPi_Fire3 -- they also provide OS images that make full use of all hardware features (camera, LCD displays auto-detected by u-boot, GPU/VPU acceleration, HDMI resolution switching in Linux sounds a bit like PITA though).

 

Still not sure for which use cases this board applies. The octa-core CPU would better be accompanied by more DRAM (though then you need to get the NanoPC-T3 Plus -- same SoC but 2 GB DRAM) and I fear making use of the processor power almost always requires a fan blowing in addition to the heatsink (without a fan I measured in idle always +60°C after a few minutes)

[tkaiser]

Just spotted another important difference between NanoPi M3 and Fire3: 'AXP288 PMIC is gone, and replaced by an STM32 Cortex M0 MCU'. So DVFS implementation is different which also explains why with Armbian there's no difference in idle consumption/temperature when clocking with 400 vs. 1400 MHz.

[Igor]

3 hours ago, tkaiser said:

USB Attached SCSI (UAS) is yet not available with mainline kernel

I didn't notice. Perhaps enabling it would help? https://github.com/armbian/build/commit/c1cb1d77f0812978160506de9646c352df0c9d48

[wtarreau]

Thomas, I think it's problematic that you only have a watt-meter including the PSU because the PSU's efficiency depends on the consumption (usually it's optimal around 50% load). Using a USB power meter would tell you the volts, amps and watts, and would even allow to detect under-power when it happens.

 

I managed to get my board to shut down only once, it was powered by my laptop's USB3 port (that's what I do all the time but I'm probably close to the limit). It never happened on a 5V/2A PSU however. Since it was not yet very hot, I suspect that it's the power controller instead which had shut it down rather than the temperature.

 

I also had to play with the critical points to avoid needlessly throttling. I seem to remember having set them to 105, 110 and 112 degrees though I may be wrong since I ran many tests. Now I packed the board inside a cardboard made "enclosure" from which the heat hardly dissipates, and it can still throttle when reaching the first critical point, but that doesn't last long.

When it happens, usually it's at 832/1024 of 1600 MHz = 1300 MHz, and more rarely it's 640/1024*1600 = 1000 MHz. I haven't run cpuburn yet though, I can if you're interested.

 

Regarding the use cases, I think they are limited but the board is awesome when they can be met. For me, it's fantastic as a developer to test threaded code scalability on up to 8 cores, given that the L2 cache is shared between all cores. Usually you need a huge power hungry CPU to get the same, here I have this in may laptop's bag. I also want to see what performance level I can reach on HTTP compression using libslz. I'm pretty sure that making some content recompression farms using such boards could be very affordable. Also the CPU supports native CRC32 instructions which are missing on x86 and affect gzip's performance, so I'll have to improve my lib to benefit form this. Miners may like to exploit some algorithms which perform well on ARMv8 and exploit the native AES and SHA2 implementations (I'm a bit clueless in this area). Last, for computer assisted vision, you have 8 cores with NEON ;-)

 

[tkaiser]

On 5/19/2018 at 10:32 AM, wtarreau said:

Thomas, I think it's problematic that you only have a watt-meter including the PSU because the PSU's efficiency depends on the consumption (usually it's optimal around 50% load)

 

I know but the only purpose of these quick measurement was to show how important a stable PSU and especially the Micro USB cable is when a board encourages its users to power through Micro USB (as it's the case with all FriendlyELEC boards even if all of them are also able to be powered via a 4 pin header).

 

And this was the interesting observation: With such an artificial high load as cpuburn-a53 consumption at the wall exceeded 14W which simply means that FriendlyELEC's provided Micro USB cable solves a lot of the Micro USB related powering problems due to very low resistance. At least the cable problem avoding the usual horrible voltage-drop with 'average USB cables'. The other problem still remains: users choosing insufficient USB chargers or PSUs that result in a voltage drop under load on their own.

 

But it's even worse: I could watch consumption increase with increasing SoC temperature, IIRC at 1300 MHz consumption difference was ~1W depending on SoC temperature at 50°C or 90°C (with huge fan then board idled at ~35°C so the increase in temperature was just waiting 30 seconds -- cpuburn-a53 is really a beast)

[wtarreau]

I agree with FriendlyElec's USB cables, I've always found they were of pretty good quality. Same for the cables coming with the MiQi by the way. I've had good experiences with a few PSU providing 2.5A under 5.2V and featuring a micro-usb cable. Since there it's not possible to connect any other way, they have to provide a good enough cable and connector. But in my opinion these good ones are almost an exception to the general trend.

 

Regarding the increase of consumption at 90 vs 50°C, I noticed this as well with the RK3288 boards from by first farm. It's in fact due to the lower efficiency of the onboard DC-DC converter. That's another reason for focusing first on optimal heat spreading!

[shaun27]

Using usb measurement device this board consumes:

 

Stock setting no devices connected (headless)

Idle 5.13v 0.32 amp

Underload 5.17v  0.85amp - 1.05amp

 

What i would like to know if anyone else noticed when you do sudo halt to shut down, the power goes through roof 5.17v 0.80amps and it stays like that.

 

New to armbian btw but not linux in general.

Edited May 21, 2018 by shaun27

[wtarreau]

@shaun27, you're pretty close to my measurements. I hadn't noticed your point about the shutdown, but I seem to remember noticing it didn't cut off. Maybe there's no way to completely shut the DC-DC and the CPUs are not stopped but looping in place.  At the very least if that's the case, I think we could improve the situation by using a WFI instruction...

 

[shaun27]

It's a tad worrying tbh thinking you have shut it down yet the CPU is getting warm. Just tested sudo halt switched fans off and CPU was getting quite warm before I had to pull the plug manually. 

[datsuns]

Can anybody that has this board tell me how long it took to receive after it shipped (ordered directly from FriendlyElect)? I'm dying to test this thing out but haven't received it yet.

 

Also , if anybody is currently running this board for a long-term project and has a cooling setup that they are happy with I would love to hear about it. From the OP it sounds like the incuded heatsink alone isn't really sufficient.

[wtarreau]

5 hours ago, datsuns said:

Can anybody that has this board tell me how long it took to receive after it shipped (ordered directly from FriendlyElect)?

 

I'd say around 2 weeks.

 

5 hours ago, datsuns said:

Also , if anybody is currently running this board for a long-term project and has a cooling setup that they are happy with I would love to hear about it. From the OP it sounds like the incuded heatsink alone isn't really sufficient.

The default heatsink is enough if you're not running at 100% CPU full-time. For my use cases, it's mostly a network endpoint and I can run it at 1 Gbps without problems even with the board confined in a cardboard made enclosure. But if you run with all CPUs saturated, you'll reach around 5W that need to be dissipated one way or another. The default heatsink and the PCB are not large enough to dissipate 5W at a low temperature. I significantly raised the temperature thresholds (113, 115, 120) to prevent it from throttling too early. Note that these thresholds are higher than the datasheet's (85°C in commercial ranges). But the thermal sensor supports up to 125°C so probably there are some industrial/military grade variants with higher ranges. For a personal project I'd say you have some headroom. For a commercial product, you probably don't want to play with this and you may have to use a small fan, or to place a thermal pad behind the board against a metal enclosure.

[datsuns]

15 hours ago, wtarreau said:

 

I'd say around 2 weeks.

 

The default heatsink is enough if you're not running at 100% CPU full-time. For my use cases, it's mostly a network endpoint and I can run it at 1 Gbps without problems even with the board confined in a cardboard made enclosure. But if you run with all CPUs saturated, you'll reach around 5W that need to be dissipated one way or another. The default heatsink and the PCB are not large enough to dissipate 5W at a low temperature. I significantly raised the temperature thresholds (113, 115, 120) to prevent it from throttling too early. Note that these thresholds are higher than the datasheet's (85°C in commercial ranges). But the thermal sensor supports up to 125°C so probably there are some industrial/military grade variants with higher ranges. For a personal project I'd say you have some headroom. For a commercial product, you probably don't want to play with this and you may have to use a small fan, or to place a thermal pad behind the board against a metal enclosure.

 

Ok, this is some good info and I will do some testing when it arrives. What do the three values here represent (113, 115, 120)? I assume these are values you picked and are celsius temperatures - is 113°C the value you chose before it starts throttling performance? And is 85°C the default cutoff for throttling?

 

Based on your time estimate my board should be here any day now so I am getting pumped to try it out. I'll do my own temp testing under 100% CPU load with and without a fan.

 

 

[datsuns]

And one more thing. The unit looks a lot smaller than the standard raspberry pi form-factor. Do the those mounting holes correspond directly to those of a raspberry pi?

[wtarreau]

 

34 minutes ago, datsuns said:

What do the three values here represent (113, 115, 120)?

 

The 3 critical points (in degrees celcius) for the thermal throttling and shutdown. I didn't understand the difference between the first two ones, as the CPU starts to throttle when the first value is reached. The second *seems* to do nothing, the 3rd one is for the forced shutdown.

 

34 minutes ago, datsuns said:

And is 85°C the default cutoff for throttling?

 

I seem to remember that it starts throttling at 80. IIRC the original values were something like 80, 85 and 105.

 

34 minutes ago, datsuns said:

I'll do my own temp testing under 100% CPU load with and without a fan.

 

It definitely is very informative to do so. Do not forget that such boards will heat much more in summer than in winter (you can more or less shift the high temperature by the difference of ambiant termperature). The most important is that your board remains 100% reliable even when it starts to throttle (the temperature can continue to rise a little bit at this point). A CPU's sensitivity to temperature may evolve over time, so keep a bit of margin. Also if you intend to use the GPU, it's not throttled and will definitely add to the thermal dissipation, this will require an extra margin.

 

34 minutes ago, datsuns said:

The unit looks a lot smaller than the standard raspberry pi form-factor. Do the those mounting holes correspond directly to those of a raspberry pi?

 

No the form factor is much smaller, and really well thought, but absolutely not compatible with RPi. Unfortunately there is no enclosure for these boards, it's really the only missing thing. You can stack many of them side by side vertically with just a rear cable for the power supply and a front cable for the network. The overall design is really nice for those who want high power densities.

 

[chwe]

Cause the SD-Card isn't that far away from the CPU... You should have in mind that 'consumer grade' SD-Cards don't like heat that much.  They are mostly rated 'up to 85°C'... And according to this one (https://www.sdcard.org/press/thoughtleadership/150715_why_sd_memory_cards.html):

 

they might not live that long at high temperatures.. I tried once to get some additional information from 'sd-card' makers (did they any tests? how well will their cards perform at higher temps? etc.) but the only answer I got was: We don't provide you any further information for consumer grade cards asking again what tests were done with industrial grade ones ended in no answer at all (not only one company, more or less every 'well known' sd-card maker refused to answer by their official support question form on their websites..  I have in mind that @tkaiser fried once a card by such a 'high temperature experiment' overnight?

 

Edit: this might be less important if you use things like FEL and the whole thing lives inside ram, but for sure if your board will run at higher temperatures for longer time (probably buildfarm? I don't know if you solve it with SD-Cards or not).. 

 

Edited May 22, 2018 by chwe

[tkaiser]

10 hours ago, chwe said:

And according to this one (https://www.sdcard.org/press/thoughtleadership/150715_why_sd_memory_cards.html):

 

they might not live that long at high temperatures

 

Sorry, but data retention times are something entirely different: how long remains data readable on an inactive SD card (storing data offline). This is not the use case we're talking here about. But of course you're right and small boards with heat dissipation through the ground plane have an issue with SD card temperatures.

[chwe]

9 hours ago, tkaiser said:

Sorry, but data retention times are something entirely different: how long remains data readable on an inactive SD card (storing data offline).

nice to learn.  I would still be interested if somebody ever saw tests, testing procedures (cause more or less every SD-Card is rated up to 85°C). I had some initial ideas testing this.. But obviously you need many cards from the same brand, from different production batches, from trusted sources but not from them self (to ensure you get what the average buyer of original cards get) to get somehow scientific valid data..  Which then I thought, it's not worth the efforts just out of curiosity.. 

 

9 hours ago, tkaiser said:

But of course you're right and small boards with heat dissipation through the ground plane have an issue with SD card temperatures.

Cause you maintain probably 'one or two' SBCs more than me..  did you notice issues related to this? So that reliable (and I think you don't use crappy SD-Cards anywhere) SD-Cards die faster on small boards if they run at higher temperatures for a longer time? 

 

Cause this is not really related to the nanopi fire 3 review, I happily split this into it's own topic to keep this one clean in case you prefer it. 

[tkaiser]

37 minutes ago, chwe said:

did you notice issues related to this? So that reliable (and I think you don't use crappy SD-Cards anywhere) SD-Cards die faster on small boards if they run at higher temperatures for a longer time?

 

No idea. The only 'good' SD card that died here so far is a SanDisk Extreme Pro 8 GB I used over 3 years intensively (since both great random IO performance and also sequential performance -- burning cards with up to 80 MB/s in my MacBook was always fun). All the other cards that died were cheap Noname/Intenso/Kingston crap.

[g40]

Greetings.

 

The FriendlyArm docs imply that the parallel camera connection is the usual 24way FPC. Can anyone confirm this is the case? 

 

Is the imaging side of things supported in mainline (or near to it)?

 

TVM

 

 

[shaun27]

Just wondering how people here managed to overclock this board because when i try to edit /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq within the os either under user account or root is says permission denied.

[wtarreau]

14 minutes ago, shaun27 said:

Just wondering how people here managed to overclock this board because when i try to edit /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq within the os either under user account or root is says permission denied.

I edited the 05.dtb file in /boot!

[datsuns]

I just got my nanopi fire3 in the mail today. Looking at the heatsink and the board layout and the physical height of the soc, how are people mounting this?  Is anybody happier with an alternative to the thermal paste? Maybe something like a copper or aluminum spacer instead? Or maybe the thermal spacer works well?

[wtarreau]

36 minutes ago, datsuns said:

I just got my nanopi fire3 in the mail today. Looking at the heatsink and the board layout and the physical height of the soc, how are people mounting this?  Is anybody happier with an alternative to the thermal paste? Maybe something like a copper or aluminum spacer instead? Or maybe the thermal spacer works well?

I must confess I absolutely don't remember what I used given that I always have everything I need for this. It's possible that I naturally placed a thermal pad in between. I don't remember having opened a thermal paste tube. Or maybe there was a pad with the heatsink. But definitely I didn't put raw aluminum on top of the CPU die without anything to make good contact in between.

 

[datsuns]

28 minutes ago, wtarreau said:

I must confess I absolutely don't remember what I used given that I always have everything I need for this. It's possible that I naturally placed a thermal pad in between. I don't remember having opened a thermal paste tube. Or maybe there was a pad with the heatsink. But definitely I didn't put raw aluminum on top of the CPU die without anything to make good contact in between. 

 

The included thermal product is more of a pad than a paste, I guess. I set it up with that and I am running some tests. It gets super hot super fast under 100% cpu load and shuts down at 100 degrees Celsius. I will probably do active cooling - not impressed with the heatsink even though its kind of large for such a small board.

 

And you mentioned before about the three throttling temp threshholds - where do I edit those? It seems to shut off before it can throttle down right now.

Edited May 30, 2018 by datsuns

[datsuns]

Well I added a tiny, cheap fan running off the 3.3v pin in conjunction with the included heatsink and thermal pad. At 100% CPU load I am getting a very stable 58-59 degrees C according to armbianmonitor. This is with an ambient room temp of about 23 degrees C.

 

It's like night and day compared to the heatsink alone. I guess these boards really like to have a fan. So if anybody out there wants to run their board hard - get a fan asap!

[wtarreau]

That's quite different from what I'm seeing given that I had made my own enclosure out of cardboard with no air around it! Of course it heats but not that fast, despite the fact that I overclocked it. If the fan helps, it's probable that you're not having a good contact between the heatsink and the CPU. Verify that the heatsink is very parallel to the board, it's possible that it touches only by one angle.

[datsuns]

1 hour ago, wtarreau said:

That's quite different from what I'm seeing given that I had made my own enclosure out of cardboard with no air around it! Of course it heats but not that fast, despite the fact that I overclocked it. If the fan helps, it's probable that you're not having a good contact between the heatsink and the CPU. Verify that the heatsink is very parallel to the board, it's possible that it touches only by one angle.

 

Ok, that's what I was kind of worried about with my initial thoughts. the problem is that the cpu is so short that you can't make direct contact without the fat thermal pad. With the fan it's been really good though and I don't mind the 3.3v draw to keep it cool. And it has been absolutely rock solid for hours without rising above 59 degrees so I am really comfortable with the setup.

 

Also note I am running all 8 cores at 100% continuously for hours. Have you replicated that type of scenario with just the heatsink?

 

I really like this unit - it's kicking ass for my application and I will probably get several more for other things.

[wtarreau]

7 hours ago, datsuns said:

Also note I am running all 8 cores at 100% continuously for hours. Have you replicated that type of scenario with just the heatsink?

I tested a miner on it ("cpuminer" I think) to give numbers to a friend interested on the subject (he was impressed by the way). I didn't let it run for hours like this, but after several minutes it started to throttle down to 1 GHz then stabilized, but didn't stop (and keep in mind it's tightly enclosed in cardboard). It's certain that the modified DTB I'm using helps here with the higher temperature thresholds, but I'm suspecting you might have too weak a power supply or micro USB cable if it stopped. That's always the risk with DVFS : it consumes very little in idle but a lot under load. I discovered one bad cable in my stock using which the board would reboot in loops. @tkaiser could tell you hundreds of horror stories about micro-usb based power inputs :-)

 

7 hours ago, datsuns said:

I really like this unit - it's kicking ass for my application and I will probably get several more for other things.

I find it really awesome and have been asking for it since I got my nanopi-fire2 about two years ago! I'm mostly interested in CPU and network, and this is the only board which comes with a CPU, some RAM, a gigabit connector and nothing else! I'm sure there's plenty of unexploited power in it and am willing to try to push it further!

 

I'm attaching my modified DTB, it adds the 1.6 GHz frequency point and the 113,115,120 degrees critical points which work fine for me and considerably limit the throttling. Save yours before replacing it (variant "rev05"). I have no idea if my values will work on your board or will even kill it, use at your own risks! And please double-check the thermal contact between your heatsink and your CPU.

s5p6818-nanopi3-rev05.1g6-1v25-113deg.dtb

[shaun27]

@datsuns double check that the thermal pad they provided didn't move when you were installing the heatsink.

 

I know when I first got it I put the thermal pad on then placed the heatsink without putting the screws in the heatsink first. This caused the thermal pad to move off slightly. Also thought I killed it by pushing down to hard and all .

 

Temp wise low usage you might get away with no active cooling but this thing does pack a bunch for its size. And I will add temp wise it does handle better then raspberry pi 3 due to the massive heatsink.

 

I've tested this for 4 days straight on xmrig and it sits @ 58c stock speed all 8 cores. 29.5hs btw for cryptonight.

 

Just make sure when you shutdown under armbian you pull the power plug because it doesn't shutdown right and CPU heats up. Under friendlyarm it does the same btw only power off button works. So sudo halt or logout doesn't really shutdown right!.

 

 

[datsuns]

10 minutes ago, shaun27 said:

@datsuns double check that the thermal pad they provided didn't move when you were installing the heatsink.

 

I know when I first got it I put the thermal pad on then placed the heatsink without putting the screws in the heatsink first. This caused the thermal pad to move off slightly. Also thought I killed it by pushing down to hard and all .

 

Temp wise low usage you might get away with no active cooling but this thing does pack a bunch for its size. And I will add temp wise it does handle better then raspberry pi 3 due to the massive heatsink.

 

I've tested this for 4 days straight on xmrig and it sits @ 58c stock speed all 8 cores. 29.5hs btw for cryptonight.

 

Just make sure when you shutdown under armbian you pull the power plug because it doesn't shutdown right and CPU heats up. Under friendlyarm it does the same btw only power off button works. So sudo halt or logout doesn't really shutdown right!. 

 

 

 When you say it heats up, what exactly do you mean? Like the cpu works harder when you attempt a shutdown or just continues to do what it was doing?

 

Also, does this come into play for reboots at all or do they function normally?  I pretty much run my boards 24/7 so it's not really a big deal but I do occasionally reboot them so hopefully that does what it should.