Nanopi fire 3 unstable gigabit ethernet

bhat · April 5, 2020

Armbianmonitor:

Hi!

For some reason my nanopi fire 3's gigabit ethernet is very unstable. When connected to my local machine directly, the ping latency is insanely high and irregular:

PING 10.42.0.101 (10.42.0.101) 56(84) bytes of data.
64 bytes from 10.42.0.101: icmp_seq=1 ttl=64 time=112 ms
64 bytes from 10.42.0.101: icmp_seq=2 ttl=64 time=333 ms
64 bytes from 10.42.0.101: icmp_seq=3 ttl=64 time=350 ms
64 bytes from 10.42.0.101: icmp_seq=4 ttl=64 time=203 ms
64 bytes from 10.42.0.101: icmp_seq=5 ttl=64 time=1014 ms
64 bytes from 10.42.0.101: icmp_seq=6 ttl=64 time=423 ms
64 bytes from 10.42.0.101: icmp_seq=7 ttl=64 time=93.4 ms
64 bytes from 10.42.0.101: icmp_seq=8 ttl=64 time=345 ms
64 bytes from 10.42.0.101: icmp_seq=9 ttl=64 time=358 ms
64 bytes from 10.42.0.101: icmp_seq=10 ttl=64 time=49.6 ms
64 bytes from 10.42.0.101: icmp_seq=11 ttl=64 time=1021 ms
64 bytes from 10.42.0.101: icmp_seq=12 ttl=64 time=299 ms
64 bytes from 10.42.0.101: icmp_seq=13 ttl=64 time=305 ms
64 bytes from 10.42.0.101: icmp_seq=14 ttl=64 time=318 ms
64 bytes from 10.42.0.101: icmp_seq=15 ttl=64 time=300 ms
64 bytes from 10.42.0.101: icmp_seq=16 ttl=64 time=1008 ms
64 bytes from 10.42.0.101: icmp_seq=17 ttl=64 time=290 ms
64 bytes from 10.42.0.101: icmp_seq=18 ttl=64 time=306 ms
64 bytes from 10.42.0.101: icmp_seq=19 ttl=64 time=211 ms
64 bytes from 10.42.0.101: icmp_seq=20 ttl=64 time=1023 ms
^C
--- 10.42.0.101 ping statistics ---
21 packets transmitted, 20 received, 4.7619% packet loss, time 20085ms
rtt min/avg/max/mdev = 49.608/418.113/1022.521/312.908 ms, pipe 2

The nanopi fire 3 is directly connected to my desktop computer via a CAT6 cable, and the desktop computer's network adapter supports gigabit ethernet. Note that it is a direct cable connection and I am getting <1ms with my other boards such as the odroid xu4.

I have also tried connecting the nanopi fire 3 directly to a netgear switch behind a router. Still terrible ping.

Surprisingly if I force nanopi fire 3 to use 100Mbps ethernet connection, the ping latency is suddenly normal.

I've tested the ethernet cable and it is working fine. And the nanopi fire 3 has a stable PSU(5.1V with no load and 5.05V under load).

To investigate further, I tested Friendlyarm's friendlycore (seems to be ubuntu 18.04, kernel 4.4, aarch64) on the fire 3. And the gigabit ethernet is working with full speed. I also tested all history versions of armbian on the nanopi fire 3, and NONE OF THEM HAVE WORKING STABLE GIGABIT ETHERNET. This makes me think that there is something wrong in the gigabit ethernet code with armbian's s5p6818 4.14 kernel, since friendlyarm's 4.4 kernel does not have that issue.

Also something worth noticing: nanopi fire 3 running armbian has less power draw(0.5 W) than friendlyarm's kernel, so it is possible that its network adapter attempted but cannot draw enough power to enable gigabit ethernet, thus the random and irregular ping.

Tl;dr: There are some bugs in Armbian's 4.14 kernel for Nanopi Fire 3 that prevents gigabit ethernet from working properly.

Igor · April 6, 2020

6 hours ago, bhat said:

There are some bugs in Armbian's 4.14 kernel for Nanopi Fire 3 that prevents gigabit ethernet from working properly.

Yes, its a known bug.

6 hours ago, bhat said:

This makes me think that there is something wrong in the gigabit ethernet code with armbian's s5p6818 4.14 kernel, since friendlyarm's 4.4 kernel does not have that issue.

Kernel and boot loader (where is most likely the source of this problem) has little in common with stock support. This work is a port from a person that no longer maintains it. (also there is no maintenance for stock kernel). We tried to port all the way to current kernel, but its insane lots of work / changes. You can still use their stock creations, where this will work, but you will have other problems ...

6 hours ago, bhat said:

NONE OF THEM HAVE WORKING STABLE GIGABIT ETHERNET

I have to remind you that this is amateur driven open source project. All sources are available which means anyone can fix or hire someone to fix the (any) problem: https://github.com/armbian/build#support My estimation is that it can easily costs few days - if you know where to look for. Few days of R&D costs more than all end users donations per year.

bhat · April 9, 2020

On 4/5/2020 at 11:37 PM, Igor said:

Yes, its a known bug.

Kernel and boot loader (where is most likely the source of this problem) has little in common with stock support. This work is a port from a person that no longer maintains it. (also there is no maintenance for stock kernel). We tried to port all the way to current kernel, but its insane lots of work / changes. You can still use their stock creations, where this will work, but you will have other problems ...

Yeah just saying that it might make sense to add information about the ethernet issue to the download page. I almost thought I got an sbc with a broken ethernet chip...

Igor · April 9, 2020

4 hours ago, bhat said:

Yeah just saying that it might make sense to add information about the ethernet issue to the download page. I almost thought I got an sbc with a broken ethernet chip...

Added.

dolphs · June 13, 2020

On 4/6/2020 at 1:45 AM, bhat said:

NONE OF THEM HAVE WORKING STABLE GIGABIT ETHERNET

is that issue current with kernel 4.4.180 and RTL8211E ( though for 5.6 ) ?

Perhaps " ethtool -K eth0 tso off " disabling TCP Segmentation Offloading will solve this,

however not sure how that impacts performance also if you consider this board will be primarily used as a wireguard vpn/ pihole server.

Can one try to see if that helps/ is acceptable for speeds over 100Mbit?

cmoski · July 12, 2020

Running into this exact issue when bringing up new nodes to add to my existing cluster.

Any idea what the last version of Armbian compatible might be? My existing cluster is running kernel 4.14.70-s5p6818 and on Dietpi v6.15 and I could clone that -- but I was hoping to use Armbian for the new additions..

tso is fixed in the latest version and by default appears to be off..

root@armbian-nanopi3-ClusterTest:~# ethtool -k eth0 | grep tcp
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [fixed]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]

Thanks!

zchrish · April 1, 2021

Any update? It looks like development of the software stack from Friendly stopped in 2017. I assume I am on my own to fix it? That's ok, I like the board and am planning to design my own like it so I need to dive into it. Thank you.

Best,

Chris

Igor · April 1, 2021

7 hours ago, zchrish said:

Any update? It looks like development of the software stack from Friendly stopped in 2017. I assume I am on my own to fix it? That's ok, I like the board and am planning to design my own like it so I need to dive into it. Thank you.

Best,

Chris

Yes. We pushed a little further then FriendlyARM with kernel and u-boot but sadly we stuck with both at some degree. For some time now and there are no plans/resources to do anything more about. Also a little less features are supported in our more recent SW stack, some porting is left to be done. Can't say which path is better to proceed - original from FA or ours. Both are not moving anywhere. Our tendency is to push support to mainline, but alone and without serious support for R&D, this can't be done.