nanopi neo4 - serious packet loss after upgrade to Buster

[ottawan]

Armbianmonitor:

http://ix.io/2dkO

But only when the connection is initiated by the remote host.

 

iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from nanopcT4, port 51356
[  5] local nanopiNeo4 port 5201 connected to nanopcT4 port 51358
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.34 MBytes  19.7 Mbits/sec  108   1.41 KBytes
[  5]   1.00-2.00   sec  2.05 MBytes  17.2 Mbits/sec  100   2.83 KBytes
[  5]   2.00-3.00   sec  4.68 MBytes  39.3 Mbits/sec  220   5.66 KBytes
[  5]   3.00-4.00   sec   199 KBytes  1.63 Mbits/sec   24   1.41 KBytes
[  5]   4.00-5.00   sec  3.25 MBytes  27.3 Mbits/sec  179   15.6 KBytes
[  5]   5.00-6.00   sec  4.53 MBytes  38.0 Mbits/sec  217   1.41 KBytes
[  5]   6.00-7.00   sec  1.10 MBytes  9.19 Mbits/sec   56   4.24 KBytes
[  5]   7.00-8.00   sec   860 KBytes  7.04 Mbits/sec   38   2.83 KBytes
[  5]   8.00-9.00   sec  5.75 MBytes  48.3 Mbits/sec  291   1.41 KBytes
[  5]   9.00-10.00  sec  4.00 MBytes  33.6 Mbits/sec  209   1.41 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  28.7 MBytes  24.1 Mbits/sec  1442             sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

iperf3 -c nanopcT4 -R
Connecting to host nanopcT4, port 5201
Reverse mode, remote host nanopcT4 is sending
[  5] local nanopiNeo4 port 55042 connected to nanopcT4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   108 MBytes   905 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   942 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   942 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   942 Mbits/sec
[  5]   5.00-6.00   sec   112 MBytes   942 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   942 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   942 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   942 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   940 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   938 Mbits/sec                  receiver

 

There was also some less significant packet loss and slower performance running Bionic. and asymmetric as well, but in the opposite direction - when neo4 is initiating connection:

 

iperf3 -c nanopcT4 -R
Connecting to host nanopcT4, port 5201
Reverse mode, remote host nanopcT4 is sending
[  4] local nanopiNeo4 port 60366 connected to nanopcT4 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  73.5 MBytes   617 Mbits/sec
[  4]   1.00-2.00   sec  78.2 MBytes   656 Mbits/sec
[  4]   2.00-3.00   sec  78.2 MBytes   656 Mbits/sec
[  4]   3.00-4.00   sec  78.4 MBytes   658 Mbits/sec
[  4]   4.00-5.00   sec  77.7 MBytes   652 Mbits/sec
[  4]   5.00-6.00   sec  77.6 MBytes   651 Mbits/sec
[  4]   6.00-7.00   sec  77.8 MBytes   653 Mbits/sec
[  4]   7.00-8.00   sec  77.7 MBytes   652 Mbits/sec
[  4]   8.00-9.00   sec  76.8 MBytes   644 Mbits/sec
[  4]   9.00-10.00  sec  79.1 MBytes   663 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   778 MBytes   653 Mbits/sec  152             sender
[  4]   0.00-10.00  sec   775 MBytes   650 Mbits/sec                  receiver

iperf Done.
iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from nanopcT4, port 51336
[  5] local nanopiNeo4 port 5201 connected to nanopcT4 port 51338
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  5]   0.00-1.00   sec   106 MBytes   889 Mbits/sec    0    404 KBytes
[  5]   1.00-2.00   sec   113 MBytes   946 Mbits/sec    0    404 KBytes
[  5]   2.00-3.00   sec   112 MBytes   936 Mbits/sec    0    404 KBytes
[  5]   3.00-4.00   sec   113 MBytes   947 Mbits/sec    0    445 KBytes
[  5]   4.00-5.00   sec   112 MBytes   943 Mbits/sec    1    328 KBytes
[  5]   5.00-6.00   sec   112 MBytes   940 Mbits/sec    0    369 KBytes
[  5]   6.00-7.00   sec   112 MBytes   939 Mbits/sec    0    392 KBytes
[  5]   7.00-8.00   sec   112 MBytes   942 Mbits/sec    0    428 KBytes
[  5]   8.00-9.00   sec   113 MBytes   945 Mbits/sec    0    428 KBytes
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0    443 KBytes
[  5]  10.00-10.02  sec  2.86 MBytes  1.06 Gbits/sec    0    443 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.02  sec  1.09 GBytes   937 Mbits/sec    1             sender
[  5]   0.00-10.02  sec  0.00 Bytes  0.00 bits/sec                  receiver

 

[Werner]

Quote

upgrade to Buster

 

running Bionic

So you upgraded from Bionic to Buster?

[ottawan]

It's not an 'upgrade' per se, rather a clean install. 

[Werner]

Just wanted to make sure...

Anyway. Both Buster and Bionic (if you did not switch branches) share the very same kernel. Therefore the issue must be searched on OS side and there is very little Armbian can do since the OS itself is barely touched.

[Igor]

5 hours ago, Werner said:

Anyway. Both Buster and Bionic (if you did not switch branches) share the very same kernel. Therefore the issue must be searched on OS side and there is very little Armbian can do since the OS itself is barely touched.

Or 3rd party hardware related. Such as cabling, switches, ...

 

@ottawan When https://github.com/armbian/autotests will receive more love, things could be cleared out at once. But now, someone needs to find some time and try to recreate ...

Edited March 5, 2020 by Igor
wording

[ottawan]

11 hours ago, Werner said:

if you did not switch branches

Great observation, thank you Werner!

 

Before the upgrade from Stretch (was not Bionic, sorry, my mistake) I was indeed running kernel v.5.x because back then I also saw some slow performance but blamed it on a new USB enclosure without proper testing, then just upgraded the kernel and that solved the problem. In fact I bet it was the very same thing.

 

Now after switching to 5.4.20 theres still some insignificant packet loss but guess I can live with it.

[ottawan]

6 hours ago, Igor said:

When https://github.com/armbian/autotests will receive more love

I'll definitely look into it now, seems neo4 is not quite popular out there, I must be one of a few 'lucky' ones :)))

[Igor]

3 hours ago, ottawan said:

I'll definitely look into it now,

I made some major changes today, but running low on time. Check this entire thread:

 

3 hours ago, ottawan said:

seems neo4 is not quite popular out there

We are mainly and well stock with bigger brother, M4 and T4. They are very similar.

[ottawan]

18 hours ago, Igor said:

M4 and T4. They are very similar.

 

That's kinda obvious, as all 3 share the very same on chip ethernet port.  But I'm still struggling to understand how come my T4 is running 4.4.192 just fine, but neo4 is experiencing a massive packet loss running 4.4.208 and one of the earlier 4.4.x versions but 5.4.20 and 5.3.x before - all good.

 

So now on that automated testing script topic: is it to be run as root or a regular user? Is it safe to run on a live system? Does it send anything automatically like armbianmonitor does? (To be honest - I was unpleasantly surprized seeing the data collected sent somewhere without review or even asking). 

[Igor]

1 hour ago, ottawan said:

as all 3 share the very same on chip ethernet port.

That is correct but experiences shows that things like PCB line length matters. Not necessarily this is the case ... just speculating.

 

1 hour ago, ottawan said:

is it to be run as root or a regular user? Is it safe to run on a live system? Does it send anything automatically like armbianmonitor does? 

- in this very early stage it runs as root. Working under normal user should be added under "to do" and rework accordingly.

- it should be safe to run it on a live system but testing might provoke your system to break. We actually want to move the system to the borders, which might result in bricking ... wireless speed test reconfigure your wireless devices and doesn't put them back, we will add more tests which I don't know what they will do ATM. This is not a end user application. 

- it doesn't send any data anywhere. This test is meant for us, for people that will attach their devices to common testing facility. We will collect data. I will sent my test results, you will sent your test results and we will display some results. They will tell us if this and that hardware feature is not working as expected or if its failed.

 

Nobody cares who is the person that made the test, so that data will not be collected at all.

 

1 hour ago, ottawan said:

I was unpleasantly surprized seeing the data collected sent somewhere without review or even asking

armbianmonitor sends out strictly anonymised data on your command and that data is stored on public servers. We take privacy and security seriously.

[ottawan]

3 hours ago, Igor said:

sends out strictly anonymised data

 

Yes I figured that out once I examined the data. But ony after it was uploaded.

 

When I ran armbianmonitor though I did not expect it to do automatic upload and my immediate reaction was WTF