ClearfogPro: Difference in switch behaviour between LK4.19 and LK5.8

[Heisath]

Armbianmonitor:

http://ix.io/2EIw

Hello,

 

I have discovered interesting behaviour of the clearfogpro switching between lk4.19 and 5.8 and would like to know if others are able to reproduce. And maybe have ideas on how to fix.

 

Steps to reproduce: 

- have a clearfogpro

- get the current armbian image https://archive.armbian.com/clearfogpro/archive/  20.08

- 'apt update && apt full-upgrade' 

- set the lan1-6 interfaces to bridging (example via following /etc/network/interfaces : https://pastebin.com/dpXEBe6g )

- reboot

- plug into your network should be GBit (on lan1-5). Verify you get ip, and can reach other devices.

- run iperf3 between clearfogpro and other gbit device in your network. Should yield about 900MBit/s

Now the interesting part:

- attach another device to the switch

- see how both status leds blink when transfering data via iperf3 ?     The bridge seems to run as a hub and not a switch.

- now plug a 100mbit/s device into the bridge.

- test speed to a gbit device with iperf (both directions): one direction will only yield about 100mbit/s. 

- while iperf3 is running to the gbit device, unplug and replug the 100mbit/s device on the switch, you should see the rate jump between 100 and 1000.....

 

NOTICE: This only happens in the direction where the clearfogpro is receiving data. Send speed is still at 1gbit/s.

 

Now use armbian-config and switch to LK4.19, reproduce steps from above. You should only see one activity led blinking and the switch functioning correctly at 1gbit/s even with a 100mbit/s device connected. 

 

So this leads me to think there's either some difference in the handling of the clearfogpro switch chip (between LK4 and 5) or some difference in bridge package. 

 

It would be great if someone could confirm so I am sure, it is not a problem with my setup.

 

root@clearfogpro:~# iperf3 -c 192.168.42.3 -t 100 -R
Connecting to host 192.168.42.3, port 5201
Reverse mode, remote host 192.168.42.3 is sending
[  5] local 192.168.42.106 port 33648 connected to 192.168.42.3 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  11.3 MBytes  94.3 Mbits/sec	# 100mbit/s device plugged into the switch 
[  5]   1.00-2.00   sec  11.2 MBytes  94.1 Mbits/sec
[  5]   2.00-3.00   sec  11.2 MBytes  94.1 Mbits/sec
[  5]   3.00-4.00   sec  11.2 MBytes  94.1 Mbits/sec
[  5]   4.00-5.00   sec  26.7 MBytes   224 Mbits/sec	# 100mbit/s device removed
[  5]   5.00-6.00   sec   107 MBytes   900 Mbits/sec
[  5]   6.00-7.00   sec   105 MBytes   878 Mbits/sec
[  5]   7.00-8.00   sec   107 MBytes   893 Mbits/sec
^C[  5]   8.00-8.61   sec  63.9 MBytes   885 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-8.61   sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-8.61   sec   454 MBytes   443 Mbits/sec                  receiver

 

[Heisath]

Quick update: Kernel 5.0.21 is also working fine. 5.1-rc1 has the defect. So there's some regression between those. Still trying to narrow it down.

[Heisath]

So I figured it out (and wasted 5 hours or so  ) ...

 

Support for bridge flags for the mv88e6xxx chips was introduced here: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/drivers/net/dsa/mv88e6xxx?h=v5.1&id=4f85901f0063e6f435125f8eb54d12e3108ab064

One of these flags is flooding. Which seems to be enabled by default, and leads to incoming packets on port A to be replayed on all other ports. This naturally lowers the receive speed on port A to the minimum transmit speed of all other ports.

 

Easy way to fix this is to issue 'bridge link set dev lan4 flood off' for all the lan ports on the clearfog.

I confirmed this also works on Lk5.8

 

Does it make sense to add a family tweak which does this for all lan ports? Or is this possible via device tree?

EDIT: Nevermind, this is not an issue with the dsa but with the default settings of a linux bridge. Let's not touch it, because not everyone will put the lan ports on the clearfog in a bridge. If they do, they hopefully stumble upon this and check the man page: https://www.man7.org/linux/man-pages/man8/bridge.8.html