Jump to content

Not getting gigabit speeds from lan to wan?

pete11235

By pete11235
in Marvell mvebu

 Share

Recommended Posts

pete11235

pete11235

Posted
Posted

Hello! 

 

recently purchased the 2g espressobin and am not understanding why I cannot obtain gigabit speeds from lan to wan. I am able to do so from the OS out/in, but from a client machine to a machine attached to the wan I am getting 1/3 the speed of gigabit. Below is the iperf results along with my systemd/network settings. Does anyone have an idea why this is occurring?

 

Client to machine in Ethernet attached to wan: 

Connecting to host 192.168.2.1, port 5001
[  4] local 192.168.22.209 port 5859 connected to 192.168.2.1 port 5001
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  36.6 MBytes   307 Mbits/sec
[  4]   1.00-2.00   sec  42.4 MBytes   355 Mbits/sec
[  4]   2.00-3.00   sec  42.1 MBytes   354 Mbits/sec
[  4]   3.00-4.00   sec  42.0 MBytes   352 Mbits/sec
[  4]   4.00-5.00   sec  42.2 MBytes   355 Mbits/sec
[  4]   5.00-6.00   sec  42.4 MBytes   355 Mbits/sec
[  4]   6.00-7.00   sec  42.0 MBytes   353 Mbits/sec
[  4]   7.00-8.00   sec  42.5 MBytes   357 Mbits/sec
[  4]   8.00-9.00   sec  42.5 MBytes   356 Mbits/sec
[  4]   9.00-10.00  sec  42.0 MBytes   353 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   417 MBytes   350 Mbits/sec                  sender
[  4]   0.00-10.00  sec   417 MBytes   349 Mbits/sec                  receiver

iperf Done.


From espressobin OS to wan connected machine: 

root@espressobin:~# iperf3 -c 192.168.2.1 -p 5001
Connecting to host 192.168.2.1, port 5001
[  4] local 192.168.2.76 port 44154 connected to 192.168.2.1 port 5001
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   105 MBytes   879 Mbits/sec    0    708 KBytes
[  4]   1.00-2.00   sec   112 MBytes   938 Mbits/sec    0    708 KBytes
[  4]   2.00-3.00   sec   111 MBytes   934 Mbits/sec    0    708 KBytes
[  4]   3.00-4.00   sec   112 MBytes   939 Mbits/sec    0    708 KBytes
[  4]   4.00-5.00   sec   111 MBytes   933 Mbits/sec    0    708 KBytes
[  4]   5.00-6.00   sec   112 MBytes   939 Mbits/sec    0    708 KBytes
[  4]   6.00-7.00   sec   111 MBytes   934 Mbits/sec    0    708 KBytes
[  4]   7.00-8.00   sec   112 MBytes   939 Mbits/sec    0    708 KBytes
[  4]   8.00-9.00   sec   111 MBytes   933 Mbits/sec    0    708 KBytes
[  4]   9.00-10.00  sec   112 MBytes   939 Mbits/sec    0    708 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.08 GBytes   931 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.08 GBytes   931 Mbits/sec                  receiver

systemd/network config: 

root@espressobin:~# cat /etc/systemd/network/*
/etc/systemd/network/10-br0.netdev:
[NetDev]
Name=br0
Kind=bridge

/etc/systemd/network/10-br0.network:
[Match]
Name=br0

[Network]
Address=192.168.22.1/24
IPForward=ipv4
IPMasquerade=yes
ConfigureWithoutCarrier=yes

/etc/systemd/network/10-eth0.network:
[Match]
Name=eth0

[Network]
DHCP=ipv4

/etc/systemd/network/10-lan0.network:
[Match]
Name=lan0

[Network]
Bridge=br0

/etc/systemd/network/10-lan1.network:
[Match]
Name=lan1

[Network]
Bridge=br0

/etc/systemd/network/10-wan.network:
[Match]
Name=wan

[Network]
DHCP=yes
BindCarrier=eth0
IPForward=ipv4
root@espressobin:~#

Linux espressobin 4.18.16-mvebu64 #402 SMP PREEMPT Fri Oct 26 17:23:18 CEST 2018 aarch64 GNU/Linux

 

Link to comment
Share on other sites
ebin-dev

ebin-dev

Posted
Posted
14 hours ago, pete11235 said:

IPMasquerade=yes

 

You don`t use the default network configuration. Masquerading is expensive and reduces bandwidth.

You could try to redistribute tasks among CPU cores and to offload networking as far as possible.

Link to comment
Share on other sites
pete11235

pete11235

Posted
  • Author
Posted

@ebin-dev thanks for getting back to me! I removed it and re-ran the test. Results did not change. I thought this setup was pretty basic, however I understand where you are going with this, to pull as much off the CPU as possible. To make sure it wasnt anything I did, before making this post, I used a new SD card and did a fresh install, ensuring only to install dnsmasq. So this is basically a fresh install. I have attached the output of "ps aux":

https://pastebin.com/6tRccgN9

 

 

Link to comment
Share on other sites
pete11235

pete11235

Posted
  • Author
Posted

Also there is nearly no load when the espressobin is not being stressed. I have attached a screenshot to show that not many things are running. I cant imagine, based on what is running, that a basic network setup would consume enough resources to degrade from gigabit

TOP1.PNG

Link to comment
Share on other sites
ebin-dev

ebin-dev

Posted
Posted
On 11/27/2018 at 4:56 PM, pete11235 said:

Also there is nearly no load when the espressobin is not being stressed.

 

You need to have a look at the distribution of tasks while the EspressoBin is being stressed.

Do you use the latest bootloader flash-image? If you do not use a 1000_800 image you need to adapt the settings of /etc/default/cpufrequtils to available values output by 'cpufreq-info'.

Otherwise the ondemand governor does not ramp up correctly. Such an issue was described some time ago here.

Link to comment
Share on other sites
pete11235

pete11235

Posted
  • Author
Posted

Mine came set with the max at 1000000 and would reach 1000MHz during load. I increased the default min from 250000 to 500000 as it was recommended in the link you provided me. No change, even after reboot and I have confirmed they are operating at their set default speeds:

root@espressobin:~# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=500000
MAX_SPEED=1000000
GOVERNOR=ondemand
root@espressobin:~#
See attached screenshot. It will operate at 1000MHz the entire time the load is applied. In the attached screenshot, I hit CTRL-C half way through to show you that when it has no load, it drops back to 500MHz

Capture.thumb.PNG.105a16eb49ce05dcf3ce3aa3e350f5c9.PNG

Link to comment
Share on other sites
pete11235

pete11235

Posted
  • Author
Posted

Also to add on to this, on a separate SD card, I installed Archlinux to rule out the OS, and I had the same issue on Arch, see my other post here:
https://espressobin.net/forums/topic/espressobin-not-getting-gigabit-speeds-lan-to-wan/

 

So I think it may possibly be something in U-Boot? I am not sure, completely guessing here 

 

Also edit - U-Boot version: 

U-Boot 2017.03-armada-18.09.1-ga92bd86-armbian (Sep 05 2018 - 21:49:34 +0200)

Link to comment
Share on other sites
chrisf

chrisf

Posted
Posted

To me that load log says one core is maxed out due to IRQ activity.  You've got a single core doing all the network stuff and it can't handle any more load.

Do you need IP masquerading?

Is it possible to distribute network IRQ handling to both cores? If you can that may increase bandwidth, but may also slightly increase latency?

Link to comment
Share on other sites
pete11235

pete11235

Posted
  • Author
Posted

I've attempted to play around with the IRQ in order to get both CPU's active, but with no avail, take a looks below:
  

root@espressobin:~# cat /proc/irq/11/smp_affinity_list
0-1
root@espressobin:~# cat /proc/irq/11/smp_affinity
3
root@espressobin:~#



And this is when I put a load on the network:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    9 root      20   0       0      0      0 R  89.4  0.0   0:31.32 ksoftirqd/0
 1738 root      20   0    8028   3376   2828 R   0.7  0.2   0:00.11 top
   18 root      20   0       0      0      0 I   0.3  0.0   0:00.31 kworker/1:+
   25 root      20   0       0      0      0 I   0.3  0.0   0:00.02 kworker/0:+
    1 root      20   0   10408   6664   4820 S   0.0  0.3   0:03.44 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 rcu_gp
    4 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 rcu_par_gp
    5 root      20   0       0      0      0 I   0.0  0.0   0:00.08 kworker/0:+
    6 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/0:+
    7 root      20   0       0      0      0 I   0.0  0.0   0:00.79 kworker/u4+
    8 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 mm_percpu_+
   10 root      20   0       0      0      0 I   0.0  0.0   0:00.27 rcu_preempt
   11 root      20   0       0      0      0 I   0.0  0.0   0:00.12 rcu_sched
   12 root      20   0       0      0      0 I   0.0  0.0   0:00.00 rcu_bh
   13 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/0
   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/0

 

Am I looking in the wrong area?

Link to comment
Share on other sites
pete11235

pete11235

Posted
  • Author
Posted

I looked, but I am not sure, but I can see clearly something looks to be happening on both CPUs (7 & 8? - #11):

  

root@espressobin:~# cat /proc/interrupts
           CPU0       CPU1
  1:          0          0     GICv3  25 Level     vgic
  3:      20398      34130     GICv3  30 Level     arch_timer
  4:          0          0     GICv3  27 Level     kvm guest timer
  6:          0          0     GICv3  23 Level     arm-pmu
  7:          0          0     GICv3  32 Level     d0010600.spi
  9:      22472          0     GICv3  44 Level     d0012000.serial
 10:        125          0     GICv3  45 Level     d0012000.serial
 11:          8          7     GICv3  74 Level     eth0
 12:          0          0     GICv3  35 Level     xhci-hcd:usb2
 13:          0          0     GICv3  49 Level     ehci_hcd:usb1
 20:       8574          0     GICv3  57 Level     mmc1
 21:         52          0     GICv3  58 Level     mmc0
 22:          0          0     GICv3  59 Level     ahci-mvebu[d00e0000.sata]
 23:          0          0     GICv3  61 Level     advk-pcie
 41:          2          0     GICv3  79 Level     d0060900.xor
 42:          2          0     GICv3  80 Level     d0060900.xor
 43:          0          0     GPIO1   3 Edge      d00d0000.sdhci cd
 47:          0          0  mv88e6xxx-g1   3 Edge      mv88e6xxx-g1-atu-prob
 49:          0          0  mv88e6xxx-g1   5 Edge      mv88e6xxx-g1-vtu-prob
 51:          0          0  mv88e6xxx-g1   7 Edge      mv88e6xxx-g2
 54:          0          0  mv88e6xxx-g2   1 Edge      !soc!internal-regs@d0000000!mdio@32004!switch0@1!mdio:11
 55:          0          0  mv88e6xxx-g2   2 Edge      !soc!internal-regs@d0000000!mdio@32004!switch0@1!mdio:12
 56:          0          0  mv88e6xxx-g2   3 Edge      !soc!internal-regs@d0000000!mdio@32004!switch0@1!mdio:13
 68:          0          0  mv88e6xxx-g2  15 Edge      mv88e6xxx-watchdog
IPI0:      3966       6990       Rescheduling interrupts
IPI1:       119        525       Function call interrupts
IPI2:         0          0       CPU stop interrupts
IPI3:         0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0       Timer broadcast interrupts
IPI5:       872        646       IRQ work interrupts
IPI6:         0          0       CPU wake-up interrupts
Err:          0
root@espressobin:~#

I was under the assumption that smp_affinity_list would handle the CPU's, but I usually don't have to dive this deep into issues... 

EDIT: This was not under load

Link to comment
Share on other sites
Guest
This topic is now closed to further replies.
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept