MichaGue Posted October 21, 2015 Share Posted October 21, 2015 Hello, I am setting up a filserver, using a BPi and a WD Red 3TB connected to the sata port of the BPi. The BPi runs armbian v4.5 with vanilla kernel. For the first tuning I used the recommandations from tkaiser (thanks a lot for that). The only difference is, that I use ondemand as govenor and the frequency values out of the armbian image (thanks a lot to igore for the great work). root@bananapi:~# sysctl -a | grep mem net.core.optmem_max = 65535 net.core.rmem_default = 163840 net.core.rmem_max = 8738000 net.core.wmem_default = 163840 net.core.wmem_max = 6553600 net.ipv4.igmp_max_memberships = 20 net.ipv4.tcp_mem = 8874 11832 17748 net.ipv4.tcp_rmem = 8192 873800 8738000 net.ipv4.tcp_wmem = 4096 655360 6553600 net.ipv4.udp_mem = 17748 23664 35496 net.ipv4.udp_rmem_min = 4096 net.ipv4.udp_wmem_min = 4096 sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.bond0.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.eth0.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" sysctl: reading key "net.ipv6.conf.tunl0.stable_secret" vm.highmem_is_dirtyable = 0 vm.lowmem_reserve_ratio = 32 32 vm.overcommit_memory = 0 root@bananapi:~# cat /proc/interrupts CPU0 CPU1 17: 0 0 GIC 29 Edge arch_timer 18: 115380 107747 GIC 30 Edge arch_timer 21: 0 0 GIC 54 Level sun4i_timer0 22: 0 0 GIC 113 Level sun5i_timer0 26: 0 0 GIC 42 Level sun4i-spi 27: 2561171 0 GIC 64 Level sunxi-mmc 28: 0 0 GIC 71 Level ehci_hcd:usb1 29: 0 0 GIC 96 Level ohci_hcd:usb3 30: 757 0 GIC 88 Level 1c18000.sata 31: 0 0 GIC 72 Level ehci_hcd:usb2 32: 0 0 GIC 97 Level ohci_hcd:usb4 35: 0 0 GIC 56 Level 1c20d00.rtc 40: 3096 0 GIC 61 Level sun4i-ts 41: 18 0 GIC 33 Level serial 44: 198 0 GIC 39 Level mv64xxx_i2c 45: 0 0 GIC 41 Level mv64xxx_i2c 46: 8 5431 GIC 117 Level eth0 61: 0 0 - 10 Edge 1c0f000.mmc cd 83: 0 0 interrupt-controller 0 Level axp20x_irq_chip 117: 0 0 axp20x_irq_chip 33 Edge axp20x-pek-dbr 118: 0 0 axp20x_irq_chip 34 Edge axp20x-pek-dbf IPI0: 0 0 CPU wakeup interrupts IPI1: 0 0 Timer broadcast interrupts IPI2: 47869 1399468 Rescheduling interrupts IPI3: 0 0 Function call interrupts IPI4: 7 7 Single function call interrupts IPI5: 0 0 CPU stop interrupts IPI6: 0 0 IRQ work interrupts IPI7: 0 0 completion interrupts Err: 0 For seperate testing I run iperf for the network and iozone for the hdd (for iozone results look here). Using iperf with the above made tuning results in the following values. BPi => GBit-Switch => Notebook root@bananapi:~# iperf -s -P 0 -i 1 -p 5001 -f m ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 0.83 MByte (default) ------------------------------------------------------------ [ 4] local 192.168.0.18 port 5001 connected with 192.168.0.16 port 51470 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 319 MBytes 268 Mbits/sec Notebook => GBit-Switch => BPi root@bananapi:~# iperf -c 192.168.0.16 -P 1 -i 1 -p 5001 -f m -t 10 -T 1 ------------------------------------------------------------ Client connecting to 192.168.0.16, TCP port 5001 TCP window size: 0.62 MByte (default) ------------------------------------------------------------ [ 3] local 192.168.0.18 port 45284 connected with 192.168.0.16 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 654 MBytes 548 Mbits/sec The first thing I do not understand is the source of the default TCP window size iperf uses (0.83 MByte as server, 0.62 MByte as client). Which parameter in the linux configuration is responsible for that? After the first run I increased the TCP window size in iperf manually which results in much better bandwidth. BPi => GBit-Switch => Notebook (manually set TCP-Window size 768K, higher value did not increase the bandwidth) root@bananapi:~# iperf -s -P 0 -i 1 -p 5001 -w 768.0K -f m ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 1.50 MByte (WARNING: requested 0.75 MByte) ------------------------------------------------------------ [ 4] local 192.168.0.18 port 5001 connected with 192.168.0.16 port 51528 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1102 MBytes 924 Mbits/sec Notebook => GBit-Switch => BPi (manually set TCP-Window Size 512K, higher value did not increase the bandwidth) root@bananapi:~# iperf -c 192.168.0.16 -P 1 -i 1 -p 5001 -w 512.0K -f m -t 10 -T 1 ------------------------------------------------------------ Client connecting to 192.168.0.16, TCP port 5001 TCP window size: 1.00 MByte (WARNING: requested 0.50 MByte) ------------------------------------------------------------ [ 3] local 192.168.0.18 port 45302 connected with 192.168.0.16 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 780 MBytes 654 Mbits/sec So the final question is, which linux parameter has to be tuned to set the manually increased TCP window size in iperf permanet for the system? Thanks a lot for your help. Best regards, Michael Link to comment Share on other sites More sharing options...
MichaGue Posted October 21, 2015 Author Share Posted October 21, 2015 Sometimes it is usefull to write problems down to find the right solution. The answer to my question seems to be easy. Just increase the default value of net.ipv4.tcp_rmem = 8192 873800 8738000 #(min default max) net.ipv4.tcp_wmem = 4096 655360 6553600 #(min default max) will change the TCP window size. Measurements will follow. 1 Link to comment Share on other sites More sharing options...
MichaGue Posted October 21, 2015 Author Share Posted October 21, 2015 Here are the results. Both (server and client) are configured with the same TCP window size. BPi through the above told values (tuning default value) and Win7 PC through TCP window size setting in Jperf. With this I get really stable bandwidth of around 929 MBit/s (BPi => Notebook) and around 643 MBit/s (Notebook => BPi). BPi => GBit Switch => Notebook (Win7 running JPerf) root@bananapi:~# iperf -s -P 0 -i 1 -p 5001 -f m ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 1.00 MByte (default) ------------------------------------------------------------ [ 4] local 192.168.0.18 port 5001 connected with 192.168.0.16 port 53909 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 92.3 MBytes 774 Mbits/sec [ 4] 1.0- 2.0 sec 113 MBytes 949 Mbits/sec [ 4] 2.0- 3.0 sec 113 MBytes 950 Mbits/sec [ 4] 3.0- 4.0 sec 113 MBytes 949 Mbits/sec [ 4] 4.0- 5.0 sec 113 MBytes 949 Mbits/sec [ 4] 5.0- 6.0 sec 113 MBytes 944 Mbits/sec [ 4] 6.0- 7.0 sec 113 MBytes 946 Mbits/sec [ 4] 7.0- 8.0 sec 113 MBytes 947 Mbits/sec [ 4] 8.0- 9.0 sec 113 MBytes 944 Mbits/sec [ 4] 9.0-10.0 sec 112 MBytes 941 Mbits/sec [ 4] 0.0-10.0 sec 1109 MBytes 929 Mbits/sec Notebook (Win 7 running Jperf) => GBit Switch => BPi root@bananapi:~# iperf -c 192.168.0.16 -P 1 -i 1 -p 5001 -f m -t 10 -T 1 ------------------------------------------------------------ Client connecting to 192.168.0.16, TCP port 5001 TCP window size: 3.00 MByte (default) ------------------------------------------------------------ [ 3] local 192.168.0.18 port 55572 connected with 192.168.0.16 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 62.4 MBytes 523 Mbits/sec [ 3] 1.0- 2.0 sec 76.4 MBytes 641 Mbits/sec [ 3] 2.0- 3.0 sec 79.0 MBytes 663 Mbits/sec [ 3] 3.0- 4.0 sec 78.9 MBytes 662 Mbits/sec [ 3] 4.0- 5.0 sec 78.6 MBytes 660 Mbits/sec [ 3] 5.0- 6.0 sec 79.1 MBytes 664 Mbits/sec [ 3] 6.0- 7.0 sec 78.6 MBytes 660 Mbits/sec [ 3] 7.0- 8.0 sec 76.0 MBytes 638 Mbits/sec [ 3] 8.0- 9.0 sec 78.8 MBytes 661 Mbits/sec [ 3] 9.0-10.0 sec 78.4 MBytes 657 Mbits/sec [ 3] 0.0-10.0 sec 766 MBytes 643 Mbits/sec Link to comment Share on other sites More sharing options...
Recommended Posts