BananaPi Network performance problem


Recommended Posts

Hi all.
I have a problem with network performance. I have a Banana Pi M1+ with Debian Jessie 8.7, kernel 4.9.10-sunxi.

 

I have already read these threads
http://forum.lemaker.org/forum.php?mod=viewthread&tid=5802&highlight=performance

http://forum.lemaker.org/forum.php?mod=viewthread&tid=12167

and based on them, I applied some changes.


I added these settings is my /etc/sysctl.conf

net.core.rmem_max = 8738000
net.core.wmem_max = 6553600
net.ipv4.tcp_rmem = 8192 873800 8738000
net.ipv4.tcp_wmem = 4096 655360 6553600
vm.min_free_kbytes = 65536
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1

and these ones in my /etc/sysfs.conf

devices/system/cpu/cpufreq/ondemand/up_threshold = 25
devices/system/cpu/cpufreq/ondemand/sampling_down_factor = 10
devices/system/cpu/cpufreq/ondemand/io_is_busy = 1

This is my /etc/cpufrequtils

ENABLE=true
MIN_SPEED=528000
MAX_SPEED=1104000
GOVERNOR=ondemand

I executed a first test with Helios Lan Test.

Then I tried to apply these settings

echo 1 > /proc/irq/$(cat /proc/interrupts | grep ahci-sunxi | cut -f 1 -d ":" | tr -d " ")/smp_affinity
echo 2 > /proc/irq/$(cat /proc/interrupts | grep eth0 | cut -f 1 -d ":" | tr -d " ")/smp_affinity
echo 2 > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo 2 > /sys/class/net/eth0/queues/tx-0/xps_cpus

and execute a second test.

Finally I tried to modify the samba configuration adding these setting in the [global] section restarting samba after changes:

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
use sendfile = yes
aio read size = 16384
aio write size = 16384
deadtime = 30

but the final result doesn't change.

This image shows the result of three different tests, corresponding to the different settings.

 

58cc14efc3f59_HELIOSLanTest_bananapi_m1plus.png.a09e68578034ec2c06ea237b2c5a62f5.png

 

Consider that the shared folder W:\temp is on a SATA HDD 3.5" (WD RE4 1TB) connected to the SATA port of the Banana PI.

I repeated the same test on the old SheevaPlug (with Gigabit ethernet), but in this case the shared folder is a SD Card folder. Here the result.

58cc1668562c0_HELIOSLanTest_sheeva.png.b8a6d6a8fc2bd50d9d9484bb46ad2cf2.png

 

Finally I did the same test connecting to another PC with Ubuntu 16.04 LTS, and this is the result.

58cc16797a901_HELIOSLanTest_ubuntu1604.png.aef5b20ffba81bea7704baaa1bb4bfed.png

 

Why I have those very poor write values?

 

Thanks

Matteo

 

 

Link to post
Share on other sites
Donate and support the project!

Just some general advice:

  • the aforementioned threads were all about legacy kernel. Adopting changes that worked there to mainline might be wrong
  • use performance governor for all tests until you get sufficient performance, then switching to a different governor is worth a try
  • please start with OS defaults first without tweaks. At least 30MB/s in both directions should be possible without any tweaks
  • test storage and network individually too (iozone and iperf3 -- just search the forum)
  • Just did a quick search and think I found it again: https://www.arm-blog.com/samba-finetuning-for-better-transfer-speeds/  (we're currently evaluating Samba to be used together with macOS clients which needs some special patches/settings and I tested with an OPi Plus 2E too and googled for these settings before which led to a severe performance boost)
Link to post
Share on other sites

Hi.

Thanks for the tips.

I removed all changes from /etc/sysctl.conf and /etc/sysfs.conf, and I set the governor = performance.

This is the output of the command sysctl -a | grep mem

net.core.optmem_max = 10240
net.core.rmem_default = 163840
net.core.rmem_max = 163840
net.core.wmem_default = 163840
net.core.wmem_max = 163840
net.ipv4.igmp_max_memberships = 20
net.ipv4.tcp_mem = 8829 11772   17658
net.ipv4.tcp_rmem = 4096        87380   6027744
net.ipv4.tcp_wmem = 4096        16384   4194304
net.ipv4.udp_mem = 17658        23545   35316
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.bond0.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.eth0.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
vm.highmem_is_dirtyable = 0
vm.lowmem_reserve_ratio = 32    32
vm.overcommit_memory = 0

They should be default values. The CPU frequency is set to 1104 Mhz. The result is almost the same.

 

58cc36a627ccd_HELIOSLanTest_bananapi_m1plus_nochanges.png.5fca9d191d239f35d93f858fe06ba6e2.png

 

Here the result of the command

iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2

on the SATA HD

                                                      random    random
      kB  reclen    write  rewrite    read    reread    read     write
  102400       4    14421    16996    33874    33599     1112     2104
  102400      16    25076    25871    89751    89404     4133     7312
  102400     512    35430    34143    92995   106661    51305    36084
  102400    1024    32965    35947    95504   110381    61425    36641
  102400   16384    33365    34797   107493   100003   118127    35999

and here the result of the iperf, using BananaPI and SheevaPlug as client/server.

BananaPI as server
iperf -s -P 0 -i 1 -p 5001 -f m
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  4] local x.x.x.14 port 5001 connected with x.x.x.13 port 37964
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  92.3 MBytes   775 Mbits/sec
[  4]  1.0- 2.0 sec  97.8 MBytes   821 Mbits/sec
[  4]  2.0- 3.0 sec  96.9 MBytes   813 Mbits/sec
[  4]  3.0- 4.0 sec   101 MBytes   850 Mbits/sec
[  4]  4.0- 5.0 sec   100 MBytes   840 Mbits/sec
[  4]  5.0- 6.0 sec   103 MBytes   860 Mbits/sec
[  4]  6.0- 7.0 sec   101 MBytes   843 Mbits/sec
[  4]  7.0- 8.0 sec   101 MBytes   849 Mbits/sec
[  4]  8.0- 9.0 sec   100 MBytes   841 Mbits/sec
[  4]  9.0-10.0 sec   100 MBytes   842 Mbits/sec
[  4]  0.0-10.0 sec   994 MBytes   833 Mbits/sec

SheevaPlug as client
iperf -c x.x.x.14 -P 1 -i 1 -p 5001 -f m -t 10 -T 1
------------------------------------------------------------
Client connecting to x.x.x.14, TCP port 5001
TCP window size: 0.02 MByte (default)
------------------------------------------------------------
[  3] local x.x.x.13 port 37964 connected with x.x.x.14 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  92.8 MBytes   778 Mbits/sec
[  3]  1.0- 2.0 sec  97.8 MBytes   820 Mbits/sec
[  3]  2.0- 3.0 sec  96.6 MBytes   811 Mbits/sec
[  3]  3.0- 4.0 sec   102 MBytes   851 Mbits/sec
[  3]  4.0- 5.0 sec   100 MBytes   839 Mbits/sec
[  3]  5.0- 6.0 sec   103 MBytes   861 Mbits/sec
[  3]  6.0- 7.0 sec   101 MBytes   844 Mbits/sec
[  3]  7.0- 8.0 sec   101 MBytes   849 Mbits/sec
[  3]  8.0- 9.0 sec   100 MBytes   842 Mbits/sec
[  3]  9.0-10.0 sec   100 MBytes   840 Mbits/sec
[  3]  0.0-10.0 sec   994 MBytes   833 Mbits/sec

SheevaPlug as server
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  4] local x.x.x.13 port 5001 connected with x.x.x.14 port 52854
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  69.3 MBytes   581 Mbits/sec
[  4]  1.0- 2.0 sec  73.4 MBytes   616 Mbits/sec
[  4]  2.0- 3.0 sec  75.2 MBytes   631 Mbits/sec
[  4]  3.0- 4.0 sec  75.8 MBytes   635 Mbits/sec
[  4]  4.0- 5.0 sec  75.6 MBytes   634 Mbits/sec
[  4]  5.0- 6.0 sec  73.6 MBytes   618 Mbits/sec
[  4]  6.0- 7.0 sec  71.0 MBytes   595 Mbits/sec
[  4]  7.0- 8.0 sec  65.4 MBytes   548 Mbits/sec
[  4]  8.0- 9.0 sec  72.2 MBytes   606 Mbits/sec
[  4]  9.0-10.0 sec  71.6 MBytes   601 Mbits/sec
[  4]  0.0-10.0 sec   723 MBytes   606 Mbits/sec

BananaPI as client
------------------------------------------------------------
Client connecting to x.x.x.13, TCP port 5001
TCP window size: 0.04 MByte (default)
------------------------------------------------------------
[  3] local x.x.x.14 port 52854 connected with x.x.x.13 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  69.4 MBytes   582 Mbits/sec
[  3]  1.0- 2.0 sec  73.5 MBytes   617 Mbits/sec
[  3]  2.0- 3.0 sec  75.2 MBytes   631 Mbits/sec
[  3]  3.0- 4.0 sec  75.8 MBytes   635 Mbits/sec
[  3]  4.0- 5.0 sec  75.5 MBytes   633 Mbits/sec
[  3]  5.0- 6.0 sec  73.8 MBytes   619 Mbits/sec
[  3]  6.0- 7.0 sec  70.8 MBytes   593 Mbits/sec
[  3]  7.0- 8.0 sec  65.5 MBytes   549 Mbits/sec
[  3]  8.0- 9.0 sec  72.2 MBytes   606 Mbits/sec
[  3]  9.0-10.0 sec  71.5 MBytes   600 Mbits/sec
[  3]  0.0-10.0 sec   723 MBytes   607 Mbits/sec

Seeing the result of iperf, I think the problem is related to samba, but I have no idea about the problem. :(

 

Thanks for support.

Matteo

 

 

 

Link to post
Share on other sites

Hi.

I did a last try.

I downloaded the latest Banana Pi+ Server images "Legacy 3.4.113" and "Mainline 4.9.7", burnt them onto SD card, installed only samba and samba-common, and ran the Helios Lan Test.

 

Here the resutl with "Legacy 3.4.113"

 

jessie_default_sata_def_values.png.fce38cd68819c06b4400600e09881fc8.png

 

and here the result with "Mainline 4.9.7"

 

jessie_next_sata_def_values.png.671f954056f0987d3bcf232e3fe957eb.png

 

I didn't apply any changes, so the problem is with Mainline kernel itself.

Am I the only one with this problem?

Please help me.

Thanks

Matteo

 

P.S. I tried this

wget -q -O - http://upgrade.armbian.com | bash

but after that the BananaPi didn't boot anymore. I had to restore the image taken before this try.

Then I compiled the legacy kernel by myself (on a VirtualBox VM with ubuntu 16.04 server) and installed all these 4 debs files

linux-firmware-image-sun7i_5.27_armhf.deb
linux-headers-sun7i_5.27_armhf.deb
linux-image-sun7i_5.27_armhf.deb
linux-u-boot-bananapipro_5.27_armhf.deb

with

dpkg -i *.deb

but I had an error during installation and also in this case the BananaPi didn't boot anymore and I had to restore the image.

 

Which is the correct way to switch to legacy kernel in my situation?

 

 

Link to post
Share on other sites

This is the output of the command dpkg -i *.deb
 

Selecting previously unselected package linux-firmware-image-sun7i.
(Reading database ... 142813 files and directories currently installed.)
Preparing to unpack linux-firmware-image-sun7i_5.27_armhf.deb ...
Unpacking linux-firmware-image-sun7i (5.27) ...
dpkg: error processing archive linux-firmware-image-sun7i_5.27_armhf.deb (--install):
 trying to overwrite '/lib/firmware/ap6210/nvram_ap6210.txt', which is also in package armbian-firmware 5.25
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Selecting previously unselected package linux-headers-sun7i.
Preparing to unpack linux-headers-sun7i_5.27_armhf.deb ...
Unpacking linux-headers-sun7i (5.27) ...
Selecting previously unselected package linux-image-sun7i.
Preparing to unpack linux-image-sun7i_5.27_armhf.deb ...
Unpacking linux-image-sun7i (5.27) ...
Selecting previously unselected package linux-u-boot-bananapipro-default.
dpkg: regarding linux-u-boot-bananapipro_5.27_armhf.deb containing linux-u-boot-bananapipro-default:
 linux-u-boot-bananapipro-next conflicts with armbian-u-boot
  linux-u-boot-bananapipro-default provides armbian-u-boot and is to be installed.

dpkg: error processing archive linux-u-boot-bananapipro_5.27_armhf.deb (--install):
 conflicting packages - not installing linux-u-boot-bananapipro-default
Setting up linux-headers-sun7i (5.27) ...
Compiling headers - please wait ...
Setting up linux-image-sun7i (5.27) ...
update-initramfs: Generating /boot/initrd.img-3.4.113-sun7i
update-initramfs: Converting to u-boot format
Errors were encountered while processing:
 linux-firmware-image-sun7i_5.27_armhf.deb
 linux-u-boot-bananapipro_5.27_armhf.deb

Thanks for support

Matteo

 

Link to post
Share on other sites

Since I have to test through Samba 4.6 anyway (Changelog) and my BPi Pro is ready and doing some jobs I decided to give it there a try. So it might not help directly when using Samba distro packages. But to be honest: I gave up the idea to use Debian or Ubuntu fileserver distro packages long ago.

 

Installation 'recipe' for Ubuntu Xenial (no idea whether dependencies match for Jessie):

 

# https://wiki.samba.org/index.php/Build_Samba_from_Source
# Use https://gist.github.com/touhonoob/5784297 as /etc/init.d/samba
# and execute 'chmod 755 /etc/init.d/samba' and 'update-rc.d samba defaults'

sudo apt-get install acl attr autoconf bison build-essential \
  debhelper dnsutils docbook-xml docbook-xsl flex gdb krb5-user \
  libacl1-dev libaio-dev libattr1-dev libblkid-dev libbsd-dev \
  libcap-dev libcups2-dev libgnutls-dev libjson-perl \
  libldap2-dev libncurses5-dev libpam0g-dev libparse-yapp-perl \
  libpopt-dev libreadline-dev perl perl-modules pkg-config \
  python-all-dev python-dev python-dnspython python-crypto \
  xsltproc zlib1g-dev libsystemd-dev libgpgme11-dev python-gpgme \
  python-m2crypto tracker libtracker-sparql-1.0-dev \
  libtracker-miner-1.0-dev
cd /usr/local/src/
wget https://download.samba.org/pub/samba/stable/samba-4.6.0.tar.gz
tar -zxf samba-4.6.0.tar.gz
cd samba-4.6.0/
./configure --enable-selftest --enable-gnutls --enable-spotlight --with-systemd --download
make -j $(grep -c processor /proc/cpuinfo)
sudo make install

 

 

This will take at least two 2 hours on an A20 devices (and ages if you're running off a slow SD card). I built a btrfs RAID-0 out of 2 Samsung SSDs to eliminate storage bottlenecks but performance was very low due to a couple of 'special settings' needed for macOS compatibility. So I just created a single ext4 partition on the SATA connected SSD, edited smb.conf to disable the 'special settings' and let LanTest run against:

Bildschirmfoto%202017-03-19%20um%2017.09

 

 

Performance still crappy but by looking at htop in parallel it's obvious that Samba gets bottlenecked by CPU (running single-threaded, I limit A20 at 960MHz max). My Iozone numbers without further tweaks:

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    17726    21178    45996    46009    25393    18961
          102400      16    26660    28987    93169    97648    67964    28841
          102400     512    32282    33169   160937   161359   139466    33054
          102400    1024    32424    33278   163319   151169   157969    33216
          102400   16384    33162    35184   193569   194492   187945    35350

'armbianmonitor -u' output: http://sprunge.us/KhTb ('nice': ATA bus error -- so the whole test was crap due to cabling issues)

Link to post
Share on other sites

Perfect, it worked.

 

Now the average Helios LanTest result are these ones.

Legacy kernel without changes in /etc/sysctl.conf and /etc/sysfs.conf (CPU frequency set to 1104 Mhz)

 

legacy_def_values.png.fd247026a4733d97957c0ac34406eb1d.png

 

Legacy kernel with the changes reported in the first post in /etc/sysctl.conf and /etc/sysfs.conf (CPU frequency set to 1104 Mhz)

 

legacy_custom_values.png.06724047211b187905b14564226f6d61.png

 

Considering that I started with an image based on the mainline kernel, now that I went back to legacy kernel installing these 3 debs files

linux-headers-sun7i_5.27_armhf.deb
linux-image-sun7i_5.27_armhf.deb
linux-u-boot-bananapipro_5.27_armhf.deb

is there something else I have to do? I'm asking this because after apt update I have this list of upgradable packages.
 

linux-dtb-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26]
linux-headers-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26]
linux-image-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26]
tzdata/stable-updates 2017a-0+deb8u1 all [upgradable from: 2016j-0+deb8u1]

I dont' want to imagine what happens with an apt upgrade :(

 

Thanks very much for the support.

 

Link to post
Share on other sites

And this is the result of some quick&dirty 'tuning' (sending eth0 processing to 2nd CPU core entirely and adjusting SMP affinity and IO nice value for the specific smbd process serving my Mac):

Bildschirmfoto%202017-03-19%20um%2017.29

 

 

Last lines of 'history' and smb.conf contents:

 1391  systemctl stop samba
 1392  vi /usr/local/samba/etc/smb.conf +
 1393  echo 2 > /proc/irq/$(cat /proc/interrupts | grep eth0 | cut -f 1 -d ":" | tr -d " ")/smp_affinity
 1394  echo 2 > /sys/class/net/eth0/queues/rx-0/rps_cpus
 1395  echo 2 > /sys/class/net/eth0/queues/tx-0/xps_cpus
 1396  sysctl -w net/core/rmem_max=8738000
 1397  sysctl -w net/core/wmem_max=6553600
 1398  sysctl -w net/ipv4/tcp_rmem="8192 873800 8738000"
 1399  sysctl -w net/ipv4/tcp_wmem="4096 655360 6553600"
 1400  sysctl -w vm/min_free_kbytes=65536
 1401  systemctl start samba
 1402  ps auxww | grep smbd
 1403  taskset -p 01 22583
 1404  ionice -c1 -p 22583
 1405  htop
 1406  history
root@bananapro:/mnt/raid-0# cat /usr/local/samba/etc/smb.conf 
[global]
    netbios name = bananapro
    server role = standalone
    workgroup = SLOW

    idmap config * : backend = tdb
    idmap config * : range = 10000-20000
    disable netbios = yes
    smb ports = 445

    debug pid = yes
    log file = /var/log/samba/%m.log
    log level = 0

    read only = no
    ea support = yes
    store dos attributes = yes

socket options = IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
min receivefile size = 2048
use sendfile = true
aio read size = 16384
aio write size = 16384
write cache size = 1024000
read raw = yes
write raw = yes
getwd cache = yes
oplocks = yes
max xmit = 32768
dead time = 15
large readwrite = yes

;    # for OS X symblink bug
;    aio read size = 1024
;    aio write size = 1024

[osx]
    path = /mnt/raid-0/samba4_6
;    vfs objects = catia fruit streams_xattr acl_xattr

 

Link to post
Share on other sites
24 minutes ago, matteobp said:

I dont' want to imagine what happens with an apt upgrade

 

When you switch kernel between DEFAULT, NEXT and DEV it's highly recommended that you remove the one you don't need. If you don't remove them from the system weird things can happen ... last updated kernel is bootable, hanging at boot, one core only, ...

linux-dtb-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26] 
linux-headers-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26] 
linux-image-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26]

Just remove those if you will use legacy kernel.

Link to post
Share on other sites

Removed. There are other two packages that seems to refers to mainline. This is the output of dpkg -l | grep next
 

ii  linux-firmware-image-next-sunxi       5.26                              armhf        Linux kernel firmware, version 4.9.10-sunxi
ii  linux-jessie-root-next-bananapi       5.25                              armhf        Armbian tweaks for jessie on bananapi (next branch)

Are they ok? If I remove "linux-firmware-image-next-sunxi", can I install "linux-firmware-image-sun7i_5.27_armhf.deb"? I had error during previous installation of this package.

 

Thanks

 

Link to post
Share on other sites
Guest
This topic is now closed to further replies.