matteobp Posted March 17, 2017 Posted March 17, 2017 Hi all. I have a problem with network performance. I have a Banana Pi M1+ with Debian Jessie 8.7, kernel 4.9.10-sunxi. I have already read these threadshttp://forum.lemaker.org/forum.php?mod=viewthread&tid=5802&highlight=performance http://forum.lemaker.org/forum.php?mod=viewthread&tid=12167 and based on them, I applied some changes. I added these settings is my /etc/sysctl.conf net.core.rmem_max = 8738000 net.core.wmem_max = 6553600 net.ipv4.tcp_rmem = 8192 873800 8738000 net.ipv4.tcp_wmem = 4096 655360 6553600 vm.min_free_kbytes = 65536 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1 net.ipv4.tcp_no_metrics_save = 1 and these ones in my /etc/sysfs.conf devices/system/cpu/cpufreq/ondemand/up_threshold = 25 devices/system/cpu/cpufreq/ondemand/sampling_down_factor = 10 devices/system/cpu/cpufreq/ondemand/io_is_busy = 1 This is my /etc/cpufrequtils ENABLE=true MIN_SPEED=528000 MAX_SPEED=1104000 GOVERNOR=ondemand I executed a first test with Helios Lan Test. Then I tried to apply these settings echo 1 > /proc/irq/$(cat /proc/interrupts | grep ahci-sunxi | cut -f 1 -d ":" | tr -d " ")/smp_affinity echo 2 > /proc/irq/$(cat /proc/interrupts | grep eth0 | cut -f 1 -d ":" | tr -d " ")/smp_affinity echo 2 > /sys/class/net/eth0/queues/rx-0/rps_cpus echo 2 > /sys/class/net/eth0/queues/tx-0/xps_cpus and execute a second test. Finally I tried to modify the samba configuration adding these setting in the [global] section restarting samba after changes: socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536 use sendfile = yes aio read size = 16384 aio write size = 16384 deadtime = 30 but the final result doesn't change. This image shows the result of three different tests, corresponding to the different settings. Consider that the shared folder W:\temp is on a SATA HDD 3.5" (WD RE4 1TB) connected to the SATA port of the Banana PI. I repeated the same test on the old SheevaPlug (with Gigabit ethernet), but in this case the shared folder is a SD Card folder. Here the result. Finally I did the same test connecting to another PC with Ubuntu 16.04 LTS, and this is the result. Why I have those very poor write values? Thanks Matteo
tkaiser Posted March 17, 2017 Posted March 17, 2017 Just some general advice: the aforementioned threads were all about legacy kernel. Adopting changes that worked there to mainline might be wrong use performance governor for all tests until you get sufficient performance, then switching to a different governor is worth a try please start with OS defaults first without tweaks. At least 30MB/s in both directions should be possible without any tweaks test storage and network individually too (iozone and iperf3 -- just search the forum) Just did a quick search and think I found it again: https://www.arm-blog.com/samba-finetuning-for-better-transfer-speeds/ (we're currently evaluating Samba to be used together with macOS clients which needs some special patches/settings and I tested with an OPi Plus 2E too and googled for these settings before which led to a severe performance boost)
matteobp Posted March 17, 2017 Author Posted March 17, 2017 Hi. Thanks for the tips. I removed all changes from /etc/sysctl.conf and /etc/sysfs.conf, and I set the governor = performance. This is the output of the command sysctl -a | grep mem net.core.optmem_max = 10240 net.core.rmem_default = 163840 net.core.rmem_max = 163840 net.core.wmem_default = 163840 net.core.wmem_max = 163840 net.ipv4.igmp_max_memberships = 20 net.ipv4.tcp_mem = 8829 11772 17658 net.ipv4.tcp_rmem = 4096 87380 6027744 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.udp_mem = 17658 23545 35316 net.ipv4.udp_rmem_min = 4096 net.ipv4.udp_wmem_min = 4096 sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.bond0.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.eth0.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" vm.highmem_is_dirtyable = 0 vm.lowmem_reserve_ratio = 32 32 vm.overcommit_memory = 0 They should be default values. The CPU frequency is set to 1104 Mhz. The result is almost the same. Here the result of the command iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2 on the SATA HD random random kB reclen write rewrite read reread read write 102400 4 14421 16996 33874 33599 1112 2104 102400 16 25076 25871 89751 89404 4133 7312 102400 512 35430 34143 92995 106661 51305 36084 102400 1024 32965 35947 95504 110381 61425 36641 102400 16384 33365 34797 107493 100003 118127 35999 and here the result of the iperf, using BananaPI and SheevaPlug as client/server. BananaPI as server iperf -s -P 0 -i 1 -p 5001 -f m ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 0.08 MByte (default) ------------------------------------------------------------ [ 4] local x.x.x.14 port 5001 connected with x.x.x.13 port 37964 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 92.3 MBytes 775 Mbits/sec [ 4] 1.0- 2.0 sec 97.8 MBytes 821 Mbits/sec [ 4] 2.0- 3.0 sec 96.9 MBytes 813 Mbits/sec [ 4] 3.0- 4.0 sec 101 MBytes 850 Mbits/sec [ 4] 4.0- 5.0 sec 100 MBytes 840 Mbits/sec [ 4] 5.0- 6.0 sec 103 MBytes 860 Mbits/sec [ 4] 6.0- 7.0 sec 101 MBytes 843 Mbits/sec [ 4] 7.0- 8.0 sec 101 MBytes 849 Mbits/sec [ 4] 8.0- 9.0 sec 100 MBytes 841 Mbits/sec [ 4] 9.0-10.0 sec 100 MBytes 842 Mbits/sec [ 4] 0.0-10.0 sec 994 MBytes 833 Mbits/sec SheevaPlug as client iperf -c x.x.x.14 -P 1 -i 1 -p 5001 -f m -t 10 -T 1 ------------------------------------------------------------ Client connecting to x.x.x.14, TCP port 5001 TCP window size: 0.02 MByte (default) ------------------------------------------------------------ [ 3] local x.x.x.13 port 37964 connected with x.x.x.14 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 92.8 MBytes 778 Mbits/sec [ 3] 1.0- 2.0 sec 97.8 MBytes 820 Mbits/sec [ 3] 2.0- 3.0 sec 96.6 MBytes 811 Mbits/sec [ 3] 3.0- 4.0 sec 102 MBytes 851 Mbits/sec [ 3] 4.0- 5.0 sec 100 MBytes 839 Mbits/sec [ 3] 5.0- 6.0 sec 103 MBytes 861 Mbits/sec [ 3] 6.0- 7.0 sec 101 MBytes 844 Mbits/sec [ 3] 7.0- 8.0 sec 101 MBytes 849 Mbits/sec [ 3] 8.0- 9.0 sec 100 MBytes 842 Mbits/sec [ 3] 9.0-10.0 sec 100 MBytes 840 Mbits/sec [ 3] 0.0-10.0 sec 994 MBytes 833 Mbits/sec SheevaPlug as server ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 0.08 MByte (default) ------------------------------------------------------------ [ 4] local x.x.x.13 port 5001 connected with x.x.x.14 port 52854 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 69.3 MBytes 581 Mbits/sec [ 4] 1.0- 2.0 sec 73.4 MBytes 616 Mbits/sec [ 4] 2.0- 3.0 sec 75.2 MBytes 631 Mbits/sec [ 4] 3.0- 4.0 sec 75.8 MBytes 635 Mbits/sec [ 4] 4.0- 5.0 sec 75.6 MBytes 634 Mbits/sec [ 4] 5.0- 6.0 sec 73.6 MBytes 618 Mbits/sec [ 4] 6.0- 7.0 sec 71.0 MBytes 595 Mbits/sec [ 4] 7.0- 8.0 sec 65.4 MBytes 548 Mbits/sec [ 4] 8.0- 9.0 sec 72.2 MBytes 606 Mbits/sec [ 4] 9.0-10.0 sec 71.6 MBytes 601 Mbits/sec [ 4] 0.0-10.0 sec 723 MBytes 606 Mbits/sec BananaPI as client ------------------------------------------------------------ Client connecting to x.x.x.13, TCP port 5001 TCP window size: 0.04 MByte (default) ------------------------------------------------------------ [ 3] local x.x.x.14 port 52854 connected with x.x.x.13 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 69.4 MBytes 582 Mbits/sec [ 3] 1.0- 2.0 sec 73.5 MBytes 617 Mbits/sec [ 3] 2.0- 3.0 sec 75.2 MBytes 631 Mbits/sec [ 3] 3.0- 4.0 sec 75.8 MBytes 635 Mbits/sec [ 3] 4.0- 5.0 sec 75.5 MBytes 633 Mbits/sec [ 3] 5.0- 6.0 sec 73.8 MBytes 619 Mbits/sec [ 3] 6.0- 7.0 sec 70.8 MBytes 593 Mbits/sec [ 3] 7.0- 8.0 sec 65.5 MBytes 549 Mbits/sec [ 3] 8.0- 9.0 sec 72.2 MBytes 606 Mbits/sec [ 3] 9.0-10.0 sec 71.5 MBytes 600 Mbits/sec [ 3] 0.0-10.0 sec 723 MBytes 607 Mbits/sec Seeing the result of iperf, I think the problem is related to samba, but I have no idea about the problem. Thanks for support. Matteo
matteobp Posted March 18, 2017 Author Posted March 18, 2017 Hi. I did a last try. I downloaded the latest Banana Pi+ Server images "Legacy 3.4.113" and "Mainline 4.9.7", burnt them onto SD card, installed only samba and samba-common, and ran the Helios Lan Test. Here the resutl with "Legacy 3.4.113" and here the result with "Mainline 4.9.7" I didn't apply any changes, so the problem is with Mainline kernel itself. Am I the only one with this problem? Please help me. Thanks Matteo P.S. I tried this wget -q -O - http://upgrade.armbian.com | bash but after that the BananaPi didn't boot anymore. I had to restore the image taken before this try. Then I compiled the legacy kernel by myself (on a VirtualBox VM with ubuntu 16.04 server) and installed all these 4 debs files linux-firmware-image-sun7i_5.27_armhf.deb linux-headers-sun7i_5.27_armhf.deb linux-image-sun7i_5.27_armhf.deb linux-u-boot-bananapipro_5.27_armhf.deb with dpkg -i *.deb but I had an error during installation and also in this case the BananaPi didn't boot anymore and I had to restore the image. Which is the correct way to switch to legacy kernel in my situation?
Igor Posted March 18, 2017 Posted March 18, 2017 1 hour ago, matteobp said: wget -q -O - http://upgrade.armbian.com | bash This is deprecated. (I just remove it from manual) 1 hour ago, matteobp said: but I had an error during installation Exactly what kind of error?
matteobp Posted March 19, 2017 Author Posted March 19, 2017 This is the output of the command dpkg -i *.deb Selecting previously unselected package linux-firmware-image-sun7i. (Reading database ... 142813 files and directories currently installed.) Preparing to unpack linux-firmware-image-sun7i_5.27_armhf.deb ... Unpacking linux-firmware-image-sun7i (5.27) ... dpkg: error processing archive linux-firmware-image-sun7i_5.27_armhf.deb (--install): trying to overwrite '/lib/firmware/ap6210/nvram_ap6210.txt', which is also in package armbian-firmware 5.25 dpkg-deb: error: subprocess paste was killed by signal (Broken pipe) Selecting previously unselected package linux-headers-sun7i. Preparing to unpack linux-headers-sun7i_5.27_armhf.deb ... Unpacking linux-headers-sun7i (5.27) ... Selecting previously unselected package linux-image-sun7i. Preparing to unpack linux-image-sun7i_5.27_armhf.deb ... Unpacking linux-image-sun7i (5.27) ... Selecting previously unselected package linux-u-boot-bananapipro-default. dpkg: regarding linux-u-boot-bananapipro_5.27_armhf.deb containing linux-u-boot-bananapipro-default: linux-u-boot-bananapipro-next conflicts with armbian-u-boot linux-u-boot-bananapipro-default provides armbian-u-boot and is to be installed. dpkg: error processing archive linux-u-boot-bananapipro_5.27_armhf.deb (--install): conflicting packages - not installing linux-u-boot-bananapipro-default Setting up linux-headers-sun7i (5.27) ... Compiling headers - please wait ... Setting up linux-image-sun7i (5.27) ... update-initramfs: Generating /boot/initrd.img-3.4.113-sun7i update-initramfs: Converting to u-boot format Errors were encountered while processing: linux-firmware-image-sun7i_5.27_armhf.deb linux-u-boot-bananapipro_5.27_armhf.deb Thanks for support Matteo
Igor Posted March 19, 2017 Posted March 19, 2017 1. Don't install linux-firmware-image-sun7i_5.27_armhf.deb (remove it) 2. Remove package: linux-u-boot-bananapipro-next 3. Run apt install -f
tkaiser Posted March 19, 2017 Posted March 19, 2017 Since I have to test through Samba 4.6 anyway (Changelog) and my BPi Pro is ready and doing some jobs I decided to give it there a try. So it might not help directly when using Samba distro packages. But to be honest: I gave up the idea to use Debian or Ubuntu fileserver distro packages long ago. Installation 'recipe' for Ubuntu Xenial (no idea whether dependencies match for Jessie): # https://wiki.samba.org/index.php/Build_Samba_from_Source # Use https://gist.github.com/touhonoob/5784297 as /etc/init.d/samba # and execute 'chmod 755 /etc/init.d/samba' and 'update-rc.d samba defaults' sudo apt-get install acl attr autoconf bison build-essential \ debhelper dnsutils docbook-xml docbook-xsl flex gdb krb5-user \ libacl1-dev libaio-dev libattr1-dev libblkid-dev libbsd-dev \ libcap-dev libcups2-dev libgnutls-dev libjson-perl \ libldap2-dev libncurses5-dev libpam0g-dev libparse-yapp-perl \ libpopt-dev libreadline-dev perl perl-modules pkg-config \ python-all-dev python-dev python-dnspython python-crypto \ xsltproc zlib1g-dev libsystemd-dev libgpgme11-dev python-gpgme \ python-m2crypto tracker libtracker-sparql-1.0-dev \ libtracker-miner-1.0-dev cd /usr/local/src/ wget https://download.samba.org/pub/samba/stable/samba-4.6.0.tar.gz tar -zxf samba-4.6.0.tar.gz cd samba-4.6.0/ ./configure --enable-selftest --enable-gnutls --enable-spotlight --with-systemd --download make -j $(grep -c processor /proc/cpuinfo) sudo make install This will take at least two 2 hours on an A20 devices (and ages if you're running off a slow SD card). I built a btrfs RAID-0 out of 2 Samsung SSDs to eliminate storage bottlenecks but performance was very low due to a couple of 'special settings' needed for macOS compatibility. So I just created a single ext4 partition on the SATA connected SSD, edited smb.conf to disable the 'special settings' and let LanTest run against: Performance still crappy but by looking at htop in parallel it's obvious that Samba gets bottlenecked by CPU (running single-threaded, I limit A20 at 960MHz max). My Iozone numbers without further tweaks: random random kB reclen write rewrite read reread read write 102400 4 17726 21178 45996 46009 25393 18961 102400 16 26660 28987 93169 97648 67964 28841 102400 512 32282 33169 160937 161359 139466 33054 102400 1024 32424 33278 163319 151169 157969 33216 102400 16384 33162 35184 193569 194492 187945 35350 'armbianmonitor -u' output: http://sprunge.us/KhTb ('nice': ATA bus error -- so the whole test was crap due to cabling issues)
matteobp Posted March 19, 2017 Author Posted March 19, 2017 Perfect, it worked. Now the average Helios LanTest result are these ones. Legacy kernel without changes in /etc/sysctl.conf and /etc/sysfs.conf (CPU frequency set to 1104 Mhz) Legacy kernel with the changes reported in the first post in /etc/sysctl.conf and /etc/sysfs.conf (CPU frequency set to 1104 Mhz) Considering that I started with an image based on the mainline kernel, now that I went back to legacy kernel installing these 3 debs files linux-headers-sun7i_5.27_armhf.deb linux-image-sun7i_5.27_armhf.deb linux-u-boot-bananapipro_5.27_armhf.deb is there something else I have to do? I'm asking this because after apt update I have this list of upgradable packages. linux-dtb-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26] linux-headers-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26] linux-image-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26] tzdata/stable-updates 2017a-0+deb8u1 all [upgradable from: 2016j-0+deb8u1] I dont' want to imagine what happens with an apt upgrade Thanks very much for the support.
tkaiser Posted March 19, 2017 Posted March 19, 2017 And this is the result of some quick&dirty 'tuning' (sending eth0 processing to 2nd CPU core entirely and adjusting SMP affinity and IO nice value for the specific smbd process serving my Mac): Last lines of 'history' and smb.conf contents: 1391 systemctl stop samba 1392 vi /usr/local/samba/etc/smb.conf + 1393 echo 2 > /proc/irq/$(cat /proc/interrupts | grep eth0 | cut -f 1 -d ":" | tr -d " ")/smp_affinity 1394 echo 2 > /sys/class/net/eth0/queues/rx-0/rps_cpus 1395 echo 2 > /sys/class/net/eth0/queues/tx-0/xps_cpus 1396 sysctl -w net/core/rmem_max=8738000 1397 sysctl -w net/core/wmem_max=6553600 1398 sysctl -w net/ipv4/tcp_rmem="8192 873800 8738000" 1399 sysctl -w net/ipv4/tcp_wmem="4096 655360 6553600" 1400 sysctl -w vm/min_free_kbytes=65536 1401 systemctl start samba 1402 ps auxww | grep smbd 1403 taskset -p 01 22583 1404 ionice -c1 -p 22583 1405 htop 1406 history root@bananapro:/mnt/raid-0# cat /usr/local/samba/etc/smb.conf [global] netbios name = bananapro server role = standalone workgroup = SLOW idmap config * : backend = tdb idmap config * : range = 10000-20000 disable netbios = yes smb ports = 445 debug pid = yes log file = /var/log/samba/%m.log log level = 0 read only = no ea support = yes store dos attributes = yes socket options = IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY min receivefile size = 2048 use sendfile = true aio read size = 16384 aio write size = 16384 write cache size = 1024000 read raw = yes write raw = yes getwd cache = yes oplocks = yes max xmit = 32768 dead time = 15 large readwrite = yes ; # for OS X symblink bug ; aio read size = 1024 ; aio write size = 1024 [osx] path = /mnt/raid-0/samba4_6 ; vfs objects = catia fruit streams_xattr acl_xattr
Igor Posted March 19, 2017 Posted March 19, 2017 24 minutes ago, matteobp said: I dont' want to imagine what happens with an apt upgrade When you switch kernel between DEFAULT, NEXT and DEV it's highly recommended that you remove the one you don't need. If you don't remove them from the system weird things can happen ... last updated kernel is bootable, hanging at boot, one core only, ... linux-dtb-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26] linux-headers-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26] linux-image-next-sunxi/jessie 5.26 armhf [upgradable from: 5.26] Just remove those if you will use legacy kernel.
matteobp Posted March 19, 2017 Author Posted March 19, 2017 Removed. There are other two packages that seems to refers to mainline. This is the output of dpkg -l | grep next ii linux-firmware-image-next-sunxi 5.26 armhf Linux kernel firmware, version 4.9.10-sunxi ii linux-jessie-root-next-bananapi 5.25 armhf Armbian tweaks for jessie on bananapi (next branch) Are they ok? If I remove "linux-firmware-image-next-sunxi", can I install "linux-firmware-image-sun7i_5.27_armhf.deb"? I had error during previous installation of this package. Thanks
Igor Posted March 19, 2017 Posted March 19, 2017 You don't need this firmware package. Remove both but install this: linux-jessie-root-bananapi
Recommended Posts