tkaiser Posted April 13, 2017 Posted April 13, 2017 Trying to tweak settings for NAS use cases I built a Debian Jessie for Clearfog Pro using vendor kernel (4.4.59-mvebu). Just for fun I made a Gigabit Ethernet LanTest with an USB3 RTL8153 -- target is a Seagate Barracuda on an ASM1062: Now with onboard eth0 it looks partially better but writes to the NAS are way lower: It seems RX packet processing happens on both CPU cores and slows the device unnecessarily down. But I found no way to adjust this: root@clearfogpro:~# grep eth0 /proc/interrupts 38: 3532614 3873924 armada_370_xp_irq 8 Level eth0 root@clearfogpro:~# echo 2 >/proc/irq/38/smp_affinity -bash: echo: write error: Input/output error root@clearfogpro:~# echo 1> /sys/class/net/eth0/queues/rx-0/rps_cpus -bash: echo: write error: Invalid argument Full debug log here: http://sprunge.us/UhAb Any ideas or pointers?
zador.blood.stained Posted April 13, 2017 Posted April 13, 2017 1 hour ago, tkaiser said: Any ideas or pointers? Does it work on mainline? Hard to say without kernel patching, there are different possibilities why setting IRQ affinity can be disabled.
tkaiser Posted April 13, 2017 Author Posted April 13, 2017 1 hour ago, zador.blood.stained said: Does it work on mainline? Will try that later. I'll make another test before: moving Clearfog around to test with another switch. Will update all this stuff tomorrow. Now playing with Zero Plus 2 and the NAS Expansion board.
tkaiser Posted April 14, 2017 Author Posted April 14, 2017 Well, crazy stuff happening: This is with 4.10.10-mvebu, it's still impossible to adjust IRQ affinity but NAS performance is magnitudes higher! Debug log: http://sprunge.us/GVde Still RX errors -- I'll grab my more professional switches tomorrow and try again. And will now put the Seagate Barracuda behind the ASM1062 since this test happened with the HDD connected to M.2 slot. I thought PCIe would not work with mainline but lspci lists the controller: 01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) BTW: Anyone has an idea which one is mPCI/mSATA slot 0 and 1? I need to patch u-boot for the mechanical mSATA-to-SATA connector in the right slot to work:
tkaiser Posted April 14, 2017 Author Posted April 14, 2017 3 minutes ago, tkaiser said: I thought PCIe would not work with mainline but lspci lists the controller Different problem: PCIe controller is there, disk behind not: root@clearfogpro:~# lspci 00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) 00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04) 01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) root@clearfogpro:~# cat /proc/partitions major minor #blocks name 179 0 7761920 mmcblk0 179 1 3885824 mmcblk0p1 179 2 3797439 mmcblk0p2 root@clearfogpro:~# uname -a Linux clearfogpro 4.10.10-mvebu #7 SMP Fri Apr 14 00:46:32 CEST 2017 armv7l GNU/Linux root@clearfogpro:~# armbianmonitor -u /var/log/armhwinfo.log has been uploaded to http://sprunge.us/IVVS Please post the URL in the Armbian forum where you've been asked for.
zador.blood.stained Posted April 14, 2017 Posted April 14, 2017 23 minutes ago, tkaiser said: Different problem: PCIe controller is there, disk behind not Can you please get VID:PID for the controller? (lspci -v) lspci displaying the device name doesn't mean that there is a driver for this device. And I don't see anything in dmesg that would suggest that the controller itself was detected.
tkaiser Posted April 14, 2017 Author Posted April 14, 2017 Back at 4.4.59-mvebu with HDD connected to M.2 port NAS performance still sucks. Numbers are not different compared to the Barracuda connected to ASM1062 before: So... next steps: figure out why ASM1062 doesn't like SATA disks connected to it on mainline kernel. I think I'll skip diagnosing what's wrong with 4.4.
tkaiser Posted April 14, 2017 Author Posted April 14, 2017 5 minutes ago, zador.blood.stained said: Can you please get VID:PID for the controller? (lspci -v) Full output on both kernels: 4.4.59-mvebu: 01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) (prog-if 01 [AHCI 1.0]) Subsystem: ASMedia Technology Inc. Device 1060 Flags: bus master, fast devsel, latency 0, IRQ 114 I/O ports at 10020 [size=8] I/O ports at 10030 [size=4] I/O ports at 10028 [size=8] I/O ports at 10034 [size=4] I/O ports at 10000 [size=32] Memory at f8010000 (32-bit, non-prefetchable) [size=512] [virtual] Expansion ROM at f8000000 [disabled] [size=64K] Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [78] Power Management version 3 Capabilities: [80] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Kernel driver in use: ahci 4.10.10-mvebu: 01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) (prog-if 01 [AHCI 1.0]) Subsystem: ASMedia Technology Inc. Device 1060 Flags: fast devsel, IRQ 53 I/O ports at 10020 [disabled] [size=8] I/O ports at 10030 [disabled] [size=4] I/O ports at 10028 [disabled] [size=8] I/O ports at 10034 [disabled] [size=4] I/O ports at 10000 [disabled] [size=32] Memory at e0010000 (32-bit, non-prefetchable) [disabled] [size=512] [virtual] Expansion ROM at e0000000 [disabled] [size=64K] Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [78] Power Management version 3 Capabilities: [80] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel
zador.blood.stained Posted April 14, 2017 Posted April 14, 2017 Most likely this is the reason # CONFIG_SATA_AHCI is not set
tkaiser Posted April 14, 2017 Author Posted April 14, 2017 37 minutes ago, zador.blood.stained said: # CONFIG_SATA_AHCI is not set Yep, you're my hero (again) root@clearfogpro:~# cat /proc/partitions major minor #blocks name 179 0 7761920 mmcblk0 179 1 3885824 mmcblk0p1 179 2 3797439 mmcblk0p2 8 0 2930266584 sda 8 16 117220824 sdb 8 17 248832 sdb1 8 18 1 sdb2 8 21 116969472 sdb5 254 0 100196352 dm-0 254 1 16769024 dm-1 /dev/sda is the Barracuda connected to one of the ASM1062 ports. No difference in performance compared to native SATA: Hmm... I really hope someone's working on DVFS since the SoC get's freaking hot. Currently: root@clearfogpro:~# cat /etc/armbianmonitor/datasources/soctemp 70833
zador.blood.stained Posted April 14, 2017 Posted April 14, 2017 4 minutes ago, tkaiser said: Hmm... I really hope someone's working on DVFS since the SoC get's freaking hot. I wanted to say that there was no progress last time I checked and wanted to link the reference stating the stability problems on multi-core CPUs (that's why DFS is disabled on A388), but a quick Google search gave me this: https://github.com/hnyman/source/commit/0ab4712d63ef1dd34fade8fe870c9de0ce41c1ac.patch 1
tkaiser Posted April 15, 2017 Author Posted April 15, 2017 11 hours ago, zador.blood.stained said: https://github.com/hnyman/source/commit/0ab4712d63ef1dd34fade8fe870c9de0ce41c1ac.patch Patch applied cleanly but installing .debs threw an error: root@clearfogpro:~# dpkg -i /tmp/*.deb (Reading database ... 61978 files and directories currently installed.) Preparing to unpack .../linux-dtb-next-mvebu_5.27_armhf.deb ... Unpacking linux-dtb-next-mvebu (5.27) over (5.27) ... Preparing to unpack .../linux-headers-next-mvebu_5.27_armhf.deb ... Unpacking linux-headers-next-mvebu (5.27) over (5.27) ... Preparing to unpack .../linux-image-next-mvebu_5.27_armhf.deb ... Unpacking linux-image-next-mvebu (5.27) over (5.27) ... Selecting previously unselected package linux-u-boot-clearfogbase-next. dpkg: regarding .../linux-u-boot-next-clearfogbase_5.27_armhf.deb containing linux-u-boot-clearfogbase-next: linux-u-boot-clearfogpro-next conflicts with armbian-u-boot linux-u-boot-clearfogbase-next provides armbian-u-boot and is to be installed. dpkg: error processing archive /tmp/linux-u-boot-next-clearfogbase_5.27_armhf.deb (--install): conflicting packages - not installing linux-u-boot-clearfogbase-next Preparing to unpack .../linux-u-boot-next-clearfogpro_5.27_armhf.deb ... Unpacking linux-u-boot-clearfogpro-next (5.27) over (5.27) ... Setting up linux-dtb-next-mvebu (5.27) ... Setting up linux-headers-next-mvebu (5.27) ... Compiling headers - please wait ... Setting up linux-image-next-mvebu (5.27) ... update-initramfs: Generating /boot/initrd.img-4.10.10-mvebu W: mdadm: /etc/mdadm/mdadm.conf defines no arrays. W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays. W: mdadm: failed to auto-generate temporary mdadm.conf file. W: mdadm: no configuration file available. update-initramfs: Converting to u-boot format Setting up linux-u-boot-clearfogpro-next (5.27) ... Updating u-boot on device /dev/mmcblk0 Errors were encountered while processing: /tmp/linux-u-boot-next-clearfogbase_5.27_armhf.deb After reboot no cpufreq sysfs nodes appear and temperature remains at high level (~65°C). Debug log here: http://sprunge.us/AFjd
zador.blood.stained Posted April 15, 2017 Posted April 15, 2017 There are multiple patches in this directory that can be useful to us, I'll check them later. Regarding your error - are you trying to install multiple u-boot packages or packages for Base on Pro? This will result in a conflict like in your log. 5 minutes ago, tkaiser said: Selecting previously unselected package linux-u-boot-clearfogbase-next.
tkaiser Posted April 15, 2017 Author Posted April 15, 2017 24 minutes ago, zador.blood.stained said: There are multiple patches in this directory that can be useful to us, I'll check them later. Great! And you were right, I copied the u-boot package for Base too by accident.
zador.blood.stained Posted April 15, 2017 Posted April 15, 2017 root@clearfogbase:~# cpufreq-info cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 CPUs which need to have their frequency coordinated by software: 0 1 maximum transition latency: 1000 us. hardware limits: 800 MHz - 1.60 GHz available frequency steps: 800 MHz, 1.60 GHz available cpufreq governors: conservative, userspace, powersave, ondemand, performance, schedutil current policy: frequency should be within 800 MHz and 1.60 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 800 MHz (asserted by call to hardware). cpufreq stats: 800 MHz:81.41%, 1.60 GHz:18.59% (7) analyzing CPU 1: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 CPUs which need to have their frequency coordinated by software: 0 1 maximum transition latency: 1000 us. hardware limits: 800 MHz - 1.60 GHz available frequency steps: 800 MHz, 1.60 GHz available cpufreq governors: conservative, userspace, powersave, ondemand, performance, schedutil current policy: frequency should be within 800 MHz and 1.60 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 800 MHz (asserted by call to hardware). cpufreq stats: 800 MHz:81.41%, 1.60 GHz:18.59% (7) root@clearfogbase:~# 1
tkaiser Posted April 15, 2017 Author Posted April 15, 2017 Nice, rebuilt the kernel, installed the debs, rebooted and have cpufreq scaling working though temperatures don't differ that much (it's maybe 1°C less): 14:22:48: 800MHz 0.13 7% 2% 3% 0% 1% 0% 62.7°C 14:22:53: 800MHz 0.12 7% 2% 3% 0% 1% 0% 63.2°C 14:22:58: 800MHz 0.19 7% 2% 3% 0% 1% 0% 63.2°C 14:23:03: 800MHz 0.17 7% 2% 3% 0% 1% 0% 63.2°C 14:23:08: 800MHz 0.16 7% 2% 2% 0% 1% 0% 63.2°C 14:23:13: 800MHz 0.15 7% 2% 3% 0% 1% 0% 63.2°C 14:23:18: 1600MHz 0.21 7% 2% 3% 0% 1% 0% 68.5°C 14:23:23: 1600MHz 0.28 8% 2% 3% 0% 1% 0% 68.0°C 14:23:28: 1600MHz 0.34 8% 2% 3% 0% 1% 0% 68.9°C 14:23:34: 1600MHz 0.39 9% 2% 4% 0% 1% 0% 69.4°C 14:23:39: 1600MHz 0.44 9% 2% 4% 0% 1% 0% 68.9°C 14:23:44: 1600MHz 0.48 9% 2% 5% 0% 1% 0% 69.4°C 14:23:49: 1600MHz 0.53 10% 2% 5% 0% 1% 0% 69.9°C BTW: Switching to the upper clockspeed was caused by: dkms install -m 8812au -v 4.2.2 Now investigating why 8812au driver didn't survived kernel upgrade while doing some stress tests to check for stability issues.
zador.blood.stained Posted April 15, 2017 Posted April 15, 2017 Just now, tkaiser said: Now investigating why 8812au driver didn't survived kernel upgrade while doing some stress tests to check for stability issues. Because DKMS is broken since... hm... since Armbian repository was created? Also DSA is broken for some reason in mainline, so it may be better to wait for 4.11 release which will have a lot of changes to the Armada boards, and then check the kernels status. 2 minutes ago, tkaiser said: Nice, rebuilt the kernel, installed the debs, rebooted and have cpufreq scaling working though temperatures don't differ that much (it's maybe 1°C less) It's almost 10°C less in idle (at 800MHz) for me, so it's at least something.
Recommended Posts