1 1
tkaiser

Low Ethernet RX performance on Clearfog

Recommended Posts

Trying to tweak settings for NAS use cases I built a Debian Jessie for Clearfog Pro using vendor kernel (4.4.59-mvebu). Just for fun I made a Gigabit Ethernet LanTest with an USB3 RTL8153 -- target is a Seagate Barracuda on an ASM1062:

Bildschirmfoto%202017-04-13%20um%2018.47

 

Now with onboard eth0 it looks partially better but writes to the NAS are way lower:

Bildschirmfoto%202017-04-13%20um%2019.17

It seems RX packet processing happens on both CPU cores and slows the device unnecessarily down. But I found no way to adjust this:

root@clearfogpro:~# grep eth0 /proc/interrupts 
 38:    3532614    3873924  armada_370_xp_irq   8 Level     eth0
root@clearfogpro:~# echo 2 >/proc/irq/38/smp_affinity
-bash: echo: write error: Input/output error
root@clearfogpro:~# echo 1> /sys/class/net/eth0/queues/rx-0/rps_cpus
-bash: echo: write error: Invalid argument

Full debug log here: http://sprunge.us/UhAb 

 

Any ideas or pointers?

Share this post


Link to post
Share on other sites
1 hour ago, zador.blood.stained said:

Does it work on mainline?

 

Will try that later.  I'll make another test before: moving Clearfog around to test with another switch. Will update all this stuff tomorrow. Now playing with Zero Plus 2 and the NAS Expansion board.

Share this post


Link to post
Share on other sites

Well, crazy stuff happening:

Bildschirmfoto%202017-04-14%20um%2021.41

This is with 4.10.10-mvebu, it's still impossible to adjust IRQ affinity but NAS performance is magnitudes higher!

 

Debug log: http://sprunge.us/GVde

 

Still RX errors -- I'll grab my more professional switches tomorrow and try again. And will now put the Seagate Barracuda behind the ASM1062 since this test happened with the HDD connected to M.2 slot. I thought PCIe would not work with mainline but lspci lists the controller: 01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

 

BTW: Anyone has an idea which one is mPCI/mSATA slot 0 and 1? I need to patch u-boot for the mechanical mSATA-to-SATA connector in the right slot to work:

Clearfog_and_friends.jpg

Share this post


Link to post
Share on other sites
3 minutes ago, tkaiser said:

I thought PCIe would not work with mainline but lspci lists the controller

 

Different problem: PCIe controller is there, disk behind not:

root@clearfogpro:~# lspci
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04)
01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
root@clearfogpro:~# cat /proc/partitions 
major minor  #blocks  name

 179        0    7761920 mmcblk0
 179        1    3885824 mmcblk0p1
 179        2    3797439 mmcblk0p2
root@clearfogpro:~# uname -a
Linux clearfogpro 4.10.10-mvebu #7 SMP Fri Apr 14 00:46:32 CEST 2017 armv7l GNU/Linux
root@clearfogpro:~# armbianmonitor -u
/var/log/armhwinfo.log has been uploaded to http://sprunge.us/IVVS
Please post the URL in the Armbian forum where you've been asked for.

 

Share this post


Link to post
Share on other sites
23 minutes ago, tkaiser said:

Different problem: PCIe controller is there, disk behind not

Can you please get VID:PID for the controller? (lspci -v)

lspci displaying the device name doesn't mean that there is a driver for this device. And I don't see anything in dmesg that would suggest that the controller itself was detected.

Share this post


Link to post
Share on other sites

Back at 4.4.59-mvebu with HDD connected to M.2 port NAS performance still sucks. Numbers are not different compared to the Barracuda connected to ASM1062 before:

Bildschirmfoto%202017-04-14%20um%2022.14

 

So... next steps: figure out why ASM1062 doesn't like SATA disks connected to it on mainline kernel. I think I'll skip diagnosing what's wrong with 4.4.

Share this post


Link to post
Share on other sites
5 minutes ago, zador.blood.stained said:

Can you please get VID:PID for the controller? (lspci -v)

Full output on both kernels:

 

4.4.59-mvebu:

01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) (prog-if 01 [AHCI 1.0])
    Subsystem: ASMedia Technology Inc. Device 1060
    Flags: bus master, fast devsel, latency 0, IRQ 114
    I/O ports at 10020 [size=8]
    I/O ports at 10030 [size=4]
    I/O ports at 10028 [size=8]
    I/O ports at 10034 [size=4]
    I/O ports at 10000 [size=32]
    Memory at f8010000 (32-bit, non-prefetchable) [size=512]
    [virtual] Expansion ROM at f8000000 [disabled] [size=64K]
    Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
    Capabilities: [78] Power Management version 3
    Capabilities: [80] Express Legacy Endpoint, MSI 00
    Capabilities: [100] Virtual Channel
    Kernel driver in use: ahci

4.10.10-mvebu:

01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) (prog-if 01 [AHCI 1.0])
        Subsystem: ASMedia Technology Inc. Device 1060
        Flags: fast devsel, IRQ 53
        I/O ports at 10020 [disabled] [size=8]
        I/O ports at 10030 [disabled] [size=4]
        I/O ports at 10028 [disabled] [size=8]
        I/O ports at 10034 [disabled] [size=4]
        I/O ports at 10000 [disabled] [size=32]
        Memory at e0010000 (32-bit, non-prefetchable) [disabled] [size=512]
        [virtual] Expansion ROM at e0000000 [disabled] [size=64K]
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [78] Power Management version 3
        Capabilities: [80] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel

 

Share this post


Link to post
Share on other sites
37 minutes ago, zador.blood.stained said:

# CONFIG_SATA_AHCI is not set

Yep, you're my hero (again) :)

 

root@clearfogpro:~# cat /proc/partitions 
major minor  #blocks  name

 179        0    7761920 mmcblk0
 179        1    3885824 mmcblk0p1
 179        2    3797439 mmcblk0p2
   8        0 2930266584 sda
   8       16  117220824 sdb
   8       17     248832 sdb1
   8       18          1 sdb2
   8       21  116969472 sdb5
 254        0  100196352 dm-0
 254        1   16769024 dm-1

/dev/sda is the Barracuda connected to one of the ASM1062 ports. No difference in performance compared to native SATA:

Bildschirmfoto%202017-04-14%20um%2023.18

 

Hmm... I really hope someone's working on DVFS since the SoC get's freaking hot. Currently:

root@clearfogpro:~# cat /etc/armbianmonitor/datasources/soctemp 
70833

 

Share this post


Link to post
Share on other sites
4 minutes ago, tkaiser said:

 

Hmm... I really hope someone's working on DVFS since the SoC get's freaking hot.

I wanted to say that there was no progress last time I checked and wanted to link the reference stating the stability problems on multi-core CPUs (that's why DFS is disabled on A388), but a quick Google search gave me this: https://github.com/hnyman/source/commit/0ab4712d63ef1dd34fade8fe870c9de0ce41c1ac.patch

 

Share this post


Link to post
Share on other sites
11 hours ago, zador.blood.stained said:

Patch applied cleanly but installing .debs threw an error:

 

root@clearfogpro:~# dpkg -i /tmp/*.deb
(Reading database ... 61978 files and directories currently installed.)
Preparing to unpack .../linux-dtb-next-mvebu_5.27_armhf.deb ...
Unpacking linux-dtb-next-mvebu (5.27) over (5.27) ...
Preparing to unpack .../linux-headers-next-mvebu_5.27_armhf.deb ...
Unpacking linux-headers-next-mvebu (5.27) over (5.27) ...
Preparing to unpack .../linux-image-next-mvebu_5.27_armhf.deb ...
Unpacking linux-image-next-mvebu (5.27) over (5.27) ...
Selecting previously unselected package linux-u-boot-clearfogbase-next.
dpkg: regarding .../linux-u-boot-next-clearfogbase_5.27_armhf.deb containing linux-u-boot-clearfogbase-next:
 linux-u-boot-clearfogpro-next conflicts with armbian-u-boot
  linux-u-boot-clearfogbase-next provides armbian-u-boot and is to be installed.

dpkg: error processing archive /tmp/linux-u-boot-next-clearfogbase_5.27_armhf.deb (--install):
 conflicting packages - not installing linux-u-boot-clearfogbase-next
Preparing to unpack .../linux-u-boot-next-clearfogpro_5.27_armhf.deb ...
Unpacking linux-u-boot-clearfogpro-next (5.27) over (5.27) ...
Setting up linux-dtb-next-mvebu (5.27) ...
Setting up linux-headers-next-mvebu (5.27) ...
Compiling headers - please wait ...
Setting up linux-image-next-mvebu (5.27) ...
update-initramfs: Generating /boot/initrd.img-4.10.10-mvebu
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
W: mdadm: no configuration file available.
update-initramfs: Converting to u-boot format
Setting up linux-u-boot-clearfogpro-next (5.27) ...
Updating u-boot on device /dev/mmcblk0
Errors were encountered while processing:
 /tmp/linux-u-boot-next-clearfogbase_5.27_armhf.deb

 

 

After reboot no cpufreq sysfs nodes appear and temperature remains at high level (~65°C). Debug log here: http://sprunge.us/AFjd

Share this post


Link to post
Share on other sites

There are multiple patches in this directory that can be useful to us, I'll check them later.

Regarding your error - are you trying to install multiple u-boot packages or packages for Base on Pro? This will result in a conflict like in your log.

5 minutes ago, tkaiser said:

Selecting previously unselected package linux-u-boot-clearfogbase-next.

 

Share this post


Link to post
Share on other sites
24 minutes ago, zador.blood.stained said:

There are multiple patches in this directory that can be useful to us, I'll check them later.

 

Great! And you were right, I copied the u-boot package for Base too by accident.

Share this post


Link to post
Share on other sites
root@clearfogbase:~# cpufreq-info
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1
  CPUs which need to have their frequency coordinated by software: 0 1
  maximum transition latency: 1000 us.
  hardware limits: 800 MHz - 1.60 GHz
  available frequency steps: 800 MHz, 1.60 GHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance, schedutil
  current policy: frequency should be within 800 MHz and 1.60 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 800 MHz (asserted by call to hardware).
  cpufreq stats: 800 MHz:81.41%, 1.60 GHz:18.59%  (7)
analyzing CPU 1:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1
  CPUs which need to have their frequency coordinated by software: 0 1
  maximum transition latency: 1000 us.
  hardware limits: 800 MHz - 1.60 GHz
  available frequency steps: 800 MHz, 1.60 GHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance, schedutil
  current policy: frequency should be within 800 MHz and 1.60 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 800 MHz (asserted by call to hardware).
  cpufreq stats: 800 MHz:81.41%, 1.60 GHz:18.59%  (7)
root@clearfogbase:~#

 

Share this post


Link to post
Share on other sites

Nice, rebuilt the kernel, installed the debs, rebooted and have cpufreq scaling working though temperatures don't differ that much (it's maybe 1°C less):

14:22:48:  800MHz  0.13   7%   2%   3%   0%   1%   0% 62.7°C
14:22:53:  800MHz  0.12   7%   2%   3%   0%   1%   0% 63.2°C
14:22:58:  800MHz  0.19   7%   2%   3%   0%   1%   0% 63.2°C
14:23:03:  800MHz  0.17   7%   2%   3%   0%   1%   0% 63.2°C
14:23:08:  800MHz  0.16   7%   2%   2%   0%   1%   0% 63.2°C
14:23:13:  800MHz  0.15   7%   2%   3%   0%   1%   0% 63.2°C
14:23:18: 1600MHz  0.21   7%   2%   3%   0%   1%   0% 68.5°C
14:23:23: 1600MHz  0.28   8%   2%   3%   0%   1%   0% 68.0°C
14:23:28: 1600MHz  0.34   8%   2%   3%   0%   1%   0% 68.9°C
14:23:34: 1600MHz  0.39   9%   2%   4%   0%   1%   0% 69.4°C
14:23:39: 1600MHz  0.44   9%   2%   4%   0%   1%   0% 68.9°C
14:23:44: 1600MHz  0.48   9%   2%   5%   0%   1%   0% 69.4°C
14:23:49: 1600MHz  0.53  10%   2%   5%   0%   1%   0% 69.9°C

BTW: Switching to the upper clockspeed was caused by:

dkms install -m 8812au -v 4.2.2

Now investigating why 8812au driver didn't survived kernel upgrade while doing some stress tests to check for stability issues.

Share this post


Link to post
Share on other sites
Just now, tkaiser said:

Now investigating why 8812au driver didn't survived kernel upgrade while doing some stress tests to check for stability issues.

Because DKMS is broken since... hm... since Armbian repository was created?

Also DSA is broken for some reason in mainline, so it may be better to wait for 4.11 release which will have a lot of changes to the Armada boards, and then check the kernels status.

 

2 minutes ago, tkaiser said:

Nice, rebuilt the kernel, installed the debs, rebooted and have cpufreq scaling working though temperatures don't differ that much (it's maybe 1°C less)

It's almost 10°C less in idle (at 800MHz) for me, so it's at least something.

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.
1 1