2 2
Spemerchina

Espressobin - etherchannel?

Recommended Posts

Hi everyone, 

 

especially armbian maintainers!

 

I'd like to make use of great performance of this board regarding SATA performance which is crippled by 1Gbps so my question is, would etherchannel work on this hardware? I understand that the board has only 1Gbps from the switch (dual LAN ports) so I guess the only possible way to exceed 1Gbps output would be to bundle WAN && [LAN01|LAN02]. Anyone tried this or believe it should work? I don't have the board to try.

 

Thanks.

Share this post


Link to post
Share on other sites

Here is the configuration of

 

Cisco switch side:

 

interface Port-channel2
 description LACP Channel for mk2
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,2
 switchport mode trunk
 spanning-tree portfast trunk
!
interface GigabitEthernet1/0/23
 description mk2 eth0
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 2 mode active
!
interface GigabitEthernet1/0/24
 description mk2 eth1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 2 mode active
!

 

armbian module config:

# /etc/modules: kernel modules to load at boot time.
 bonding mode=4 miimon=100 lacp_rate=1

 

armbian networking:

#/etc/network/interfaces 
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
 
auto eth0
    iface eth0 inet manual
    bond-master bond0
 
auto eth1
     iface eth1 inet manual
     bond-master bond0
 
auto bond0
     iface bond0 inet static
     address 10.0.0.80
     gateway 10.0.0.1
     netmask 255.255.255.0
 
 
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 4
bond-slaves none

thanks

Share this post


Link to post
Share on other sites

It was my understanding that the whole Topaz switch has only the 1 Gbps connection to the SOC, and that all three ethernet ports were from that switch.  The labels for wan, lan0, and lan1 are just labels, on switch ports 1, 2, and 3.

Share this post


Link to post
Share on other sites
On 2/7/2019 at 3:20 PM, ManoftheSea said:

It was my understanding that the whole Topaz switch has only the 1 Gbps connection to the SOC, and that all three ethernet ports were from that switch.  The labels for wan, lan0, and lan1 are just labels, on switch ports 1, 2, and 3.

 

unfortunately you are right, Topaz is connected to SoC via RGMII (1Gbps) so no chance for port channel. 

Share this post


Link to post
Share on other sites

How about with a mPCIE network card?

 

Currently there is Syba Gigabit Ethernet Mini PCI Express card on Amazon for about $17. That particular card only gets you one additional port, but I have seen options with two ports (although I haven't found any 2-port models anywhere near this price point).

 

Would it be possible to use a gigabit port on the PCI expansion slot in an etherchannel with one of the ports of the built-in switch to achieve a 2gbps link? I haven't figured out how to configure those built-in switch ports to anything other than the 3-port bridge yet, but I wonder how limited our options are with that Topaz switch in the middle.

 

If the 2-port gigabit expansion card wasn't so cost-prohibitive, I think the mPCIE slot would have the bandwidth to do 2gbps by itself. Judging by tkaiser's benchmarks of the mPCI SATA expansion board, it looks like he's hitting between 250,000 and nearly 300,000 kiloBytes/sec when using a single drive on the expansion board. Of course that's dependent on the drive and several other factors, but that gives us an upper limit of at least 1.9 to 2.3 gigabits/sec.

 

And of course this all assumes you're not already using the mPCIE slot for more SATA ports. :-\

Share this post


Link to post
Share on other sites
12 hours ago, Jbobspants said:

How about with a mPCIE network card?

 

Currently there is Syba Gigabit Ethernet Mini PCI Express card on Amazon for about $17. That particular card only gets you one additional port, but I have seen options with two ports (although I haven't found any 2-port models anywhere near this price point).

 

Would it be possible to use a gigabit port on the PCI expansion slot in an etherchannel with one of the ports of the built-in switch to achieve a 2gbps link? I haven't figured out how to configure those built-in switch ports to anything other than the 3-port bridge yet, but I wonder how limited our options are with that Topaz switch in the middle.

 

If the 2-port gigabit expansion card wasn't so cost-prohibitive, I think the mPCIE slot would have the bandwidth to do 2gbps by itself. Judging by tkaiser's benchmarks of the mPCI SATA expansion board, it looks like he's hitting between 250,000 and nearly 300,000 kiloBytes/sec when using a single drive on the expansion board. Of course that's dependent on the drive and several other factors, but that gives us an upper limit of at least 1.9 to 2.3 gigabits/sec.

 

And of course this all assumes you're not already using the mPCIE slot for more SATA ports. :-\

 

not a bad idea :) By looking at http://espressobin.net/wp-content/uploads/2017/01/ESPRESSObin-V3-Hardware-Block-diagram-v3-1.pdf , one could imagine that we might get much higher than 2Gbps if we go 10Gbps PCI-E adapter route.. SATA has it's own direct lane to SoC..

Share this post


Link to post
Share on other sites
On 2/15/2019 at 5:33 AM, Spemerchina said:

 

not a bad idea :) By looking at http://espressobin.net/wp-content/uploads/2017/01/ESPRESSObin-V3-Hardware-Block-diagram-v3-1.pdf , one could imagine that we might get much higher than 2Gbps if we go 10Gbps PCI-E adapter route.. SATA has it's own direct lane to SoC..

 

True, the onboard SATA has it's own direct path, and that's what some of tkaiser's benchmarks were testing...the onboard SATA port vs ports on his mPCIE SATA expansion board. It seems the onboard port is significantly faster, while the tests on the mPCIE SATA ports were limited to roughly 2Gbps. I'm not sure if the bottleneck was the cheap 88SE9215 SATA card, or the mPCIE path. I looked for specs on the mPCIE port, but was not able to determine how many PCIE lanes or what PCIE revision it uses (this might be common knowledge to the community, but I wasn't able to dig it up in a quick search).

 

A 10Gbps card would be very cool, but in the same test, he came to the conclusion that overall speed using both onboard and mPCIE SATA simultaneously (in RAID 0/striping mode) was limited by the CPU. I know SATA w/software RAID and network benchmarks are very different, but in a real-world scenario, I doubt we'd be able to get anywhere near 10Gbps.

Share this post


Link to post
Share on other sites
10 hours ago, Jbobspants said:

 

True, the onboard SATA has it's own direct path, and that's what some of tkaiser's benchmarks were testing...the onboard SATA port vs ports on his mPCIE SATA expansion board. It seems the onboard port is significantly faster, while the tests on the mPCIE SATA ports were limited to roughly 2Gbps. I'm not sure if the bottleneck was the cheap 88SE9215 SATA card, or the mPCIE path. I looked for specs on the mPCIE port, but was not able to determine how many PCIE lanes or what PCIE revision it uses (this might be common knowledge to the community, but I wasn't able to dig it up in a quick search).

 

A 10Gbps card would be very cool, but in the same test, he came to the conclusion that overall speed using both onboard and mPCIE SATA simultaneously (in RAID 0/striping mode) was limited by the CPU. I know SATA w/software RAID and network benchmarks are very different, but in a real-world scenario, I doubt we'd be able to get anywhere near 10Gbps.

image.png.6666a885e13bef33f292d08720f105a7.png

 

It looks like we have PCIe 2.0 which supports 500/500MB simultaneous read/write bandwidth. There are some cheap Asus 10Gbe adapters that could possible use all the bandwidth available (read from SATA, write to PCIe and in the opposite direction), so we can see close to 400/400 out of 10Gbe port without RAID overhead. I'm not interested in RAID, just a JBOD performance :)

 

Edit: Not sure about the exact wiring in Espressobin and capabilities of shared SERDES lanes among PCIe2.0 and SATA, some research says around 7.5Gbps total throughput, so I'll lower expectations to 3Gbe full duplex speeds with separte 10Gbe card in mPCI slot..

 

I believe @tkaiser could say a word or two on the matter :)

 

 

 

Share this post


Link to post
Share on other sites
12 hours ago, Spemerchina said:

Edit: Not sure about the exact wiring in Espressobin and capabilities of shared SERDES lanes among PCIe2.0 and SATA, some research says around 7.5Gbps total throughput, so I'll lower expectations to 3Gbe full duplex speeds with separte 10Gbe card in mPCI slot..

 

I believe @tkaiser could say a word or two on the matter :)

 

 

 

You have to make a choice with the shared SERDES lanes.

Espressobin has already made that for you with it's PCB layout - one lane goes to SATA, one to USB3 and one to PCIe.

There is only one lane on the PCIe slot, which is also apparently limited to 2.5Gb in early boards due issues running at PCI 2 speeds.

The switch is connected via a 1Gb link to the 3270.

 

Your physical limitations will be 2.5* or 5Gb on the PCIe slot and 3Gb on the SATA 2 connection. The 3Gb link uses 8b/10b encoding, so it's limited to 300MB/s.

 

 

* OpenWRT has reduce PCIe speed to 1.0 

https://git.openwrt.org/?p=openwrt/openwrt.git;a=commitdiff;h=772258044b48036699302840abf96cd34c4e5078

 

Share this post


Link to post
Share on other sites

Good info, guys!

 

@Spemerchina any chance you could post a link to that 10gb Asus card you mentioned?

 

My single-port network card arrived from Amazon today, along with a half-size to full-size mPCIE bracket. Everything went together easily and was recognized by Armbian right away. As soon as I booted there was a new "enp0s0" device in "ip link", and with a little fiddling in /etc/systemd/network/ I was able to get it to come up at boot and pull an IP automatically.

Spoiler

File: /etc/systemd/network/10-enp0s0.network


[Match]
Name=enp0s0

[Network]
DHCP=ipv4 

 

Not much time for testing today, unfortunately, and I'll need to bring home a managed switch from work in order to actually test etherchannel performance.

 

I also ordered this 2-port card from Ebay. It's coming from China so it will be 3 weeks, minimum, but at under $40 I figure it's worth a chance. Of course there are no details on the chip or anything, so there's no telling how much fighting I'll have getting the drivers to work.

Share this post


Link to post
Share on other sites

So I haven't been able to successfully get two ports into a bond yet, seems to be an issue with the Topaz switch. I'm not sure if there's a different configuration needed because of the way those three built-in interfaces are somehow sub-interfaces of eth0, I've never worked with anything in this configuration before. Basically the single port on the mPCIE card joins the bond and I get an IP address from DHCP on the bond interface, but I can't get any other interfaces to join the bond. I even tried creating a bond with lan0 and lan1 together (thinking those ports are more similar and might work better for some reason), but neither of them come up, and neither join the bond. Does anyone have a suggestion, considering the way this switch is managed?

 

I also tried a USB 3 gigabit adapter as a second port for testing, but I'm having weird driver issues. I'm using an ASIX AX88179 module which is listed as supported, but after a reboot it never comes up (no link light, although it works fine on my Fedora laptop). It seems to be recognized by the kernel, dmesg recognizes the device and creates an interface, but it just won't come online or show that it's connected. I tried unplugging/replugging the module a few times, once it actually came up on USB2 and worked fine (presumably at USB2 speeds), but I haven't been able to replicate that. Dmesg output from that time is below.

 

 

Spoiler

espressobin:/etc/systemd/network$ dmesg

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 4.19.20-mvebu64 (root@armbian.com) (gcc version 7.2.1 20171011 (Linaro GCC 7.2-2017.11)) #5.75 SMP PREEMPT Fri Feb 8 09:54:18 CET 2019
[    0.000000] Machine model: Globalscale Marvell ESPRESSOBin Board
[    0.000000] earlycon: ar3700_uart0 at MMIO 0x00000000d0012000 (options '')
[    0.000000] bootconsole [ar3700_uart0] enabled
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 16 MiB at 0x000000007dc00000
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000007fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x7ffea180-0x7ffeb93f]
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000000000000-0x000000007fffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000]   node   0: [mem 0x0000000004200000-0x000000007fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000007fffffff]
[    0.000000] On node 0 totalpages: 523776
[    0.000000]   DMA32 zone: 8192 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages reserved
[    0.000000]   DMA32 zone: 523776 pages, LIFO batch:63
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.1
[    0.000000] random: get_random_bytes called from start_kernel+0xa8/0x418 with crng_init=0
[    0.000000] percpu: Embedded 23 pages/cpu @(____ptrval____) s56984 r8192 d29032 u94208
[    0.000000] pcpu-alloc: s56984 r8192 d29032 u94208 alloc=23*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for ARM erratum 845719
[    0.000000] Speculative Store Bypass Disable mitigation not required
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 515584
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 root=UUID=a02e20f8-402e-4469-87e0-9d0c9507ea15 rootfstype=ext4 rootwait loglevel=1 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u mtdparts=spi0.0:1536k(uboot),64k(uboot-environment),-(reserved)
[    0.000000] Memory: 2021360K/2095104K available (11388K kernel code, 740K rwdata, 3228K rodata, 640K init, 517K bss, 57360K reserved, 16384K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=2.
[    0.000000]     Tasks RCU enabled.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: no VLPI support, no direct LPI support
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000d1d40000
[    0.000000] arch_timer: cp15 timer(s) running at 12.50MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2e2049cda, max_idle_ns: 440795202628 ns
[    0.000004] sched_clock: 56 bits at 12MHz, resolution 80ns, wraps every 4398046511080ns
[    0.000165] Console: colour dummy device 80x25
[    0.000236] Calibrating delay loop (skipped), value calculated using timer frequency.. 25.00 BogoMIPS (lpj=50000)
[    0.000245] pid_max: default: 32768 minimum: 301
[    0.000318] Security Framework initialized
[    0.001399] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.001938] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.001977] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.001998] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.024073] ASID allocator initialised with 32768 entries
[    0.032057] rcu: Hierarchical SRCU implementation.
[    0.040271] EFI services will not be available.
[    0.048087] smp: Bringing up secondary CPUs ...
[    0.080241] Detected VIPT I-cache on CPU1
[    0.080271] GICv3: CPU1: found redistributor 1 region 0:0x00000000d1d60000
[    0.080305] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.080398] smp: Brought up 1 node, 2 CPUs
[    0.080402] SMP: Total of 2 processors activated.
[    0.080407] CPU features: detected: GIC system register CPU interface
[    0.080411] CPU features: detected: 32-bit EL0 Support
[    0.083689] CPU: All CPU(s) started at EL2
[    0.083701] alternatives: patching kernel code
[    0.084557] devtmpfs: initialized
[    0.086958] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.086982] futex hash table entries: 512 (order: 3, 32768 bytes)
[    0.087556] xor: measuring software checksum speed
[    0.124254]    8regs     :  1919.000 MB/sec
[    0.164287]    8regs_prefetch:  1710.000 MB/sec
[    0.204329]    32regs    :  2359.000 MB/sec
[    0.244370]    32regs_prefetch:  1983.000 MB/sec
[    0.244373] xor: using function: 32regs (2359.000 MB/sec)
[    0.244433] pinctrl core: initialized pinctrl subsystem
[    0.245020] DMI not present or invalid.
[    0.245320] NET: Registered protocol family 16
[    0.245742] audit: initializing netlink subsys (disabled)
[    0.245912] audit: type=2000 audit(0.244:1): state=initialized audit_enabled=0 res=1
[    0.246509] cpuidle: using governor ladder
[    0.248459] cpuidle: using governor menu
[    0.248755] vdso: 2 pages (1 code @ (____ptrval____), 1 data @ (____ptrval____))
[    0.248761] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.250523] DMA: preallocated 2048 KiB pool for atomic allocations
[    0.250675] Serial: AMBA PL011 UART driver
[    0.260884] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.261258] cryptd: max_cpu_qlen set to 1000
[    0.328635] raid6: int64x1  gen()   389 MB/s
[    0.396696] raid6: int64x1  xor()   364 MB/s
[    0.464799] raid6: int64x2  gen()   590 MB/s
[    0.532902] raid6: int64x2  xor()   502 MB/s
[    0.600941] raid6: int64x4  gen()   897 MB/s
[    0.669062] raid6: int64x4  xor()   641 MB/s
[    0.737171] raid6: int64x8  gen()   834 MB/s
[    0.805235] raid6: int64x8  xor()   630 MB/s
[    0.873310] raid6: neonx1   gen()   619 MB/s
[    0.941368] raid6: neonx1   xor()   706 MB/s
[    1.009484] raid6: neonx2   gen()  1015 MB/s
[    1.077516] raid6: neonx2   xor()  1014 MB/s
[    1.145638] raid6: neonx4   gen()  1321 MB/s
[    1.213694] raid6: neonx4   xor()  1223 MB/s
[    1.281808] raid6: neonx8   gen()  1433 MB/s
[    1.349866] raid6: neonx8   xor()  1304 MB/s
[    1.349869] raid6: using algorithm neonx8 gen() 1433 MB/s
[    1.349871] raid6: .... xor() 1304 MB/s, rmw enabled
[    1.349874] raid6: using neon recovery algorithm
[    1.350097] ACPI: Interpreter disabled.
[    1.350624] vgaarb: loaded
[    1.350927] SCSI subsystem initialized
[    1.351134] libata version 3.00 loaded.
[    1.351313] usbcore: registered new interface driver usbfs
[    1.351344] usbcore: registered new interface driver hub
[    1.351392] usbcore: registered new device driver usb
[    1.351576] pps_core: LinuxPPS API ver. 1 registered
[    1.351579] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.351590] PTP clock support registered
[    1.352387] clocksource: Switched to clocksource arch_sys_counter
[    1.352530] VFS: Disk quotas dquot_6.6.0
[    1.352589] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.352745] pnp: PnP ACPI: disabled
[    1.358123] NET: Registered protocol family 2
[    1.358630] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes)
[    1.358703] TCP established hash table entries: 16384 (order: 5, 131072 bytes)
[    1.358871] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[    1.359116] TCP: Hash tables configured (established 16384 bind 16384)
[    1.359273] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[    1.359326] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[    1.359522] NET: Registered protocol family 1
[    1.359987] RPC: Registered named UNIX socket transport module.
[    1.359992] RPC: Registered udp transport module.
[    1.359994] RPC: Registered tcp transport module.
[    1.359997] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.360003] NET: Registered protocol family 44
[    1.360012] PCI: CLS 0 bytes, default 64
[    1.360198] Trying to unpack rootfs image as initramfs...
[    1.729592] Freeing initrd memory: 7704K
[    1.730160] hw perfevents: enabled with armv8_pmuv3 PMU driver, 7 counters available
[    1.730329] kvm [1]: 8-bit VMID
[    1.730625] kvm [1]: vgic-v2@d1da0000
[    1.730646] kvm [1]: GIC system register CPU interface enabled
[    1.730688] kvm [1]: vgic interrupt IRQ1
[    1.730754] kvm [1]: Hyp mode initialized successfully
[    1.734408] Initialise system trusted keyrings
[    1.734591] workingset: timestamp_bits=44 max_order=19 bucket_order=0
[    1.741385] NFS: Registering the id_resolver key type
[    1.741411] Key type id_resolver registered
[    1.741414] Key type id_legacy registered
[    1.741423] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    1.741588] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[    1.741944] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[    1.747310] NET: Registered protocol family 38
[    1.747324] Key type asymmetric registered
[    1.747328] Asymmetric key parser 'x509' registered
[    1.747384] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    1.747494] io scheduler noop registered
[    1.747498] io scheduler deadline registered
[    1.747672] io scheduler cfq registered (default)
[    1.747676] io scheduler mq-deadline registered
[    1.747680] io scheduler kyber registered
[    1.747786] io scheduler bfq registered
[    1.750765] advk-pcie d0070000.pcie: host bridge /soc/pcie@d0070000 ranges:
[    1.750790] advk-pcie d0070000.pcie:   MEM 0xe8000000..0xe8ffffff -> 0xe8000000
[    1.750802] advk-pcie d0070000.pcie:    IO 0xe9000000..0xe900ffff -> 0xe9000000
[    1.850850] advk-pcie d0070000.pcie: link up
[    1.851030] advk-pcie d0070000.pcie: PCI host bridge to bus 0000:00
[    1.851039] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.851045] pci_bus 0000:00: root bus resource [mem 0xe8000000-0xe8ffffff]
[    1.851052] pci_bus 0000:00: root bus resource [io  0x0000-0xffff] (bus address [0xe9000000-0xe900ffff])
[    1.851121] pci 0000:00:00.0: [10ec:8168] type 00 class 0x020000
[    1.851208] pci 0000:00:00.0: reg 0x10: [io  0x0000-0x00ff]
[    1.851283] pci 0000:00:00.0: reg 0x18: [mem 0xe8000000-0xe8000fff 64bit pref]
[    1.851333] pci 0000:00:00.0: reg 0x20: [mem 0xe8004000-0xe8007fff 64bit pref]
[    1.851571] pci 0000:00:00.0: supports D1 D2
[    1.851577] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    1.852560] pci 0000:00:00.0: BAR 4: assigned [mem 0xe8000000-0xe8003fff 64bit pref]
[    1.852601] pci 0000:00:00.0: BAR 2: assigned [mem 0xe8004000-0xe8004fff 64bit pref]
[    1.852637] pci 0000:00:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    1.854199] mv_xor d0060900.xor: Marvell shared XOR driver
[    1.885073] mv_xor d0060900.xor: Marvell XOR (Descriptor Mode): ( xor cpy intr )
[    1.913075] mv_xor d0060900.xor: Marvell XOR (Descriptor Mode): ( xor cpy intr )
[    1.915700] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    1.917331] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    1.918000] brd: module loaded
[    1.924599] loop: module loaded
[    1.925719] ahci-mvebu d00e0000.sata: AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl platform mode
[    1.925733] ahci-mvebu d00e0000.sata: flags: ncq sntf led only pmp fbs pio slum part sxs
[    1.926733] scsi host0: ahci-mvebu
[    1.926988] ata1: SATA max UDMA/133 mmio [mem 0xd00e0000-0xd00e1fff] port 0x100 irq 22
[    1.928044] m25p80 spi0.0: mx25u3235f (4096 Kbytes)
[    1.928064] 3 cmdlinepart partitions found on MTD device spi0.0
[    1.928067] Creating 3 MTD partitions on "spi0.0":
[    1.928075] 0x000000000000-0x000000180000 : "uboot"
[    1.928984] 0x000000180000-0x000000190000 : "uboot-environment"
[    1.929627] 0x000000190000-0x000000400000 : "reserved"
[    1.930750] libphy: Fixed MDIO Bus: probed
[    1.931110] tun: Universal TUN/TAP device driver, 1.6
[    1.931296] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    1.931298] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.931345] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[    1.931347] igb: Copyright (c) 2007-2014 Intel Corporation.
[    1.931386] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[    1.931389] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    1.931420] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[    1.931422] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    1.931615] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[    1.931618] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[    1.931731] ixgb: Intel(R) PRO/10GbE Network Driver - version 1.0.135-k2-NAPI
[    1.931734] ixgb: Copyright (c) 1999-2008 Intel Corporation.
[    1.932012] libphy: orion_mdio_bus: probed
[    1.933458] mvneta d0030000.ethernet eth0: Using device tree mac address f0:ad:4e:03:64:7f
[    1.933553] sky2: driver version 1.30
[    1.938847] libphy: r8169: probed
[    1.939589] r8169 0000:00:00.0 eth1: RTL8168evl/8111evl, 00:13:3b:12:17:15, XID 2c900800, IRQ 44
[    1.939597] r8169 0000:00:00.0 eth1: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[    1.940024] VFIO - User Level meta-driver version: 0.3
[    1.940679] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.940684] ehci-pci: EHCI PCI platform driver
[    1.940716] ehci-platform: EHCI generic platform driver
[    1.940793] ehci-orion: EHCI orion driver
[    1.940950] orion-ehci d005e000.usb: EHCI Host Controller
[    1.940971] orion-ehci d005e000.usb: new USB bus registered, assigned bus number 1
[    1.941520] orion-ehci d005e000.usb: irq 13, io mem 0xd005e000
[    1.956442] orion-ehci d005e000.usb: USB 2.0 started, EHCI 1.00
[    1.956586] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19
[    1.956592] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.956597] usb usb1: Product: EHCI Host Controller
[    1.956602] usb usb1: Manufacturer: Linux 4.19.20-mvebu64 ehci_hcd
[    1.956607] usb usb1: SerialNumber: d005e000.usb
[    1.957082] hub 1-0:1.0: USB hub found
[    1.957106] hub 1-0:1.0: 1 port detected
[    1.957432] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.957452] ohci-pci: OHCI PCI platform driver
[    1.957513] ohci-platform: OHCI generic platform driver
[    1.957919] xhci-hcd d0058000.usb: xHCI Host Controller
[    1.957938] xhci-hcd d0058000.usb: new USB bus registered, assigned bus number 2
[    1.958806] xhci-hcd d0058000.usb: hcc params 0x0a000998 hci version 0x100 quirks 0x0000000000010010
[    1.958858] xhci-hcd d0058000.usb: irq 12, io mem 0xd0058000
[    1.959169] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19
[    1.959176] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.959181] usb usb2: Product: xHCI Host Controller
[    1.959186] usb usb2: Manufacturer: Linux 4.19.20-mvebu64 xhci-hcd
[    1.959190] usb usb2: SerialNumber: d0058000.usb
[    1.959642] hub 2-0:1.0: USB hub found
[    1.959666] hub 2-0:1.0: 1 port detected
[    1.959923] xhci-hcd d0058000.usb: xHCI Host Controller
[    1.959938] xhci-hcd d0058000.usb: new USB bus registered, assigned bus number 3
[    1.959951] xhci-hcd d0058000.usb: Host supports USB 3.0  SuperSpeed
[    1.959996] usb usb3: We don't know the algorithms for LPM for this host, disabling LPM.
[    1.960083] usb usb3: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 4.19
[    1.960089] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.960094] usb usb3: Product: xHCI Host Controller
[    1.960099] usb usb3: Manufacturer: Linux 4.19.20-mvebu64 xhci-hcd
[    1.960104] usb usb3: SerialNumber: d0058000.usb
[    1.960485] hub 3-0:1.0: USB hub found
[    1.960508] hub 3-0:1.0: 1 port detected
[    1.960946] usbcore: registered new interface driver usb-storage
[    1.961272] mousedev: PS/2 mouse device common for all mice
[    1.961592] i2c /dev entries driver
[    1.961741] IR NEC protocol handler initialized
[    1.961744] IR RC5(x/sz) protocol handler initialized
[    1.961746] IR RC6 protocol handler initialized
[    1.961748] IR JVC protocol handler initialized
[    1.961751] IR Sony protocol handler initialized
[    1.961754] IR SANYO protocol handler initialized
[    1.961756] IR Sharp protocol handler initialized
[    1.961759] IR MCE Keyboard/mouse protocol handler initialized
[    1.961761] IR XMP protocol handler initialized
[    1.962346] device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: dm-devel@redhat.com
[    1.962641] sdhci: Secure Digital Host Controller Interface driver
[    1.962643] sdhci: Copyright(c) Pierre Ossman
[    1.962646] sdhci-pltfm: SDHCI platform and OF driver helper
[    1.962876] xenon-sdhci d00d0000.sdhci: Got CD GPIO
[    1.988992] mmc0: SDHCI controller on d00d8000.sdhci [d00d8000.sdhci] using ADMA
[    1.989653] ledtrig-cpu: registered to indicate activity on CPUs
[    1.990062] usbcore: registered new interface driver usbhid
[    1.990066] usbhid: USB HID core driver
[    1.991142] NET: Registered protocol family 10
[    1.992099] Segment Routing with IPv6
[    1.992157] NET: Registered protocol family 17
[    1.992223] Key type dns_resolver registered
[    1.992967] registered taskstats version 1
[    1.992972] Loading compiled-in X.509 certificates
[    1.994606] Btrfs loaded, crc32c=crc32c-generic
[    1.996814] d0012000.serial: ttyMV0 at MMIO 0xd0012000 (irq = 0, base_baud = 1562500) is a mvebu-uart
[    1.996839] console [ttyMV0] enabled
[    1.996842] bootconsole [ar3700_uart0] disabled
[    1.997799] xenon-sdhci d00d0000.sdhci: Got CD GPIO
[    1.997942] xenon-sdhci d00d0000.sdhci: Linked as a consumer to regulator.1
[    2.033240] mmc1: SDHCI controller on d00d0000.sdhci [d00d0000.sdhci] using ADMA
[    2.033827] hctosys: unable to open rtc device (rtc0)
[    2.095832] mmc1: new ultra high speed SDR104 SDHC card at address 0001
[    2.099245] mmcblk1: mmc1:0001 EB1QT 29.8 GiB
[    2.109374]  mmcblk1: p1
[    2.192970] usb 3-1: new SuperSpeed Gen 1 USB device number 2 using xhci-hcd
[    2.220255] usb 3-1: New USB device found, idVendor=0b95, idProduct=1790, bcdDevice= 1.00
[    2.220295] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    2.220326] usb 3-1: Product: AX88179
[    2.220354] usb 3-1: Manufacturer: ASIX Elec. Corp.
[    2.220460] usb 3-1: SerialNumber: 008CAE4CE9C7F5
[    2.404584] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    2.404997] ata1.00: ATA-10: KINGSTON SA400S37480G, SBFKB1C2, max UDMA/133
[    2.405029] ata1.00: 937703088 sectors, multi 1: LBA48 NCQ (depth 32)
[    2.405456] ata1.00: configured for UDMA/133
[    2.407791] scsi 0:0:0:0: Direct-Access     ATA      KINGSTON SA400S3 B1C2 PQ: 0 ANSI: 5
[    2.411524] sd 0:0:0:0: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[    2.411787] sd 0:0:0:0: [sda] Write Protect is off
[    2.411827] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.412280] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.417400]  sda: sda1 sda2 sda3 sda4
[    2.418820] sd 0:0:0:0: [sda] Attached SCSI removable disk
[    2.419224] Freeing unused kernel memory: 640K
[    2.419315] Run /init as init process
[    2.868476] r8169 0000:00:00.0 enp0s0: renamed from eth1
[    2.945302] mv88e6085 d0032004.mdio-mii:01: switch 0x3410 detected: Marvell 88E6341, revision 0
[    2.977213] random: fast init done
[    3.006173] libphy: mdio: probed
[    3.911722] mv88e6085 d0032004.mdio-mii:01 wan (uninitialized): PHY [!soc!internal-regs@d0000000!mdio@32004!switch0@1!mdio:11] driver [Marvell 88E6390]
[    3.991838] mv88e6085 d0032004.mdio-mii:01 lan0 (uninitialized): PHY [!soc!internal-regs@d0000000!mdio@32004!switch0@1!mdio:12] driver [Marvell 88E6390]
[    4.071870] mv88e6085 d0032004.mdio-mii:01 lan1 (uninitialized): PHY [!soc!internal-regs@d0000000!mdio@32004!switch0@1!mdio:13] driver [Marvell 88E6390]
[    4.078508] DSA: tree 0 setup
[    4.517265] EXT4-fs (mmcblk1p1): mounted filesystem with writeback data mode. Opts: (null)
[    4.947253] systemd[1]: System time before build time, advancing clock.
[    4.986578] systemd[1]: systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[    4.986944] systemd[1]: Detected architecture arm64.
[    5.009662] systemd[1]: Set hostname to <espressobin>.
[    5.574618] random: systemd: uninitialized urandom read (16 bytes read)
[    5.574697] systemd[1]: Reached target Swap.
[    5.584576] random: systemd: uninitialized urandom read (16 bytes read)
[    5.584893] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    5.601046] random: systemd: uninitialized urandom read (16 bytes read)
[    5.601960] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    5.616756] systemd[1]: Reached target Paths.
[    5.632099] systemd[1]: Created slice User and Session Slice.
[    5.646325] systemd[1]: Created slice System Slice.
[    5.656937] systemd[1]: Reached target Slices.
[    6.084494] EXT4-fs (mmcblk1p1): re-mounted. Opts: commit=600,errors=remount-ro
[    6.614148] systemd-journald[261]: Received request to flush runtime journal from PID 1
[    7.055559] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    7.153109] br0: port 1(lan1) entered blocking state
[    7.153147] br0: port 1(lan1) entered disabled state
[    7.161234] device lan1 entered promiscuous mode
[    7.161245] device eth0 entered promiscuous mode
[    7.173693] br0: port 2(lan0) entered blocking state
[    7.173724] br0: port 2(lan0) entered disabled state
[    7.182840] device lan0 entered promiscuous mode
[    7.192514] br0: port 3(wan) entered blocking state
[    7.192562] br0: port 3(wan) entered disabled state
[    7.199154] device wan entered promiscuous mode
[    7.212705] r8169 0000:00:00.0: Direct firmware load for rtl_nic/rtl8168e-3.fw failed with error -2
[    7.212728] r8169 0000:00:00.0 enp0s0: unable to load firmware patch rtl_nic/rtl8168e-3.fw (-2)
[    7.213019] RTL8211E Gigabit Ethernet r8169-0:00: attached PHY driver [RTL8211E Gigabit Ethernet] (mii_bus:phy_addr=r8169-0:00, irq=IGNORE)
[    7.342247] IPv6: ADDRCONF(NETDEV_UP): enp0s0: link is not ready
[    7.382424] mvneta d0030000.ethernet eth0: configuring for fixed/rgmii-id link mode
[    7.382528] mvneta d0030000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[    7.417020] mv88e6085 d0032004.mdio-mii:01 lan1: configuring for phy/ link mode
[    7.417481] br0: bridge flag offload is not supported 1(lan1)
[    7.419961] mv88e6085 d0032004.mdio-mii:01 lan0: configuring for phy/ link mode
[    7.420483] br0: bridge flag offload is not supported 2(lan0)
[    7.439291] mv88e6085 d0032004.mdio-mii:01 wan: configuring for phy/ link mode
[    7.440064] br0: bridge flag offload is not supported 3(wan)
[   10.132141] r8169 0000:00:00.0 enp0s0: Link is Up - 1Gbps/Full - flow control rx/tx
[   10.132183] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s0: link becomes ready
[   10.544842] mv88e6085 d0032004.mdio-mii:01 lan1: Link is Up - 1Gbps/Full - flow control rx/tx
[   10.645050] mv88e6085 d0032004.mdio-mii:01 lan0: Link is Up - 1Gbps/Full - flow control rx/tx
[   11.425754] ax88179_178a 3-1:1.0 eth1: register 'ax88179_178a' at usb-d0058000.usb-1, ASIX AX88179 USB 3.0 Gigabit Ethernet, 8c:ae:4c:e9:c7:f5
[   11.432599] usbcore: registered new interface driver ax88179_178a
[   12.711088] random: crng init done
[   12.711102] random: 7 urandom warning(s) missed due to ratelimiting
[   13.631806] ax88179_178a 3-1:1.0 enx8cae4ce9c7f5: renamed from eth1
[   13.924049] br0: port 2(lan0) entered blocking state
[   13.924062] br0: port 2(lan0) entered forwarding state
[   13.924187] br0: port 1(lan1) entered blocking state
[   13.924196] br0: port 1(lan1) entered forwarding state
[   13.926547] IPv6: ADDRCONF(NETDEV_UP): br0: link is not ready
[   13.938542] IPv6: ADDRCONF(NETDEV_CHANGE): br0: link becomes ready
[   14.342063] IPv6: ADDRCONF(NETDEV_UP): enx8cae4ce9c7f5: link is not ready
[  226.216153] xhci-hcd d0058000.usb: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 4
[  226.216204] xhci-hcd d0058000.usb: Looking for event-dma 0000000000000000 trb-start 000000007de797d0 trb-end 000000007de797d0 seg-start 000000007de79000 seg-end 000000007de79ff0
[  226.615674] usb 2-1: new high-speed USB device number 2 using xhci-hcd
[  226.770098] usb 2-1: New USB device found, idVendor=0b95, idProduct=1790, bcdDevice= 1.00
[  226.770138] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  226.770164] usb 2-1: Product: AX88179
[  226.770187] usb 2-1: Manufacturer: ASIX Elec. Corp.
[  226.770211] usb 2-1: SerialNumber: 008CAE4CE9C7F5
[  226.928312] usb 3-1: USB disconnect, device number 2
[  226.929270] ax88179_178a 3-1:1.0 enx8cae4ce9c7f5: unregister 'ax88179_178a' usb-d0058000.usb-1, ASIX AX88179 USB 3.0 Gigabit Ethernet
[  226.929648] ax88179_178a 3-1:1.0 enx8cae4ce9c7f5: Failed to read reg index 0x0002: -19
[  226.929695] ax88179_178a 3-1:1.0 enx8cae4ce9c7f5: Failed to write reg index 0x0002: -19
[  226.952016] ax88179_178a 3-1:1.0 enx8cae4ce9c7f5 (unregistered): Failed to write reg index 0x0002: -19
[  226.952075] ax88179_178a 3-1:1.0 enx8cae4ce9c7f5 (unregistered): Failed to write reg index 0x0001: -19
[  226.952118] ax88179_178a 3-1:1.0 enx8cae4ce9c7f5 (unregistered): Failed to write reg index 0x0002: -19
[  227.129719] ax88179_178a 2-1:1.0 eth1: register 'ax88179_178a' at usb-d0058000.usb-1, ASIX AX88179 USB 3.0 Gigabit Ethernet, 8c:ae:4c:e9:c7:f5
[  227.145659] ax88179_178a 2-1:1.0 enx8cae4ce9c7f5: renamed from eth1
[  227.577186] IPv6: ADDRCONF(NETDEV_UP): enx8cae4ce9c7f5: link is not ready
[  229.635218] usb 2-1: USB disconnect, device number 2
[  229.641388] ax88179_178a 2-1:1.0 enx8cae4ce9c7f5: unregister 'ax88179_178a' usb-d0058000.usb-1, ASIX AX88179 USB 3.0 Gigabit Ethernet
[  229.641822] ax88179_178a 2-1:1.0 enx8cae4ce9c7f5: Failed to read reg index 0x0002: -19
[  229.641870] ax88179_178a 2-1:1.0 enx8cae4ce9c7f5: Failed to write reg index 0x0002: -19
[  229.687724] ax88179_178a 2-1:1.0 enx8cae4ce9c7f5 (unregistered): Failed to write reg index 0x0002: -19
[  229.687783] ax88179_178a 2-1:1.0 enx8cae4ce9c7f5 (unregistered): Failed to write reg index 0x0001: -19
[  229.687825] ax88179_178a 2-1:1.0 enx8cae4ce9c7f5 (unregistered): Failed to write reg index 0x0002: -19
[  234.823791] usb 3-1: new SuperSpeed Gen 1 USB device number 3 using xhci-hcd
[  234.847151] usb 3-1: No LPM exit latency info found, disabling LPM.
[  234.847212] usb 3-1: no configurations
[  234.847244] usb 3-1: can't read configurations, error -22
[  235.071789] usb 3-1: new SuperSpeed Gen 1 USB device number 4 using xhci-hcd
[  235.095080] usb 3-1: No LPM exit latency info found, disabling LPM.
[  235.095140] usb 3-1: no configurations
[  235.095169] usb 3-1: can't read configurations, error -22
[  235.123870] usb usb3-port1: attempt power cycle
[  236.327372] usb 2-1: new high-speed USB device number 4 using xhci-hcd
[  236.481796] usb 2-1: New USB device found, idVendor=0b95, idProduct=1790, bcdDevice= 1.00
[  236.481835] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  236.481860] usb 2-1: Product: AX88179
[  236.481884] usb 2-1: Manufacturer: ASIX Elec. Corp.
[  236.481908] usb 2-1: SerialNumber: 008CAE4CE9C7F5
[  236.828813] ax88179_178a 2-1:1.0 eth1: register 'ax88179_178a' at usb-d0058000.usb-1, ASIX AX88179 USB 3.0 Gigabit Ethernet, 8c:ae:4c:e9:c7:f5
[  236.884168] ax88179_178a 2-1:1.0 enx8cae4ce9c7f5: renamed from eth1
[  237.361073] IPv6: ADDRCONF(NETDEV_UP): enx8cae4ce9c7f5: link is not ready
[  240.296068] ax88179_178a 2-1:1.0 enx8cae4ce9c7f5: ax88179 - Link status is: 1
[  240.308707] IPv6: ADDRCONF(NETDEV_CHANGE): enx8cae4ce9c7f5: link becomes ready
[  594.568500] mv88e6085 d0032004.mdio-mii:01 lan1: Link is Down
[  594.572192] br0: port 1(lan1) entered disabled state
[  594.601929] mv88e6085 d0032004.mdio-mii:01 lan0: Link is Down
[  594.604102] br0: port 2(lan0) entered disabled state
[  594.629329] br0: port 3(wan) entered disabled state
[  594.709901] mv88e6085 d0032004.mdio-mii:01 lan1: configuring for phy/ link mode
[  594.709959] mv88e6085 d0032004.mdio-mii:01 lan1: Link is Up - 1Gbps/Full - flow control rx/tx
[  594.717912] br0: port 1(lan1) entered blocking state
[  594.717927] br0: port 1(lan1) entered forwarding state
[  594.725552] br0: bridge flag offload is not supported 1(lan1)
[  594.754378] mv88e6085 d0032004.mdio-mii:01 lan0: configuring for phy/ link mode
[  594.754436] mv88e6085 d0032004.mdio-mii:01 lan0: Link is Up - 1Gbps/Full - flow control rx/tx
[  594.762629] br0: port 2(lan0) entered blocking state
[  594.762642] br0: port 2(lan0) entered forwarding state
[  594.762846] br0: bridge flag offload is not supported 2(lan0)
[  594.780645] mv88e6085 d0032004.mdio-mii:01 wan: configuring for phy/ link mode
[  594.787467] br0: bridge flag offload is not supported 3(wan)

 

 

Of particular note is the module is recognized at boot (2.192970), I unplug it at 226.216153, plug it back in at 234.823791, there's an issue with the device configuration and it apparently powercycles USB3, then recognizes it as USB2:

 

[ 235.071789] usb 3-1: new SuperSpeed Gen 1 USB device number 4 using xhci-hcd

[ 235.095080] usb 3-1: No LPM exit latency info found, disabling LPM.

[ 235.095140] usb 3-1: no configurations

[ 235.095169] usb 3-1: can't read configurations, error -22

[ 235.123870] usb usb3-port1: attempt power cycle

[ 236.327372] usb 2-1: new high-speed USB device number 4 using xhci-hcd

 

I'm not sure how normal that sequence is, I haven't used USB much on these SBCs, and I'm not really sure where to look next to try to get the USB3 adapter working at gigabit speeds.

 

Anyway, now I'm kinda stuck on etherchannel testing until I can solve one of the below problems:

  1. Figure out how to configure one of the Topaz switch ports in a bonded interface
  2. Figure out the USB3 driver issues on my USB module
  3. My 2-port mPCIE card arrives in a few weeks (and I can sort out any driver issues with it)

I'm definitely open to suggestions, and I will post any breakthroughs I might have.

Share this post


Link to post
Share on other sites
3 hours ago, Jbobspants said:

So I haven't been able to successfully get two ports into a bond yet

Why are you trying to bond two ports?

There is only a 1Gb link between the switch and the SoC, you're not going to get any more bandwidth

Share this post


Link to post
Share on other sites

Firs of all I admire everyone's quest for speed.   Finding every bit of performance available in these boards is a lot of fun.....  But allow me to provide a a brief sermon on link aggregation to manage expectations for those who many not have much experience.

 

So I've gone down the NIC bonding rabbit hole many times on many pieces of equipment.   I even have LACP trunks going to my garage on principle.    It's really really hard to get a performance payoff on bonded gig links.    Single TCP streams are still only sent down 1 link at a time.     So things like iperf testing, most basic file IO tests etc, won't even send traffic on more than 1 link.   The smarter hashing algorithms will load balance the links to a degree typically based on MAC or IP:port for service.   Aka servers with dozens and dozens of clients work well because the bandwidth accross the 2 links can be distributed.  Even protocols like NFS 4.1 and SMB3.0 that have concepts of parallism still don't perform well with just 2 endpoints.    Typically it's been backup servers that are being absolutely pounded at the same time by many nodes at night, or VM hosts that I've ever gotten over 1 gig of traffic on LACP.

 

The best performance I've been able to get using multiple gig links is by properly impementing iscsi with several initiators on a node and multipathing.   Since iSCSI is multipathing SCSI IO calls and not tcp it's able to deal wit the out of order stuff and you can really hammer your links.

 

Best performance bet probably is using 10G and maybe you'll hit a few gigsabits on that.   IMHO I'd just focus on shaving latency down on your services and keep your networking stack simple.    If performance is really an issue... i'd probaly just brute force with a junky intel box.. or by a more purpose built board.

 

PS in defense of the 1 gig SGMII lane using for the topaz switch to the CPU.... that's still 1GIG of full duplex traffic.. aka.. that's 1GIG of potential packet forwarding/filtering between WAN and LAN on the topaz.   In theory it's more potent than my edge router lite....  i'll have to test one day.

 

 

Share this post


Link to post
Share on other sites
(edited)
2 hours ago, chrisf said:

Why are you trying to bond two ports?

There is only a 1Gb link between the switch and the SoC, you're not going to get any more bandwidth

Yes, like @ManoftheSea pointed out above, we are aware that the three onboard ports have a 1gig bottleneck at the SoC. However, with a 1gbps mPCIE card in addition to one of the built-int ports, there is a theoretical 2gbps path. I was only testing the lan0 + lan1 etherchannel group with the sole purpose of trying to get any of the built-in ports to show up in an etherchannel/bond interface (unsuccessfully, so far).

 

@lanefu, I appreciate your experience and insight, and I'd say you're right-on with your assessment of "typical" traffic going to a home fileserver. You have a good point to level-set for anyone coming across this thread, and this most certainly wouldn't be a great option for a HTPC or home media server. From my perspective, however, the Espressobin has a [theoretical] 6gbps path to SATA, but it's a shame the network bottlenecks to 1gbps. In my hypothetical use-case, there would be several hosts simultaneously accessing data on this server. Each of those hosts would have a single 1gig NIC, so of course there would be no point in going above that speed for any single connection. However, as more and more hosts try to access this share a the same time, a handful of 1gig clients could easily exceed a single interface, so an etherchannel would make sense.

 

Also, my experience comes from a work environment where a single link is a potential single point of failure. We don't install anything without redundant links, and an etherchannel is a great way to allow for automated failover in the even of one link failing without having to run some additional heartbeat software. I realize that's probably not something your typical home user is concerned with, but IMO it would be cool to have.

 

One final note along the lines of @lanefu's post, I should mention that in all my tests so far with dual 1gig links to the Espressobin, I am hitting a CPU bottleneck before ever getting close to 2gbps network speed. Obviously I don't have an etherchannel working yet (and I really don't know how that will affect CPU utilization for network throughput), but with one of the built-in ports on VLAN A, and the mPCIE port on VLAN B, using four other test boxes (2 on VLAN B, 2 on VLAN A), all doing reads, writes, or simultaneous reads and writes, I have been unable to achieve much over 100MB/sec total before the Espressobin cores both peg at 100%. I've done a few tests with NFS exports, and a few with NBD exports, but all my tests so far have been limited by the CPU on the server. I'll continue to test and tweak my setup, but at this point, I'm not sure this is the right platform for a high speed NAS server.

 

All that said, I'm still trying to figure out how to add one of the built-in ports to a bond interface... Any suggestions would be greatly appreciated! :D

 

Edit: Just to clarify, 1gbps (gigabits per second) is about 125MB/sec, minus overhead. 2gbps would be about 250MB/sec, give or take.

 

Edited by Jbobspants
Additional clarification

Share this post


Link to post
Share on other sites

I feel like you would already know this, but, are you removing the ports from the bridge before you try to bond them?

 

Can you share your config files for how you're trying to bond?  With systemd-networkd?

Share this post


Link to post
Share on other sites
On 2/21/2019 at 4:57 AM, Jbobspants said:

Good info, guys!

 

@Spemerchina any chance you could post a link to that 10gb Asus card you mentioned?

 

My single-port network card arrived from Amazon today, along with a half-size to full-size mPCIE bracket. Everything went together easily and was recognized by Armbian right away. As soon as I booted there was a new "enp0s0" device in "ip link", and with a little fiddling in /etc/systemd/network/ I was able to get it to come up at boot and pull an IP automatically.

  Reveal hidden contents

File: /etc/systemd/network/10-enp0s0.network



[Match]
Name=enp0s0

[Network]
DHCP=ipv4 

 

Not much time for testing today, unfortunately, and I'll need to bring home a managed switch from work in order to actually test etherchannel performance.

 

I also ordered this 2-port card from Ebay. It's coming from China so it will be 3 weeks, minimum, but at under $40 I figure it's worth a chance. Of course there are no details on the chip or anything, so there's no telling how much fighting I'll have getting the drivers to work.

Hi, just google ASUS XG-C100C .

Share this post


Link to post
Share on other sites
20 hours ago, Spemerchina said:

Hi, just google ASUS XG-C100C .

Ah, OK that's a PCIE x4 card, I don't think we'll have much luck with it in an Espressobin mPCIE x1 slot. Might be fun to get a third 10gig port on a Machiatobin, though ;)

Share this post


Link to post
Share on other sites
20 hours ago, ManoftheSea said:

I feel like you would already know this, but, are you removing the ports from the bridge before you try to bond them?

 

Can you share your config files for how you're trying to bond?  With systemd-networkd?

 

Thanks for the nudge... from your post I assume it should just work like any other port. And yes, I had removed the port from the bridge before trying to add it to the bond. :)

 

Ok, so I went ahead and did a fresh install of the latest official download, 4.19.20-mvebu64. Reconfigured from scratch, and sure enough, I was able to get the "wan" port and my mPCIE "enp0s0" to both join bonded port. Things don't work exactly as I expect, though. From a fresh reboot, for whatever reason the wan port doesn't automatically join the bond, but the enp0s0 port does come up. If I do "systemctl restart systemd-networkd", it comes back up with both ports in the bond. I've also tried a shut/no shut on the switch, but it only comes back up with enp0s0, wan is still not in the bond until I restart systemd-networkd.

 

Full config from /etc/systemd/network, output of "ip addr" and "cat /proc/net/bonding/bond1" (before and after restarting systemd-networkd) are below.

 

Spoiler

Welcome to ARMBIAN 5.75 stable Ubuntu 18.04.1 LTS 4.19.20-mvebu64   
System load:   0.48 0.13 0.04  	Up time:       0 min		
Memory usage:  5 % of 1998MB 	IP:            
Usage of /:    2% of 59G    	

[ 0 security updates available, 3 updates total: apt upgrade ]
Last check: 2019-02-27 20:34

[ General system configuration (beta): armbian-config ]

Last login: Wed Feb 27 20:31:28 2019 from 192.168.1.99

[me]@espressobin:~$ 
[me]@espressobin:~$ 
[me]@espressobin:~$ 
[me]@espressobin:~$ for f in /etc/systemd/network/*; do echo ============; echo $f; cat $f; done
============
/etc/systemd/network/10-bond1.netdev
[NetDev]
Name=bond1
Kind=bond

[Bond]
Mode=802.3ad
TransmitHashPolicy=layer3+4
MIIMonitorSec=1s
LACPTransmitRate=fast
============
/etc/systemd/network/10-bond1.network
[Match]
Name=bond1

[Network]
DHCP=ipv4
BindCarrier=wan enp0s0
============
/etc/systemd/network/10-br0.netdev
[NetDev]
Name=br0
Kind=bridge
============
/etc/systemd/network/10-br0.network
[Match]
Name=br0

[Network]
Address=192.168.0.51/23
Gateway=192.168.1.0
DNS=192.168.1.0
============
/etc/systemd/network/10-enp0s0.network
[Match]
Name=enp0s0

[Network]
Bond=bond1
============
/etc/systemd/network/10-eth0.network
[Match]
Name=eth0

[Network]
DHCP=ipv4 
============
/etc/systemd/network/10-lan0.network
[Match]
Name=lan0

[Network]
Bridge=br0
============
/etc/systemd/network/10-lan1.network
[Match]
Name=lan1

[Network]
Bridge=br0
============
/etc/systemd/network/10-wan.network
[Match]
Name=wan 

[Network]
Bond=bond1
[me]@espressobin:~$ 
[me]@espressobin:~$ 
[me]@espressobin:~$ 
[me]@espressobin:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1024
    link/ether f0:ad:4e:03:64:7f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f2ad:4eff:fe03:647f/64 scope link 
       valid_lft forever preferred_lft forever
3: enp0s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond1 state UP group default qlen 1000
    link/ether 96:fa:b8:57:ad:c3 brd ff:ff:ff:ff:ff:ff
4: wan@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f0:ad:4e:03:64:7f brd ff:ff:ff:ff:ff:ff
5: lan0@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
    link/ether f0:ad:4e:03:64:7f brd ff:ff:ff:ff:ff:ff
6: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
    link/ether f0:ad:4e:03:64:7f brd ff:ff:ff:ff:ff:ff
7: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether a2:7a:46:64:45:cb brd ff:ff:ff:ff:ff:ff
8: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 96:fa:b8:57:ad:c3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.52/24 brd 192.168.2.255 scope global dynamic bond1
       valid_lft 86378sec preferred_lft 86378sec
    inet6 fe80::94fa:b8ff:fe57:adc3/64 scope link 
       valid_lft forever preferred_lft forever
9: enx8cae4ce9c7f5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8c:ae:4c:e9:c7:f5 brd ff:ff:ff:ff:ff:ff
[me]@espressobin:~$ 
[me]@espressobin:~$ cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 1000
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable

Slave Interface: enp0s0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:13:3b:12:17:15
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0
[me]@espressobin:~$ 
[me]@espressobin:~$ 
[me]@espressobin:~$ sudo systemctl restart systemd-networkd
[sudo] password for [me]: 
[me]@espressobin:~$ 
[me]@espressobin:~$ 
[me]@espressobin:~$ 
[me]@espressobin:~$ 
[me]@espressobin:~$ cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 1000
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable

Slave Interface: enp0s0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: 00:13:3b:12:17:15
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0

Slave Interface: wan
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: f0:ad:4e:03:64:7f
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
[me]@espressobin:~$ 
[me]@espressobin:~$ 
[me]@espressobin:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1024
    link/ether f0:ad:4e:03:64:7f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f2ad:4eff:fe03:647f/64 scope link 
       valid_lft forever preferred_lft forever
3: enp0s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond1 state UP group default qlen 1000
    link/ether 96:fa:b8:57:ad:c3 brd ff:ff:ff:ff:ff:ff
4: wan@eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc noqueue master bond1 state UP group default qlen 1000
    link/ether 96:fa:b8:57:ad:c3 brd ff:ff:ff:ff:ff:ff
5: lan0@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
    link/ether f0:ad:4e:03:64:7f brd ff:ff:ff:ff:ff:ff
6: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
    link/ether f0:ad:4e:03:64:7f brd ff:ff:ff:ff:ff:ff
7: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether a2:7a:46:64:45:cb brd ff:ff:ff:ff:ff:ff
8: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 96:fa:b8:57:ad:c3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.52/24 brd 192.168.2.255 scope global dynamic bond1
       valid_lft 85933sec preferred_lft 85933sec
    inet6 fe80::94fa:b8ff:fe57:adc3/64 scope link 
       valid_lft forever preferred_lft forever
9: enx8cae4ce9c7f5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8c:ae:4c:e9:c7:f5 brd ff:ff:ff:ff:ff:ff
[me]@espressobin:~$ 
[me]@espressobin:~$ 

 

 

On a related note, my USB3 network adapter is still not getting link light after a reboot, but that is less of an issue now that I have 2 other ports successfully joined to the bond.

Share this post


Link to post
Share on other sites

@Jbobspants I think the switch ports are dependent on the eth0 port being up first, maybe it has something to do with timing/ordering?

Share this post


Link to post
Share on other sites

hmm, that reminds me that I have a similar problem, when the system comes up, the wan port doesn't pull an address (ipv4 or ipv6) until I restart systemd-networkd.  I think chrisf is on to something about timing.

 

Share this post


Link to post
Share on other sites

That's a very good possibility, guys. I assume it's something a bit more complex than just renaming the files in the network directory to change the order?

 

I tried this:

garrettj@espressobin:~$ la /etc/systemd/network/

total 48
drwxr-xr-x 2 root root 4096 Feb 28 18:01 .
drwxr-xr-x 5 root root 4096 Feb 28 15:09 ..
-rw-r--r-- 1 root root   30 Feb  9 16:29 10-br0.netdev
-rw-r--r-- 1 root root   88 Feb 27 16:58 10-br0.network
-rw-r--r-- 1 root root   40 Feb  9 16:29 10-eth0.network
-rw-r--r-- 1 root root   40 Feb  9 16:29 10-lan0.network
-rw-r--r-- 1 root root   40 Feb  9 16:29 10-lan1.network
-rw-r--r-- 1 root root  118 Feb 27 16:18 20-bond1.netdev
-rw-r--r-- 1 root root   79 Feb 28 17:21 20-bond1.network
-rw-r--r-- 1 root root   42 Feb 27 16:25 20-enp0s0.network
-rw-r--r-- 1 root root   40 Feb 27 16:24 20-wan.network

And just for the heck of it, I also tried renaming 10-eth0.network to 01-eth0.network too, but neither seems to make a difference.

 

I assume it's something a bit lower-level than this, but I'm really not sure what to look at.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
2 2