Jump to content

antsu

Members
  • Posts

    27
  • Joined

  • Last visited

Posts posted by antsu

  1. This time it survived a little longer, but sure enough, today I woke up to my H64 unreachable on the 2.5G port again:

     

    [255036.090052] xhci-hcd xhci-hcd.0.auto: xHCI host not responding to stop endpoint command.
    [255036.090061] xhci-hcd xhci-hcd.0.auto: USBSTS:
    [255036.103643] xhci-hcd xhci-hcd.0.auto: xHCI host controller not responding, assume dead
    [255036.103681] xhci-hcd xhci-hcd.0.auto: HC died; cleaning up
    [255036.103734] r8152 4-1.4:1.0 eth1: Stop submitting intr, status -108
    [255036.103774] r8152 4-1.4:1.0 eth1: get_registers -110
    [255036.103822] r8152 4-1.4:1.0 eth1: Tx status -108
    [255036.103831] r8152 4-1.4:1.0 eth1: Tx status -108
    [255036.103839] r8152 4-1.4:1.0 eth1: Tx status -108
    [255036.103850] r8152 4-1.4:1.0 eth1: Tx status -108
    [255036.103868] usb 3-1: USB disconnect, device number 2
    [255036.109516] usb 4-1: USB disconnect, device number 2
    [255036.109526] usb 4-1.1: USB disconnect, device number 3
    [255036.131518] usb 4-1.4: USB disconnect, device number 4
    [255429.047792] zio pool=backups vdev=/dev/disk/by-id/usb-WD_Elements_XXXX_XXXXXXXXXXXXXXX-0:0-part1 error=5 type=1 offset=3186896842752 size=4096 flags=180880
    [255429.047897] zio pool=backups vdev=/dev/disk/by-id/usb-WD_Elements_XXXX_XXXXXXXXXXXXXXX-0:0-part1 error=5 type=1 offset=270336 size=8192 flags=b08c1
    [255429.047937] zio pool=backups vdev=/dev/disk/by-id/usb-WD_Elements_XXXX_XXXXXXXXXXXXXXX-0:0-part1 error=5 type=1 offset=14000475086848 size=8192 flags=b08c1
    [255429.047966] zio pool=backups vdev=/dev/disk/by-id/usb-WD_Elements_XXXX_XXXXXXXXXXXXXXX-0:0-part1 error=5 type=1 offset=14000475348992 size=8192 flags=b08c1
    [255429.048323] WARNING: Pool 'backups' has encountered an uncorrectable I/O failure and has been suspended.
    [255433.136397] WARNING: Pool 'backups' has encountered an uncorrectable I/O failure and has been suspended.

     

    I'm going back to using just the 1G port for now, since my H64 has been much more stable after the IO scheduler changes suggested by @ShadowDance in another thread, but happy to do more tests if anyone wants to try to figure out what's happening.

  2. I was trying to avoid opening a new instability topic, but this one seems different enough from the others to deserve it's own discussion.

    A few days ago I applied the 1G fix (pic attached) to the 2.5G port - which was not in use until then, since all my hardware is 1G max. I did this with the intention of using both ports: the 1G port goes to an isolated VLAN that serves VM images to my Proxmox hosts, and the 2.5G interface (running at 1G speed) serves files to the rest of my network.

    While the fix appears to work and the port is able to communicate at 1G speeds seemingly fine, after changing to this setup I'm experiencing frequent drops in the USB bus, which causes both the 2.5G port and my USB HDD (which has its own PSU) to disconnect, requiring a reboot to regain connectivity.

     

    When this happens, I get these messages in dmesg:

    [48657.415910] xhci-hcd xhci-hcd.0.auto: xHCI host not responding to stop endpoint command.
    [48657.415930] xhci-hcd xhci-hcd.0.auto: USBSTS:
    [48657.429522] xhci-hcd xhci-hcd.0.auto: xHCI host controller not responding, assume dead
    [48657.429604] xhci-hcd xhci-hcd.0.auto: HC died; cleaning up
    [48657.429952] r8152 4-1.4:1.0 eth1: Stop submitting intr, status -108
    [48657.430068] r8152 4-1.4:1.0 eth1: get_registers -110
    [48657.430193] r8152 4-1.4:1.0 eth1: Tx status -108
    [48657.430223] r8152 4-1.4:1.0 eth1: Tx status -108
    [48657.430241] r8152 4-1.4:1.0 eth1: Tx status -108
    [48657.430262] r8152 4-1.4:1.0 eth1: Tx status -108
    [48657.430327] usb 3-1: USB disconnect, device number 2
    [48657.431551] usb 4-1: USB disconnect, device number 2
    [48657.431572] usb 4-1.1: USB disconnect, device number 3
    [48657.467516] usb 4-1.4: USB disconnect, device number 4
    

     

    The problem seems to manifest more quickly if I push more data through that interface.

    I'm running a clean (new) install of Armbian 21.02.3, with OMV and ZFS. Full dmesg and boot log below.

     

    Spoiler

    [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
    [    0.000000] Linux version 5.10.21-rockchip64 (root@hirsute) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #21.02.3 SMP PREEMPT Mon Mar 8 01:05:08 UTC 2021
    [    0.000000] Machine model: Helios64
    [    0.000000] efi: UEFI not found.
    [    0.000000] cma: Reserved 128 MiB at 0x00000000ecc00000
    [    0.000000] NUMA: No NUMA configuration found
    [    0.000000] NUMA: Faking a node at [mem 0x0000000000200000-0x00000000f7ffffff]
    [    0.000000] NUMA: NODE_DATA [mem 0xf77d9100-0xf77dafff]
    [    0.000000] Zone ranges:
    [    0.000000]   DMA      [mem 0x0000000000200000-0x000000003fffffff]
    [    0.000000]   DMA32    [mem 0x0000000040000000-0x00000000f7ffffff]
    [    0.000000]   Normal   empty
    [    0.000000] Movable zone start for each node
    [    0.000000] Early memory node ranges
    [    0.000000]   node   0: [mem 0x0000000000200000-0x00000000f7ffffff]
    [    0.000000] Initmem setup node 0 [mem 0x0000000000200000-0x00000000f7ffffff]
    [    0.000000] On node 0 totalpages: 1015296
    [    0.000000]   DMA zone: 4088 pages used for memmap
    [    0.000000]   DMA zone: 0 pages reserved
    [    0.000000]   DMA zone: 261632 pages, LIFO batch:63
    [    0.000000]   DMA32 zone: 11776 pages used for memmap
    [    0.000000]   DMA32 zone: 753664 pages, LIFO batch:63
    [    0.000000] psci: probing for conduit method from DT.
    [    0.000000] psci: PSCIv1.0 detected in firmware.
    [    0.000000] psci: Using standard PSCI v0.2 function IDs
    [    0.000000] psci: MIGRATE_INFO_TYPE not supported.
    [    0.000000] psci: SMC Calling Convention v1.0
    [    0.000000] percpu: Embedded 32 pages/cpu s94040 r8192 d28840 u131072
    [    0.000000] pcpu-alloc: s94040 r8192 d28840 u131072 alloc=32*4096
    [    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5
    [    0.000000] Detected VIPT I-cache on CPU0
    [    0.000000] CPU features: detected: ARM erratum 845719
    [    0.000000] CPU features: detected: GIC system register CPU interface
    [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 999432
    [    0.000000] Policy zone: DMA32
    [    0.000000] Kernel command line: root=UUID=c559a861-be25-4b41-a161-d0c3d75c0089 rootwait rootfstype=ext4 console=ttyS2,1500000 console=tty1 consoleblank=0 loglevel=1 ubootpart=559a8d18-01 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u,0x1058:0x25a3:u   cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1
    [    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
    [    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
    [    0.000000] mem auto-init: stack:off, heap alloc:on, heap free:off
    [    0.000000] software IO TLB: mapped [mem 0x000000003bfff000-0x000000003ffff000] (64MB)
    [    0.000000] Memory: 3739360K/4061184K available (14848K kernel code, 2632K rwdata, 5972K rodata, 4352K init, 580K bss, 190752K reserved, 131072K cma-reserved)
    [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=6, Nodes=1
    [    0.000000] rcu: Preemptible hierarchical RCU implementation.
    [    0.000000] rcu:     RCU event tracing is enabled.
    [    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=6.
    [    0.000000]     Trampoline variant of Tasks RCU enabled.
    [    0.000000]     Rude variant of Tasks RCU enabled.
    [    0.000000]     Tracing variant of Tasks RCU enabled.
    [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
    [    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=6
    [    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
    [    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
    [    0.000000] GICv3: 256 SPIs implemented
    [    0.000000] GICv3: 0 Extended SPIs implemented
    [    0.000000] GICv3: Distributor has no Range Selector support
    [    0.000000] GICv3: 16 PPIs implemented
    [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000fef00000
    [    0.000000] ITS [mem 0xfee20000-0xfee3ffff]
    [    0.000000] ITS@0x00000000fee20000: allocated 65536 Devices @40100000 (flat, esz 8, psz 64K, shr 0)
    [    0.000000] ITS: using cache flushing for cmd queue
    [    0.000000] GICv3: using LPI property table @0x00000000400d0000
    [    0.000000] GIC: using cache flushing for LPI property table
    [    0.000000] GICv3: CPU0: using allocated LPI pending table @0x00000000400e0000
    [    0.000000] GICv3: GIC: PPI partition interrupt-partition-0[0] { /cpus/cpu@0[0] /cpus/cpu@1[1] /cpus/cpu@2[2] /cpus/cpu@3[3] }
    [    0.000000] GICv3: GIC: PPI partition interrupt-partition-1[1] { /cpus/cpu@100[4] /cpus/cpu@101[5] }
    [    0.000000] random: get_random_bytes called from start_kernel+0x698/0x850 with crng_init=0
    [    0.000000] arch_timer: cp15 timer(s) running at 24.00MHz (phys).
    [    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x588fe9dc0, max_idle_ns: 440795202592 ns
    [    0.000007] sched_clock: 56 bits at 24MHz, resolution 41ns, wraps every 4398046511097ns
    [    0.001519] Console: colour dummy device 80x25
    [    0.001587] printk: console [tty1] enabled
    [    0.001719] Calibrating delay loop (skipped), value calculated using timer frequency.. 48.00 BogoMIPS (lpj=96000)
    [    0.001741] pid_max: default: 32768 minimum: 301
    [    0.001908] LSM: Security Framework initializing
    [    0.001950] Yama: becoming mindful.
    [    0.002095] AppArmor: AppArmor initialized
    [    0.002198] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
    [    0.002231] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
    [    0.005942] rcu: Hierarchical SRCU implementation.
    [    0.007121] Platform MSI: interrupt-controller@fee20000 domain created
    [    0.007775] PCI/MSI: /interrupt-controller@fee00000/interrupt-controller@fee20000 domain created
    [    0.010316] EFI services will not be available.
    [    0.011034] smp: Bringing up secondary CPUs ...
    [    0.012211] Detected VIPT I-cache on CPU1
    [    0.012262] GICv3: CPU1: found redistributor 1 region 0:0x00000000fef20000
    [    0.012282] GICv3: CPU1: using allocated LPI pending table @0x00000000400f0000
    [    0.012360] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
    [    0.013787] Detected VIPT I-cache on CPU2
    [    0.013838] GICv3: CPU2: found redistributor 2 region 0:0x00000000fef40000
    [    0.013858] GICv3: CPU2: using allocated LPI pending table @0x0000000040180000
    [    0.013935] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
    [    0.015392] Detected VIPT I-cache on CPU3
    [    0.015442] GICv3: CPU3: found redistributor 3 region 0:0x00000000fef60000
    [    0.015463] GICv3: CPU3: using allocated LPI pending table @0x0000000040190000
    [    0.015540] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
    [    0.016971] CPU features: detected: Spectre-v2
    [    0.016979] CPU features: detected: Spectre-v4
    [    0.016984] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
    [    0.016992] Detected PIPT I-cache on CPU4
    [    0.017029] GICv3: CPU4: found redistributor 100 region 0:0x00000000fef80000
    [    0.017044] GICv3: CPU4: using allocated LPI pending table @0x00000000401a0000
    [    0.017093] CPU4: Booted secondary processor 0x0000000100 [0x410fd082]
    [    0.018365] Detected PIPT I-cache on CPU5
    [    0.018404] GICv3: CPU5: found redistributor 101 region 0:0x00000000fefa0000
    [    0.018419] GICv3: CPU5: using allocated LPI pending table @0x00000000401b0000
    [    0.018469] CPU5: Booted secondary processor 0x0000000101 [0x410fd082]
    [    0.018652] smp: Brought up 1 node, 6 CPUs
    [    0.018681] SMP: Total of 6 processors activated.
    [    0.018694] CPU features: detected: 32-bit EL0 Support
    [    0.018705] CPU features: detected: CRC32 instructions
    [    0.018716] CPU features: detected: 32-bit EL1 Support
    [    0.045491] CPU: All CPU(s) started at EL2
    [    0.045560] alternatives: patching kernel code
    [    0.049357] devtmpfs: initialized
    [    0.066016] Registered cp15_barrier emulation handler
    [    0.066040] Registered setend emulation handler
    [    0.066278] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
    [    0.066350] futex hash table entries: 2048 (order: 5, 131072 bytes, linear)
    [    0.069344] pinctrl core: initialized pinctrl subsystem
    [    0.070017] DMI not present or invalid.
    [    0.070627] NET: Registered protocol family 16
    [    0.074014] DMA: preallocated 2048 KiB GFP_KERNEL pool for atomic allocations
    [    0.075467] DMA: preallocated 2048 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
    [    0.076203] DMA: preallocated 2048 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
    [    0.076288] audit: initializing netlink subsys (disabled)
    [    0.076538] audit: type=2000 audit(0.076:1): state=initialized audit_enabled=0 res=1
    [    0.077839] thermal_sys: Registered thermal governor 'fair_share'
    [    0.077846] thermal_sys: Registered thermal governor 'bang_bang'
    [    0.077854] thermal_sys: Registered thermal governor 'step_wise'
    [    0.077861] thermal_sys: Registered thermal governor 'user_space'
    [    0.078347] cpuidle: using governor menu
    [    0.078682] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
    [    0.078907] ASID allocator initialised with 65536 entries
    [    0.080260] Serial: AMBA PL011 UART driver
    [    0.142602] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
    [    0.142616] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
    [    0.142626] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
    [    0.142634] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
    [    0.149240] cryptd: max_cpu_qlen set to 1000
    [    0.228343] raid6: neonx8   gen()  1668 MB/s
    [    0.296495] raid6: neonx8   xor()  1228 MB/s
    [    0.364654] raid6: neonx4   gen()  1682 MB/s
    [    0.432822] raid6: neonx4   xor()  1265 MB/s
    [    0.500988] raid6: neonx2   gen()  1506 MB/s
    [    0.569142] raid6: neonx2   xor()  1171 MB/s
    [    0.637313] raid6: neonx1   gen()  1178 MB/s
    [    0.705465] raid6: neonx1   xor()   933 MB/s
    [    0.773625] raid6: int64x8  gen()   995 MB/s
    [    0.841838] raid6: int64x8  xor()   576 MB/s
    [    0.909955] raid6: int64x4  gen()   986 MB/s
    [    0.978123] raid6: int64x4  xor()   591 MB/s
    [    1.046275] raid6: int64x2  gen()   927 MB/s
    [    1.114448] raid6: int64x2  xor()   523 MB/s
    [    1.182610] raid6: int64x1  gen()   729 MB/s
    [    1.250779] raid6: int64x1  xor()   390 MB/s
    [    1.250786] raid6: using algorithm neonx4 gen() 1682 MB/s
    [    1.250793] raid6: .... xor() 1265 MB/s, rmw enabled
    [    1.250801] raid6: using neon recovery algorithm
    [    1.252261] vcc12v_dcin_bkup: supplied by vcc12v_dcin
    [    1.252697] vcc12v_hdd: supplied by vcc12v_dcin_bkup
    [    1.266156] vcc5v0_perdev: supplied by vcc12v_dcin_bkup
    [    1.266646] pcie_power: supplied by vcc5v0_perdev
    [    1.266767] vcc5v0_usb: supplied by vcc5v0_perdev
    [    1.266858] vcc5v0_typec: supplied by vcc5v0_usb
    [    1.266921] vcc5v0_hdd: supplied by vcc12v_dcin_bkup
    [    1.267540] vcc5v0_sys: supplied by vcc12v_dcin_bkup
    [    1.268095] vcc3v3_sys_s3: supplied by vcc5v0_sys
    [    1.268153] vcc0v9_s3: supplied by vcc3v3_sys_s3
    [    1.268240] avdd_1v8_s0: supplied by vcc3v3_sys_s3
    [    1.268316] vcc3v0_sd: supplied by vcc3v3_sys_s3
    [    1.269331] usblan_power: supplied by vcc5v0_usb
    [    1.269920] iommu: Default domain type: Translated
    [    1.273379] vgaarb: loaded
    [    1.273909] SCSI subsystem initialized
    [    1.274155] libata version 3.00 loaded.
    [    1.274431] usbcore: registered new interface driver usbfs
    [    1.274479] usbcore: registered new interface driver hub
    [    1.274527] usbcore: registered new device driver usb
    [    1.275267] pps_core: LinuxPPS API ver. 1 registered
    [    1.275274] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
    [    1.275293] PTP clock support registered
    [    1.275622] EDAC MC: Ver: 3.0.0
    [    1.276855] NetLabel: Initializing
    [    1.276864] NetLabel:  domain hash size = 128
    [    1.276870] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
    [    1.276963] NetLabel:  unlabeled traffic allowed by default
    [    1.277465] clocksource: Switched to clocksource arch_sys_counter
    [    1.401375] VFS: Disk quotas dquot_6.6.0
    [    1.401479] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
    [    1.402370] AppArmor: AppArmor Filesystem Enabled
    [    1.411643] NET: Registered protocol family 2
    [    1.412630] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
    [    1.412878] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
    [    1.413416] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)
    [    1.414137] TCP: Hash tables configured (established 32768 bind 32768)
    [    1.414348] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
    [    1.414519] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
    [    1.414862] NET: Registered protocol family 1
    [    1.414896] NET: Registered protocol family 44
    [    1.414913] PCI: CLS 0 bytes, default 64
    [    1.415158] Trying to unpack rootfs image as initramfs...
    [    2.539985] Freeing initrd memory: 15388K
    [    2.541309] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
    [    2.541821] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
    [    2.542600] kvm [1]: IPA Size Limit: 40 bits
    [    2.543758] kvm [1]: vgic-v2@fff20000
    [    2.543796] kvm [1]: GIC system register CPU interface enabled
    [    2.544057] kvm [1]: vgic interrupt IRQ18
    [    2.544362] kvm [1]: Hyp mode initialized successfully
    [    2.555092] Initialise system trusted keyrings
    [    2.555141] Key type blacklist registered
    [    2.555378] workingset: timestamp_bits=44 max_order=20 bucket_order=0
    [    2.561906] zbud: loaded
    [    2.563702] squashfs: version 4.0 (2009/01/31) Phillip Lougher
    [    2.564554] fuse: init (API version 7.32)
    [    2.566009] integrity: Platform Keyring initialized
    [    2.630363] xor: measuring software checksum speed
    [    2.634282]    8regs           :  2531 MB/sec
    [    2.637677]    32regs          :  2917 MB/sec
    [    2.641796]    arm64_neon      :  2398 MB/sec
    [    2.641804] xor: using function: 32regs (2917 MB/sec)
    [    2.641820] Key type asymmetric registered
    [    2.641827] Asymmetric key parser 'x509' registered
    [    2.641881] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
    [    2.642067] io scheduler mq-deadline registered
    [    2.642076] io scheduler kyber registered
    [    2.642214] io scheduler bfq registered
    [    2.661560] rockchip-pcie f8000000.pcie: host bridge /pcie@f8000000 ranges:
    [    2.661606] rockchip-pcie f8000000.pcie:      MEM 0x00fa000000..0x00fbdfffff -> 0x00fa000000
    [    2.661628] rockchip-pcie f8000000.pcie:       IO 0x00fbe00000..0x00fbefffff -> 0x00fbe00000
    [    2.671607] dma-pl330 ff6d0000.dma-controller: Loaded driver for PL330 DMAC-241330
    [    2.671620] dma-pl330 ff6d0000.dma-controller:     DBUFF-32x8bytes Num_Chans-6 Num_Peri-12 Num_Events-12
    [    2.673092] dma-pl330 ff6e0000.dma-controller: Loaded driver for PL330 DMAC-241330
    [    2.673105] dma-pl330 ff6e0000.dma-controller:     DBUFF-128x8bytes Num_Chans-8 Num_Peri-20 Num_Events-16
    [    2.678834] vdd_log: supplied by regulator-dummy
    [    2.680278] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled
    [    2.683575] ff180000.serial: ttyS0 at MMIO 0xff180000 (irq = 44, base_baud = 1500000) is a 16550A
    [    2.684637] ff1a0000.serial: ttyS2 at MMIO 0xff1a0000 (irq = 45, base_baud = 1500000) is a 16550A
    [    2.684926] printk: console [ttyS2] enabled
    [    2.686458] Serial: AMBA driver
    [    2.688039] cacheinfo: Unable to detect cache hierarchy for CPU 0
    [    2.700101] loop: module loaded
    [    2.708865] libphy: Fixed MDIO Bus: probed
    [    2.709643] tun: Universal TUN/TAP device driver, 1.6
    [    2.710803] thunder_xcv, ver 1.0
    [    2.710849] thunder_bgx, ver 1.0
    [    2.710899] nicpf, ver 1.0
    [    2.712093] e1000e: Intel(R) PRO/1000 Network Driver
    [    2.712100] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
    [    2.712162] igb: Intel(R) Gigabit Ethernet Network Driver
    [    2.712169] igb: Copyright (c) 2007-2014 Intel Corporation.
    [    2.712208] igbvf: Intel(R) Gigabit Virtual Function Network Driver
    [    2.712215] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
    [    2.712520] sky2: driver version 1.30
    [    2.713265] VFIO - User Level meta-driver version: 0.3
    [    2.732394] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
    [    2.732431] ehci-pci: EHCI PCI platform driver
    [    2.732475] ehci-platform: EHCI generic platform driver
    [    2.734894] ehci-platform fe380000.usb: EHCI Host Controller
    [    2.734922] ehci-platform fe380000.usb: new USB bus registered, assigned bus number 1
    [    2.736074] ehci-platform fe380000.usb: irq 38, io mem 0xfe380000
    [    2.749488] ehci-platform fe380000.usb: USB 2.0 started, EHCI 1.00
    [    2.749770] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
    [    2.749780] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [    2.749790] usb usb1: Product: EHCI Host Controller
    [    2.749798] usb usb1: Manufacturer: Linux 5.10.21-rockchip64 ehci_hcd
    [    2.749806] usb usb1: SerialNumber: fe380000.usb
    [    2.750361] hub 1-0:1.0: USB hub found
    [    2.750403] hub 1-0:1.0: 1 port detected
    [    2.751220] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
    [    2.751244] ohci-pci: OHCI PCI platform driver
    [    2.751300] ohci-platform: OHCI generic platform driver
    [    2.751630] ohci-platform fe3a0000.usb: Generic Platform OHCI controller
    [    2.751652] ohci-platform fe3a0000.usb: new USB bus registered, assigned bus number 2
    [    2.751786] ohci-platform fe3a0000.usb: irq 39, io mem 0xfe3a0000
    [    2.813718] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 5.10
    [    2.813730] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [    2.813738] usb usb2: Product: Generic Platform OHCI controller
    [    2.813746] usb usb2: Manufacturer: Linux 5.10.21-rockchip64 ohci_hcd
    [    2.813755] usb usb2: SerialNumber: fe3a0000.usb
    [    2.814265] hub 2-0:1.0: USB hub found
    [    2.814308] hub 2-0:1.0: 1 port detected
    [    2.815951] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
    [    2.815976] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 3
    [    2.816159] xhci-hcd xhci-hcd.0.auto: hcc params 0x0220fe64 hci version 0x110 quirks 0x0000010002010010
    [    2.816227] xhci-hcd xhci-hcd.0.auto: irq 76, io mem 0xfe900000
    [    2.816656] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
    [    2.816667] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [    2.816675] usb usb3: Product: xHCI Host Controller
    [    2.816684] usb usb3: Manufacturer: Linux 5.10.21-rockchip64 xhci-hcd
    [    2.816692] usb usb3: SerialNumber: xhci-hcd.0.auto
    [    2.817218] hub 3-0:1.0: USB hub found
    [    2.817267] hub 3-0:1.0: 1 port detected
    [    2.817643] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
    [    2.817660] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 4
    [    2.817676] xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0 SuperSpeed
    [    2.817762] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
    [    2.817908] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.10
    [    2.817918] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [    2.817927] usb usb4: Product: xHCI Host Controller
    [    2.817935] usb usb4: Manufacturer: Linux 5.10.21-rockchip64 xhci-hcd
    [    2.817943] usb usb4: SerialNumber: xhci-hcd.0.auto
    [    2.818461] hub 4-0:1.0: USB hub found
    [    2.818511] hub 4-0:1.0: 1 port detected
    [    2.818998] usbcore: registered new interface driver usb-storage
    [    2.821778] mousedev: PS/2 mouse device common for all mice
    [    2.823240] i2c /dev entries driver
    [    2.827379] pca953x 2-0020: using no AI
    [    2.837324] rk808 0-001b: chip id: 0x0
    [    2.841238] rk808-regulator rk808-regulator: there is no dvs0 gpio
    [    2.841270] rk808-regulator rk808-regulator: there is no dvs1 gpio
    [    2.841281] rk808-regulator rk808-regulator: max buck steps per change: 4
    [    2.842434] vdd_center: supplied by vcc5v0_sys
    [    2.843089] vdd_cpu_l: supplied by vcc5v0_sys
    [    2.843446] vcc_ddr_s3: supplied by vcc5v0_sys
    [    2.844114] vcc1v8_sys_s3: supplied by vcc5v0_sys
    [    2.844770] vcc1v8_dvp: supplied by vcc5v0_sys
    [    2.845283] vcc3v0_touch: supplied by vcc5v0_sys
    [    2.846634] vcc1v8_s3: supplied by vcc5v0_sys
    [    2.846922] vcc_sdio_s0: Bringing 3300000uV into 3000000-3000000uV
    [    2.848081] vcc_sdio_s0: supplied by vcc5v0_sys
    [    2.848604] vcca3v0_codec: supplied by vcc5v0_sys
    [    2.849805] vcc1v5_s3: supplied by vcc5v0_sys
    [    2.850312] vcca1v8_codec: supplied by vcc5v0_sys
    [    2.851521] vcc3v0_s3: supplied by vcc5v0_sys
    [    2.851675] vcc3v3_sys_s0: failed to get the current voltage: -EPROBE_DEFER
    [    2.851698] vcc3v3_sys_s0: supplied by vcc3v3_sys_s3
    [    2.852274] vcc3v3_s0: supplied by vcc3v3_sys_s3
    [    2.856656] rk808-rtc rk808-rtc: registered as rtc0
    [    2.857231] rk808-rtc rk808-rtc: setting system clock to 2021-03-26T10:48:14 UTC (1616755694)
    [    2.860390] fan53555-regulator 0-0040: FAN53555 Option[8] Rev[1] Detected!
    [    2.861728] vcc1v8_sys_s0: supplied by vcc1v8_sys_s3
    [    2.861826] avdd_0v9_s0: supplied by vcc1v8_sys_s3
    [    2.861932] vdd_cpu_b: supplied by vcc5v0_sys
    [    2.865041] fan53555-regulator 0-0041: FAN53555 Option[8] Rev[1] Detected!
    [    2.866323] vdd_gpu: supplied by vcc5v0_sys
    [    2.874108] dw_wdt ff848000.watchdog: No valid TOPs array specified
    [    2.876072] sdhci: Secure Digital Host Controller Interface driver
    [    2.876081] sdhci: Copyright(c) Pierre Ossman
    [    2.876117] Synopsys Designware Multimedia Card Interface Driver
    [    2.877574] sdhci-pltfm: SDHCI platform and OF driver helper
    [    2.878011] dwmmc_rockchip fe320000.mmc: IDMAC supports 32-bit address mode.
    [    2.878044] dwmmc_rockchip fe320000.mmc: Using internal DMA controller.
    [    2.878062] dwmmc_rockchip fe320000.mmc: Version ID is 270a
    [    2.878140] dwmmc_rockchip fe320000.mmc: DW MMC controller at irq 36,32 bit host data width,256 deep fifo
    [    2.878705] dwmmc_rockchip fe320000.mmc: Got CD GPIO
    [    2.880843] mmc2: CQHCI version 5.10
    [    2.882320] ledtrig-cpu: registered to indicate activity on CPUs
    [    2.883517] hid: raw HID events driver (C) Jiri Kosina
    [    2.883652] usbcore: registered new interface driver usbhid
    [    2.883660] usbhid: USB HID core driver
    [    2.889665] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
    [    2.889934] drop_monitor: Initializing network drop monitor service
    [    2.890763] NET: Registered protocol family 10
    [    2.903775] random: fast init done
    [    2.907276] mmc2: SDHCI controller on fe330000.sdhci [fe330000.sdhci] using ADMA
    [    2.944344] Segment Routing with IPv6
    [    2.944462] NET: Registered protocol family 17
    [    2.944634] 8021q: 802.1Q VLAN Support v1.8
    [    2.944700] Key type dns_resolver registered
    [    2.945268] registered taskstats version 1
    [    2.945286] Loading compiled-in X.509 certificates
    [    2.948881] Loaded X.509 cert 'Build time autogenerated kernel key: 98b5fbd0004968eadd44123544af94c22f57c8e4'
    [    2.955599] zswap: loaded using pool zstd/z3fold
    [    2.956051] Key type ._fscrypt registered
    [    2.956059] Key type .fscrypt registered
    [    2.956066] Key type fscrypt-provisioning registered
    [    2.959366] Btrfs loaded, crc32c=crc32c-generic
    [    2.978782] mmc2: Command Queue Engine enabled
    [    2.978820] mmc2: new HS400 Enhanced strobe MMC card at address 0001
    [    2.979918] mmcblk2: mmc2:0001 AJTD4R 14.6 GiB
    [    2.980233] mmcblk2boot0: mmc2:0001 AJTD4R partition 1 4.00 MiB
    [    2.980560] mmcblk2boot1: mmc2:0001 AJTD4R partition 2 4.00 MiB
    [    2.980823] mmcblk2rpmb: mmc2:0001 AJTD4R partition 3 4.00 MiB, chardev (243:0)
    [    2.982381]  mmcblk2: p1
    [    2.982516] mmc_host mmc1: Bus speed (slot 0) = 148500000Hz (slot req 150000000Hz, actual 148500000HZ div = 0)
    [    2.989643] Key type encrypted registered
    [    2.989665] AppArmor: AppArmor sha1 policy hashing enabled
    [    3.040961] rockchip-pcie f8000000.pcie: host bridge /pcie@f8000000 ranges:
    [    3.041009] rockchip-pcie f8000000.pcie:      MEM 0x00fa000000..0x00fbdfffff -> 0x00fa000000
    [    3.041032] rockchip-pcie f8000000.pcie:       IO 0x00fbe00000..0x00fbefffff -> 0x00fbe00000
    [    3.098770] rockchip-pcie f8000000.pcie: PCI host bridge to bus 0000:00
    [    3.098785] pci_bus 0000:00: root bus resource [bus 00-1f]
    [    3.098796] pci_bus 0000:00: root bus resource [mem 0xfa000000-0xfbdfffff]
    [    3.098808] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0xfbe00000-0xfbefffff])
    [    3.098863] pci 0000:00:00.0: [1d87:0100] type 01 class 0x060400
    [    3.099017] pci 0000:00:00.0: supports D1
    [    3.099027] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
    [    3.104908] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
    [    3.105163] pci 0000:01:00.0: [197b:0585] type 00 class 0x010601
    [    3.105234] pci 0000:01:00.0: reg 0x10: initial BAR value 0x00000000 invalid
    [    3.105244] pci 0000:01:00.0: reg 0x10: [io  size 0x0080]
    [    3.105280] pci 0000:01:00.0: reg 0x14: initial BAR value 0x00000000 invalid
    [    3.105289] pci 0000:01:00.0: reg 0x14: [io  size 0x0080]
    [    3.105324] pci 0000:01:00.0: reg 0x18: initial BAR value 0x00000000 invalid
    [    3.105332] pci 0000:01:00.0: reg 0x18: [io  size 0x0080]
    [    3.105368] pci 0000:01:00.0: reg 0x1c: initial BAR value 0x00000000 invalid
    [    3.105376] pci 0000:01:00.0: reg 0x1c: [io  size 0x0080]
    [    3.105411] pci 0000:01:00.0: reg 0x20: initial BAR value 0x00000000 invalid
    [    3.105419] pci 0000:01:00.0: reg 0x20: [io  size 0x0080]
    [    3.105473] pci 0000:01:00.0: reg 0x24: [mem 0x00000000-0x00001fff]
    [    3.105510] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
    [    3.105549] pci 0000:01:00.0: Max Payload Size set to 256 (was 128, max 512)
    [    3.105903] pci 0000:01:00.0: PME# supported from D3hot
    [    3.106149] pci 0000:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of 15.752 Gb/s with 8.0 GT/s PCIe x2 link)
    [    3.111786] pci_bus 0000:01: busn_res: [bus 01-1f] end is updated to 01
    [    3.111816] pci 0000:00:00.0: BAR 14: assigned [mem 0xfa000000-0xfa0fffff]
    [    3.111841] pci 0000:01:00.0: BAR 6: assigned [mem 0xfa000000-0xfa00ffff pref]
    [    3.111852] pci 0000:01:00.0: BAR 5: assigned [mem 0xfa010000-0xfa011fff]
    [    3.111873] pci 0000:01:00.0: BAR 0: no space for [io  size 0x0080]
    [    3.111881] pci 0000:01:00.0: BAR 0: failed to assign [io  size 0x0080]
    [    3.111890] pci 0000:01:00.0: BAR 1: no space for [io  size 0x0080]
    [    3.111898] pci 0000:01:00.0: BAR 1: failed to assign [io  size 0x0080]
    [    3.111907] pci 0000:01:00.0: BAR 2: no space for [io  size 0x0080]
    [    3.111915] pci 0000:01:00.0: BAR 2: failed to assign [io  size 0x0080]
    [    3.111923] pci 0000:01:00.0: BAR 3: no space for [io  size 0x0080]
    [    3.111931] pci 0000:01:00.0: BAR 3: failed to assign [io  size 0x0080]
    [    3.111939] pci 0000:01:00.0: BAR 4: no space for [io  size 0x0080]
    [    3.111947] pci 0000:01:00.0: BAR 4: failed to assign [io  size 0x0080]
    [    3.111961] pci 0000:00:00.0: PCI bridge to [bus 01]
    [    3.111974] pci 0000:00:00.0:   bridge window [mem 0xfa000000-0xfa0fffff]
    [    3.112212] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
    [    3.112529] pcieport 0000:00:00.0: PME: Signaling with IRQ 90
    [    3.233493] usb 3-1: new high-speed USB device number 2 using xhci-hcd
    [    3.596245] mmc_host mmc1: Timeout sending command (cmd 0x200000 arg 0x0 status 0x80200000)
    [    4.096333] dwmmc_rockchip fe320000.mmc: Busy; trying anyway
    [    4.096472] pcieport 0000:00:00.0: AER: enabled with IRQ 90
    [    4.096837] ahci 0000:01:00.0: version 3.0
    [    4.096863] ahci 0000:01:00.0: enabling device (0000 -> 0002)
    [    4.097513] ahci 0000:01:00.0: SSS flag set, parallel bus scan disabled
    [    4.097614] ahci 0000:01:00.0: AHCI 0001.0301 32 slots 5 ports 6 Gbps 0x1f impl SATA mode
    [    4.097628] ahci 0000:01:00.0: flags: 64bit ncq sntf stag pm led clo pmp fbs pio slum part ccc apst boh
    [    4.101263] scsi host0: ahci
    [    4.102033] scsi host1: ahci
    [    4.102673] scsi host2: ahci
    [    4.103301] scsi host3: ahci
    [    4.103934] scsi host4: ahci
    [    4.104159] ata1: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010100 irq 91
    [    4.104171] ata2: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010180 irq 92
    [    4.104181] ata3: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010200 irq 93
    [    4.104191] ata4: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010280 irq 94
    [    4.104200] ata5: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010300 irq 95
    [    4.112256] input: gpio-keys as /devices/platform/gpio-keys/input/input0
    [    4.113049] of_cfs_init
    [    4.113116] of_cfs_init: OK
    [    4.577498] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    [    4.578327] ata1.00: supports DRM functions and may not be fully accessible
    [    4.578396] ata1.00: ATA-10: CT500MX500SSD1, M3CR023, max UDMA/133
    [    4.578410] ata1.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 32), AA
    [    4.579194] ata1.00: supports DRM functions and may not be fully accessible
    [    4.579850] ata1.00: configured for UDMA/133
    [    4.580481] scsi 0:0:0:0: Direct-Access     ATA      CT500MX500SSD1   023  PQ: 0 ANSI: 5
    [    4.582610] sd 0:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/466 GiB)
    [    4.582628] sd 0:0:0:0: [sda] 4096-byte physical blocks
    [    4.582712] sd 0:0:0:0: [sda] Write Protect is off
    [    4.582728] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
    [    4.582866] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    [    4.721528] mmc_host mmc1: Timeout sending command (cmd 0x200000 arg 0x0 status 0x80200000)
    [    5.221614] dwmmc_rockchip fe320000.mmc: Busy; trying anyway
    [    5.223451]  sda: sda1 sda2 sda3
    [    5.223526] usb 3-1: New USB device found, idVendor=2109, idProduct=2815, bcdDevice= 7.04
    [    5.223538] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
    [    5.223547] usb 3-1: Product: USB2.0 Hub             
    [    5.223555] usb 3-1: Manufacturer: VIA Labs, Inc.         
    [    5.226961] sd 0:0:0:0: [sda] Attached SCSI disk
    [    5.282122] hub 3-1:1.0: USB hub found
    [    5.282254] hub 3-1:1.0: 4 ports detected
    [    5.376836] usb 4-1: new SuperSpeed Gen 1 USB device number 2 using xhci-hcd
    [    5.489530] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    [    5.496988] usb 4-1: New USB device found, idVendor=2109, idProduct=0815, bcdDevice= 7.04
    [    5.497000] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
    [    5.497010] usb 4-1: Product: USB3.0 Hub             
    [    5.497018] usb 4-1: Manufacturer: VIA Labs, Inc.         
    [    5.499437] ata2.00: ATA-10: ST4000VN008-2DR166, SC60, max UDMA/133
    [    5.499452] ata2.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 32), AA
    [    5.501000] ata2.00: configured for UDMA/133
    [    5.501677] scsi 1:0:0:0: Direct-Access     ATA      ST4000VN008-2DR1 SC60 PQ: 0 ANSI: 5
    [    5.503047] sd 1:0:0:0: [sdb] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
    [    5.503064] sd 1:0:0:0: [sdb] 4096-byte physical blocks
    [    5.503138] sd 1:0:0:0: [sdb] Write Protect is off
    [    5.503154] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
    [    5.503274] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    [    5.522073] hub 4-1:1.0: USB hub found
    [    5.522242] hub 4-1:1.0: 4 ports detected
    [    5.562407]  sdb: sdb1 sdb9
    [    5.565341] sd 1:0:0:0: [sdb] Attached SCSI disk
    [    5.619654] dwmmc_rockchip fe320000.mmc: Successfully tuned phase to 299
    [    5.619682] mmc1: new ultra high speed SDR104 SDHC card at address aaaa
    [    5.620732] mmcblk1: mmc1:aaaa SC32G 29.7 GiB
    [    5.626476]  mmcblk1: p1
    [    5.977500] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    [    5.993513] ata3.00: ATA-10: ST4000VN008-2DR166, SC60, max UDMA/133
    [    5.993532] ata3.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 32), AA
    [    5.995060] ata3.00: configured for UDMA/133
    [    5.995652] scsi 2:0:0:0: Direct-Access     ATA      ST4000VN008-2DR1 SC60 PQ: 0 ANSI: 5
    [    5.997033] sd 2:0:0:0: [sdc] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
    [    5.997050] sd 2:0:0:0: [sdc] 4096-byte physical blocks
    [    5.997133] sd 2:0:0:0: [sdc] Write Protect is off
    [    5.997149] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
    [    5.997289] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    [    6.054919]  sdc: sdc1 sdc9
    [    6.057868] sd 2:0:0:0: [sdc] Attached SCSI disk
    [    6.169562] usb 4-1.1: new SuperSpeed Gen 1 USB device number 3 using xhci-hcd
    [    6.190714] usb 4-1.1: New USB device found, idVendor=1058, idProduct=25a3, bcdDevice=10.30
    [    6.190726] usb 4-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
    [    6.190734] usb 4-1.1: Product: Elements 25A3
    [    6.190743] usb 4-1.1: Manufacturer: Western Digital
    [    6.190751] usb 4-1.1: SerialNumber: 395248583834474C
    [    6.191769] usb-storage 4-1.1:1.0: USB Mass Storage device detected
    [    6.192240] usb-storage 4-1.1:1.0: Quirks match for vid 1058 pid 25a3: 800000
    [    6.192393] scsi host5: usb-storage 4-1.1:1.0
    [    6.273567] usb 4-1.4: new SuperSpeed Gen 1 USB device number 4 using xhci-hcd
    [    6.294570] usb 4-1.4: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=30.00
    [    6.294581] usb 4-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
    [    6.294590] usb 4-1.4: Product: USB 10/100/1G/2.5G LAN
    [    6.294599] usb 4-1.4: Manufacturer: Realtek
    [    6.294607] usb 4-1.4: SerialNumber: 000000001
    [    6.473500] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    [    6.486448] ata4.00: ATA-10: ST4000VN008-2DR166, SC60, max UDMA/133
    [    6.486459] ata4.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 32), AA
    [    6.487945] ata4.00: configured for UDMA/133
    [    6.488591] scsi 3:0:0:0: Direct-Access     ATA      ST4000VN008-2DR1 SC60 PQ: 0 ANSI: 5
    [    6.489985] sd 3:0:0:0: [sdd] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
    [    6.490002] sd 3:0:0:0: [sdd] 4096-byte physical blocks
    [    6.490077] sd 3:0:0:0: [sdd] Write Protect is off
    [    6.490093] sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00
    [    6.490215] sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    [    6.552189]  sdd: sdd1 sdd9
    [    6.555154] sd 3:0:0:0: [sdd] Attached SCSI disk
    [    6.965495] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    [    6.975169] ata5.00: ATA-10: ST4000VN008-2DR166, SC60, max UDMA/133
    [    6.975179] ata5.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 32), AA
    [    6.976676] ata5.00: configured for UDMA/133
    [    6.977090] scsi 4:0:0:0: Direct-Access     ATA      ST4000VN008-2DR1 SC60 PQ: 0 ANSI: 5
    [    6.978329] sd 4:0:0:0: [sde] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
    [    6.978350] sd 4:0:0:0: [sde] 4096-byte physical blocks
    [    6.978425] sd 4:0:0:0: [sde] Write Protect is off
    [    6.978441] sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00
    [    6.978564] sd 4:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
    [    7.035234]  sde: sde1 sde9
    [    7.038236] sd 4:0:0:0: [sde] Attached SCSI disk
    [    7.044053] Freeing unused kernel memory: 4352K
    [    7.053665] Run /init as init process
    [    7.053674]   with arguments:
    [    7.053682]     /init
    [    7.053688]   with environment:
    [    7.053696]     HOME=/
    [    7.053702]     TERM=linux
    [    7.053708]     ubootpart=559a8d18-01
    [    7.053715]     cgroup_enable=memory
    [    7.053721]     cgroup_memory=1
    [    7.202315] scsi 5:0:0:0: Direct-Access     WD       Elements 25A3    1030 PQ: 0 ANSI: 6
    [    7.204780] sd 5:0:0:0: [sdf] Spinning up disk...
    [    7.705053] input: adc-keys as /devices/platform/adc-keys/input/input1
    [    7.873428] rk_gmac-dwmac fe300000.ethernet: IRQ eth_wake_irq not found
    [    7.873536] rk_gmac-dwmac fe300000.ethernet: IRQ eth_lpi not found
    [    7.873908] rk_gmac-dwmac fe300000.ethernet: PTP uses main clock
    [    7.878462] rk_gmac-dwmac fe300000.ethernet: clock input or output? (input).
    [    7.878489] rk_gmac-dwmac fe300000.ethernet: TX delay(0x28).
    [    7.878504] rk_gmac-dwmac fe300000.ethernet: RX delay(0x20).
    [    7.878530] rk_gmac-dwmac fe300000.ethernet: integrated PHY? (no).
    [    7.878653] rk_gmac-dwmac fe300000.ethernet: cannot get clock clk_mac_speed
    [    7.878664] rk_gmac-dwmac fe300000.ethernet: clock input from PHY
    [    7.887161] rk_gmac-dwmac fe300000.ethernet: init for RGMII
    [    7.899327] rk_gmac-dwmac fe300000.ethernet: User ID: 0x10, Synopsys ID: 0x35
    [    7.899346] rk_gmac-dwmac fe300000.ethernet:     DWMAC1000
    [    7.899357] rk_gmac-dwmac fe300000.ethernet: DMA HW capability register supported
    [    7.899365] rk_gmac-dwmac fe300000.ethernet: RX Checksum Offload Engine supported
    [    7.899373] rk_gmac-dwmac fe300000.ethernet: COE Type 2
    [    7.899381] rk_gmac-dwmac fe300000.ethernet: TX Checksum insertion supported
    [    7.899389] rk_gmac-dwmac fe300000.ethernet: Wake-Up On Lan supported
    [    7.899568] rk_gmac-dwmac fe300000.ethernet: Normal descriptors
    [    7.899578] rk_gmac-dwmac fe300000.ethernet: Ring mode enabled
    [    7.899586] rk_gmac-dwmac fe300000.ethernet: Enable RX Mitigation via HW Watchdog Timer
    [    8.038399] usbcore: registered new interface driver uas
    [    8.039728] libphy: stmmac: probed
    [    8.039753] RTL8211F Gigabit Ethernet stmmac-0:00: attached PHY driver [RTL8211F Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:00, irq=POLL)
    [    8.039768] RTL8211F Gigabit Ethernet stmmac-0:01: attached PHY driver [RTL8211F Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:01, irq=POLL)
    [    8.094834] random: crng init done
    [    8.225517] .
    [    8.480919] async_tx: api initialized (async)
    [    9.182852] EXT4-fs (mmcblk1p1): mounted filesystem with writeback data mode. Opts: (null)
    [    9.253504] .
    [    9.813223] systemd[1]: Inserted module 'autofs4'
    [    9.880774] systemd[1]: systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
    [    9.881819] systemd[1]: Detected architecture arm64.
    [    9.915138] systemd[1]: Set hostname to <derfflinger.lan>.
    [   10.090719] dw-apb-uart ff1a0000.serial: forbid DMA for kernel console
    [   10.273494] .
    [   10.624569] systemd[1]: /lib/systemd/system/rpc-statd.service:13: PIDFile= references path below legacy directory /var/run/, updating /var/run/rpc.statd.pid → /run/rpc.statd.pid; please update the unit file accordingly.
    [   10.733822] systemd[1]: /lib/systemd/system/smbd.service:9: PIDFile= references path below legacy directory /var/run/, updating /var/run/samba/smbd.pid → /run/samba/smbd.pid; please update the unit file accordingly.
    [   10.751120] systemd[1]: /lib/systemd/system/nmbd.service:9: PIDFile= references path below legacy directory /var/run/, updating /var/run/samba/nmbd.pid → /run/samba/nmbd.pid; please update the unit file accordingly.
    [   10.778540] systemd[1]: /lib/systemd/system/fancontrol.service:9: PIDFile= references path below legacy directory /var/run/, updating /var/run/fancontrol.pid → /run/fancontrol.pid; please update the unit file accordingly.
    [   10.912401] systemd[1]: zfs-share.service: Found ordering cycle on nfs-server.service/start
    [   10.912428] systemd[1]: zfs-share.service: Found dependency on zfs.target/start
    [   10.912447] systemd[1]: zfs-share.service: Found dependency on zfs-share.service/start
    [   11.026712] lm75 2-004c: supply vs not found, using dummy regulator
    [   11.029802] lm75 2-004c: hwmon2: sensor 'lm75'
    [   11.061059] spl: loading out-of-tree module taints kernel.
    [   11.061814] spl: module verification failed: signature and/or required key missing - tainting kernel
    [   11.072348] EXT4-fs (mmcblk1p1): re-mounted. Opts: commit=600,errors=remount-ro
    [   11.104083] icp: module license 'CDDL' taints kernel.
    [   11.104100] Disabling lock debugging due to kernel taint
    [   11.174485] RPC: Registered named UNIX socket transport module.
    [   11.174503] RPC: Registered udp transport module.
    [   11.174511] RPC: Registered tcp transport module.
    [   11.174519] RPC: Registered tcp NFSv4.1 backchannel transport module.
    [   11.297546] .
    [   11.356362] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
    [   11.665855] ZFS: Loaded module v2.0.3-1~bpo10+1, ZFS pool version 5000, ZFS filesystem version 5
    [   11.957812] rk_gmac-dwmac fe300000.ethernet eth0: PHY [stmmac-0:00] driver [RTL8211F Gigabit Ethernet] (irq=POLL)
    [   11.966232] rk_gmac-dwmac fe300000.ethernet eth0: No Safety Features support found
    [   11.966265] rk_gmac-dwmac fe300000.ethernet eth0: PTP not supported by HW
    [   11.967224] rk_gmac-dwmac fe300000.ethernet eth0: configuring for phy/rgmii link mode
    [   12.005891] input: beeper as /devices/platform/beeper/input/input2
    [   12.106905] sd 0:0:0:0: Attached scsi generic sg0 type 0
    [   12.115018] sd 1:0:0:0: Attached scsi generic sg1 type 0
    [   12.115698] sd 2:0:0:0: Attached scsi generic sg2 type 0
    [   12.116898] sd 3:0:0:0: Attached scsi generic sg3 type 0
    [   12.117156] sd 4:0:0:0: Attached scsi generic sg4 type 0
    [   12.117312] sd 5:0:0:0: Attached scsi generic sg5 type 0
    [   12.172139] rockchip-vop ff8f0000.vop: Adding to iommu group 2
    [   12.179954] rockchip-vop ff900000.vop: Adding to iommu group 3
    [   12.220271] mc: Linux media interface: v0.10
    [   12.240501] videodev: Linux video capture interface: v2.00
    [   12.256911] OF: graph: no port node found in /i2c@ff3d0000/typec-portc@22
    [   12.267795] rockchip-drm display-subsystem: bound ff8f0000.vop (ops vop_component_ops [rockchipdrm])
    [   12.267894] [drm] unsupported AFBC format[3231564e]
    [   12.269420] rockchip-drm display-subsystem: bound ff900000.vop (ops vop_component_ops [rockchipdrm])
    [   12.269599] rockchip-drm display-subsystem: bound fec00000.dp (ops cdn_dp_component_ops [rockchipdrm])
    [   12.273268] rockchip-drm display-subsystem: [drm] Cannot find any crtc or sizes
    [   12.274193] [drm] Initialized rockchip 1.0.0 20140818 for display-subsystem on minor 0
    [   12.276593] cdn-dp fec00000.dp: [drm:cdn_dp_pd_event_work [rockchipdrm]] Not connected. Disabling cdn
    [   12.279359] OF: graph: no port node found in /syscon@ff770000/usb2-phy@e450/otg-port
    [   12.285183] xhci-hcd xhci-hcd.3.auto: xHCI Host Controller
    [   12.285202] xhci-hcd xhci-hcd.3.auto: new USB bus registered, assigned bus number 5
    [   12.285312] xhci-hcd xhci-hcd.3.auto: hcc params 0x0220fe64 hci version 0x110 quirks 0x0000010002010010
    [   12.285327] xhci-hcd xhci-hcd.3.auto: irq 75, io mem 0xfe800000
    [   12.285568] usb usb5: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
    [   12.285573] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [   12.285577] usb usb5: Product: xHCI Host Controller
    [   12.285580] usb usb5: Manufacturer: Linux 5.10.21-rockchip64 xhci-hcd
    [   12.285583] usb usb5: SerialNumber: xhci-hcd.3.auto
    [   12.285931] hub 5-0:1.0: USB hub found
    [   12.285954] hub 5-0:1.0: 1 port detected
    [   12.286137] xhci-hcd xhci-hcd.3.auto: xHCI Host Controller
    [   12.286147] xhci-hcd xhci-hcd.3.auto: new USB bus registered, assigned bus number 6
    [   12.286154] xhci-hcd xhci-hcd.3.auto: Host supports USB 3.0 SuperSpeed
    [   12.286198] usb usb6: We don't know the algorithms for LPM for this host, disabling LPM.
    [   12.286265] usb usb6: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.10
    [   12.286269] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [   12.286272] usb usb6: Product: xHCI Host Controller
    [   12.286275] usb usb6: Manufacturer: Linux 5.10.21-rockchip64 xhci-hcd
    [   12.286278] usb usb6: SerialNumber: xhci-hcd.3.auto
    [   12.286643] hub 6-0:1.0: USB hub found
    [   12.286663] hub 6-0:1.0: 1 port detected
    [   12.321518] .ready
    [   12.323127] sd 5:0:0:0: [sdf] Very big device. Trying to use READ CAPACITY(16).
    [   12.323301] sd 5:0:0:0: [sdf] 27344699392 512-byte logical blocks: (14.0 TB/12.7 TiB)
    [   12.323308] sd 5:0:0:0: [sdf] 4096-byte physical blocks
    [   12.323613] sd 5:0:0:0: [sdf] Write Protect is off
    [   12.323621] sd 5:0:0:0: [sdf] Mode Sense: 47 00 10 08
    [   12.323937] sd 5:0:0:0: [sdf] No Caching mode page found
    [   12.323945] sd 5:0:0:0: [sdf] Assuming drive cache: write through
    [   12.336496] rockchip-rga ff680000.rga: HW Version: 0x03.02
    [   12.336931] rockchip-rga ff680000.rga: Registered rockchip-rga as /dev/video0
    [   12.344976] rockchip_vdec: module is from the staging directory, the quality is unknown, you have been warned.
    [   12.346046] rkvdec ff660000.video-codec: Adding to iommu group 1
    [   12.367793] hantro_vpu: module is from the staging directory, the quality is unknown, you have been warned.
    [   12.370055] hantro-vpu ff650000.video-codec: Adding to iommu group 0
    [   12.371138] hantro-vpu ff650000.video-codec: registered rockchip,rk3399-vpu-enc as /dev/video2
    [   12.371364] hantro-vpu ff650000.video-codec: registered rockchip,rk3399-vpu-dec as /dev/video3
    [   12.374119] panfrost ff9a0000.gpu: clock rate = 500000000
    [   12.377938] panfrost ff9a0000.gpu: mali-t860 id 0x860 major 0x2 minor 0x0 status 0x0
    [   12.377980] panfrost ff9a0000.gpu: features: 00000000,100e77bf, issues: 00000000,24040400
    [   12.377987] panfrost ff9a0000.gpu: Features: L2:0x07120206 Shader:0x00000000 Tiler:0x00000809 Mem:0x1 MMU:0x00002830 AS:0xff JS:0x7
    [   12.377993] panfrost ff9a0000.gpu: shader_present=0xf l2_present=0x1
    [   12.404677] [drm] Initialized panfrost 1.1.0 20180908 for ff9a0000.gpu on minor 1
    [   12.406494]  sdf: sdf1 sdf9
    [   12.409005] sd 5:0:0:0: [sdf] Attached SCSI disk
    [   12.475003] cdn-dp fec00000.dp: [drm:cdn_dp_pd_event_work [rockchipdrm]] Not connected. Disabling cdn
    [   12.706056] usb 4-1.4: reset SuperSpeed Gen 1 USB device number 4 using xhci-hcd
    [   12.720502] cdn-dp fec00000.dp: [drm:cdn_dp_pd_event_work [rockchipdrm]] Not connected. Disabling cdn
    [   12.842267] r8152 4-1.4:1.0 (unnamed net_device) (uninitialized): netif_napi_add() called with weight 256
    [   12.857698] r8152 4-1.4:1.0 eth1: v2.14.0 (2020/09/24)
    [   12.857721] r8152 4-1.4:1.0 eth1: This product is covered by one or more of the following patents:
                           US6,570,884, US6,115,776, and US6,327,625.

    [   12.857946] usbcore: registered new interface driver r8152
    [   13.285587] rockchip-drm display-subsystem: [drm] Cannot find any crtc or sizes
    [   13.376212] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl
    [   13.617036] cdn-dp fec00000.dp: [drm:cdn_dp_pd_event_work [rockchipdrm]] Not connected. Disabling cdn
    [   16.066439] rk_gmac-dwmac fe300000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
    [   16.066483] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    [   16.517337] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    [   16.518338] r8152 4-1.4:1.0 eth1: carrier on
    [   23.267240] systemd[1]: Started Import ZFS pools by cache file.
    [   23.268818] systemd[1]: Reached target ZFS pool import target.
    [   23.275788] systemd[1]: Starting Wait for ZFS Volume (zvol) links in /dev...
    [   23.278492] systemd[1]: Starting Mount ZFS filesystems...
    [   23.384824] systemd[1]: Started Wait for ZFS Volume (zvol) links in /dev.
    [   23.385403] systemd[1]: Reached target ZFS volumes are ready.
    [   24.112115] systemd[1]: Started Mount ZFS filesystems.
    [   24.113331] systemd[1]: Reached target Local File Systems.
    [   24.118830] systemd[1]: Starting Cleanup PHP session files...
    [   24.119585] systemd[1]: Condition check resulted in Mark the need to relabel after reboot being skipped.
    [   24.416171] systemd-journald[1500]: Received request to flush runtime journal from PID 1
    [   35.809318] vcc5v0_typec: disabling
    [   65.064158] softdog: initialized. soft_noboot=0 soft_margin=60 sec soft_panic=0 (nowayout=0)
    [   65.064166] softdog:              soft_reboot_cmd=<not set> soft_active_on_boot=0
    [ 6299.289436] NFSD: Using UMH upcall client tracking operations.
    [ 6299.289445] NFSD: starting 90-second grace period (net f00000e1)
    [48657.415910] xhci-hcd xhci-hcd.0.auto: xHCI host not responding to stop endpoint command.
    [48657.415930] xhci-hcd xhci-hcd.0.auto: USBSTS:
    [48657.429522] xhci-hcd xhci-hcd.0.auto: xHCI host controller not responding, assume dead
    [48657.429604] xhci-hcd xhci-hcd.0.auto: HC died; cleaning up
    [48657.429952] r8152 4-1.4:1.0 eth1: Stop submitting intr, status -108
    [48657.430068] r8152 4-1.4:1.0 eth1: get_registers -110
    [48657.430193] r8152 4-1.4:1.0 eth1: Tx status -108
    [48657.430223] r8152 4-1.4:1.0 eth1: Tx status -108
    [48657.430241] r8152 4-1.4:1.0 eth1: Tx status -108
    [48657.430262] r8152 4-1.4:1.0 eth1: Tx status -108
    [48657.430327] usb 3-1: USB disconnect, device number 2
    [48657.431551] usb 4-1: USB disconnect, device number 2
    [48657.431572] usb 4-1.1: USB disconnect, device number 3
    [48657.467516] usb 4-1.4: USB disconnect, device number 4

     

    Spoiler

    DDR Version 1.24 20191016
    In
    soft reset
    SRX
    channel 0
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 1
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 0 training pass!
    channel 1 training pass!
    change freq to 416MHz 0,1
    Channel 0: LPDDR4,416MHz
    Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
    Channel 1: LPDDR4,416MHz
    Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
    256B stride
    channel 0
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 1
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 0 training pass!
    channel 1 training pass!
    channel 0, cs 0, advanced training done
    channel 1, cs 0, advanced training done
    change freq to 856MHz 1,0
    ch 0 ddrconfig = 0x101, ddrsize = 0x40
    ch 1 ddrconfig = 0x101, ddrsize = 0x40
    pmugrf_os_reg[2] = 0x32C1F2C1, stride = 0xD
    ddr_set_rate to 328MHZ
    ddr_set_rate to 666MHZ
    ddr_set_rate to 928MHZ
    channel 0, cs 0, advanced training done
    channel 1, cs 0, advanced training done
    ddr_set_rate to 416MHZ, ctl_index 0
    ddr_set_rate to 856MHZ, ctl_index 1
    support 416 856 328 666 928 MHz, current 856MHz
    OUT
    Boot1: 2019-03-14, version: 1.19
    CPUId = 0x0
    ChipType = 0x10, 322
    SdmmcInit=2 0
    BootCapSize=100000
    UserCapSize=14910MB
    FwPartOffset=2000 , 100000
    mmc0:cmd5,20
    SdmmcInit=0 0
    BootCapSize=0
    UserCapSize=30436MB
    FwPartOffset=2000 , 0
    StorageInit ok = 78095
    SecureMode = 0
    SecureInit read PBA: 0x4
    SecureInit read PBA: 0x404
    SecureInit read PBA: 0x804
    SecureInit read PBA: 0xc04
    SecureInit read PBA: 0x1004
    SecureInit read PBA: 0x1404
    SecureInit read PBA: 0x1804
    SecureInit read PBA: 0x1c04
    SecureInit ret = 0, SecureMode = 0
    atags_set_bootdev: ret:(0)
    GPT 0x3380ec0 signature is wrong
    recovery gpt...
    GPT 0x3380ec0 signature is wrong
    recovery gpt fail!
    LoadTrust Addr:0x4000
    No find bl30.bin
    No find bl32.bin
    Load uboot, ReadLba = 2000
    Load OK, addr=0x200000, size=0xe5b60
    RunBL31 0x40000
    NOTICE:  BL31: v1.3(debug):42583b6
    NOTICE:  BL31: Built : 07:55:13, Oct 15 2019
    NOTICE:  BL31: Rockchip release version: v1.1
    INFO:    GICv3 with legacy support detected. ARM GICV3 driver initialized in EL3
    INFO:    Using opteed sec cpu_context!
    INFO:    boot cpu mask: 0
    INFO:    plat_rockchip_pmu_init(1190): pd status 3e
    INFO:    BL31: Initializing runtime services
    WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
    ERROR:   Error initializing runtime service opteed_fast
    INFO:    BL31: Preparing for EL3 exit to normal world
    INFO:    Entry point address = 0x200000
    INFO:    SPSR = 0x3c9


    U-Boot 2020.10-armbian (Mar 08 2021 - 14:54:58 +0000)

    SoC: Rockchip rk3399
    Reset cause: unknown reset
    DRAM:  3.9 GiB
    PMIC:  RK808
    SF: Detected w25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
    MMC:   mmc@fe320000: 1, sdhci@fe330000: 0
    Loading Environment from MMC... *** Warning - bad CRC, using default environment

    In:    serial
    Out:   serial
    Err:   serial
    Model: Helios64
    Revision: 1.2 - 4GB non ECC
    Net:   eth0: ethernet@fe300000
    scanning bus for devices...
    starting USB...
    Bus usb@fe380000: USB EHCI 1.00
    Bus dwc3: usb maximum-speed not found
    Register 2000140 NbrPorts 2
    Starting the controller
    USB XHCI 1.10
    scanning bus usb@fe380000 for devices... 1 USB Device(s) found
    scanning bus dwc3 for devices... cannot reset port 4!?
    Device NOT ready
       Request Sense returned 02 04 01
    5 USB Device(s) found
           scanning usb for storage devices... 0 Storage Device(s) found
    Hit any key to stop autoboot:  0
    switch to partitions #0, OK
    mmc1 is current device
    Scanning mmc 1:1...
    Found U-Boot script /boot/boot.scr
    3185 bytes read in 6 ms (517.6 KiB/s)
    ## Executing script at 00500000
    Boot script loaded from mmc 1
    182 bytes read in 5 ms (35.2 KiB/s)
    15760315 bytes read in 672 ms (22.4 MiB/s)
    28582400 bytes read in 1213 ms (22.5 MiB/s)
    81843 bytes read in 13 ms (6 MiB/s)
    2698 bytes read in 10 ms (262.7 KiB/s)
    Applying kernel provided DT fixup script (rockchip-fixup.scr)
    ## Executing script at 09000000
    Moving Image from 0x2080000 to 0x2200000, end=3de0000
    ## Loading init Ramdisk from Legacy Image at 06000000 ...
       Image Name:   uInitrd
       Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
       Data Size:    15760251 Bytes = 15 MiB
       Load Address: 00000000
       Entry Point:  00000000
       Verifying Checksum ... OK
    ## Flattened Device Tree blob at 01f00000
       Booting using the fdt blob at 0x1f00000
       Loading Ramdisk to f4fe7000, end f5eeeb7b ... OK
       Loading Device Tree to 00000000f4f6a000, end 00000000f4fe6fff ... OK

    Starting kernel ...

     

    1g-fix.jpg

  3. New crash, but different behaviour and conditions this time. :(

    It had a kernel oops at around 5am this morning, resulting in a graceful-ish reboot. From my limited understanding, the error seems to be related with swap (the line "Comm: swapper"), so I have disabled the swap partition for now and will continue to monitor.

    The major difference this time is that there was nothing happening at that time, it was just sitting idle.

     

    Spoiler

    [20091.475339] Internal error: Oops: 96000004 [#1] PREEMPT SMP
    [20091.475837] Modules linked in: softdog governor_performance quota_v2 quota_tree snd_soc_hdmi_codec r8152 gpio_charger leds_pwm pwm_fan hantro_vpu(C) rockchip_vdec(C) panfrost v4l2_h264 gpu_sched snd_soc_rockchip_i2s snd_soc_core snd_pcm_dmaengine videobuf2_dma_contig rockchip_rga snd_pcm videobuf2_vmalloc v4l2_mem2mem videobuf2_dma_sg videobuf2_memops videobuf2_v4l2 videobuf2_common snd_timer videodev snd fusb302 soundcore mc tcpm typec rockchipdrm dw_mipi_dsi dw_hdmi analogix_dp sg drm_kms_helper cec rc_core drm drm_panel_orientation_quirks gpio_beeper cpufreq_dt zfs(POE) nfsd zunicode(POE) zzstd(OE) zlua(OE) auth_rpcgss nfs_acl lockd grace zcommon(POE) znvpair(POE) zavl(POE) icp(POE) spl(OE) ledtrig_netdev lm75 sunrpc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod uas realtek dwmac_rk stmmac_platform stmmac pcs_xpcs adc_keys
    [20091.483031] CPU: 4 PID: 0 Comm: swapper/4 Tainted: P         C OE     5.10.21-rockchip64 #21.02.3
    [20091.483807] Hardware name: Helios64 (DT)
    [20091.484154] pstate: 00000085 (nzcv daIf -PAN -UAO -TCO BTYPE=--)
    [20091.484690] pc : irq_work_tick+0x30/0xd0
    [20091.485037] lr : irq_work_tick+0x20/0xd0
    [20091.485382] sp : ffff800011c0bdb0
    [20091.485674] x29: ffff800011c0bdb0 x28: 00001245cd652623
    [20091.486143] x27: ffff0000f779b6c0 x26: 0000000000000002
    [20091.486611] x25: 0000000000000080 x24: ffff80001156a000
    [20091.487079] x23: ffff8000118ba208 x22: ffff80001014da70
    [20091.487547] x21: ffff8000118b99c8 x20: ffff80001157c9f8
    [20091.488014] x19: ffff80001157c9f0 x18: 0000000000000000
    [20091.488483] x17: 0000000000000000 x16: 0000000000000000
    [20091.488950] x15: 0000000bfedea982 x14: 00000000000000f9
    [20091.489418] x13: 00000001004b7f26 x12: 00000000001513fd
    [20091.489886] x11: ffff8000118b7000 x10: ffff80001194ef28
    [20091.490354] x9 : ffff80001194ef20 x8 : ffff800011b72320
    [20091.490821] x7 : ffff800011952000 x6 : 00000070660d244a
    [20091.491289] x5 : b51e853d6cb16636 x4 : ffff8000e6228000
    [20091.491757] x3 : 0000000000010001 x2 : ffff80001156a000
    [20091.492225] x1 : ffff8000112a16a8 x0 : ffff8000e6228000
    [20091.492693] Call trace:
    [20091.492913]  irq_work_tick+0x30/0xd0
    [20091.493231]  update_process_times+0x88/0xa0
    [20091.493600]  tick_sched_handle.isra.19+0x40/0x58
    [20091.494006]  tick_sched_timer+0x58/0xb0
    [20091.494346]  __hrtimer_run_queues+0x104/0x388
    [20091.494730]  hrtimer_interrupt+0xf4/0x250
    [20091.495086]  arch_timer_handler_phys+0x30/0x40
    [20091.495478]  handle_percpu_devid_irq+0xa0/0x298
    [20091.495876]  generic_handle_irq+0x30/0x48
    [20091.496229]  __handle_domain_irq+0x94/0x108
    [20091.496600]  gic_handle_irq+0xc0/0x140
    [20091.496932]  el1_irq+0xc0/0x180
    [20091.497212]  arch_cpu_idle+0x18/0x28
    [20091.497528]  default_idle_call+0x44/0x1bc
    [20091.497884]  do_idle+0x204/0x278
    [20091.498169]  cpu_startup_entry+0x28/0x60
    [20091.498517]  secondary_start_kernel+0x170/0x180
    [20091.498917] Code: f0009dd3 9127c273 f8605aa0 91002274 (f8606a80)
    [20091.499458] ---[ end trace edea193d828b5483 ]---
    [20091.499865] Kernel panic - not syncing: Oops: Fatal exception in interrupt
    [20091.500467] SMP: stopping secondary CPUs
    [20091.500818] Kernel Offset: disabled
    [20091.501127] CPU features: 0x0240022,6100200c
    [20091.501502] Memory Limit: none
    [20091.501781] Rebooting in 90 seconds..

     

  4. @ShadowDance I think you're definitely on to something here! I just ran the Rsync jobs after setting the scheduler to none for all disks and it completed successfully without crashing.

    I'll keep an eye on it and report any other crashes, but for now thank you very much!

     

    Update: It's now a little over 3 hours into a ZFS scrub, and I restarted all my VMs simultaneously *while* doing the scrub and it has not rebooted nor complained about anything on dmesg. This is very promising!

     

    Update 2: Scrub finished without problems!

  5. @aprayogaAs I mentioned, the crash happens when some Rsync jobs are run. I'll try to describe my setup as best as possible.

    My Helios64 is running LK 5.10.21, OMV 5.6.2-1, and has a RAID-Z2 array + a simple EXT4 partition.

    The drives are:

    - Bay 1: 500GB Crucial MX500.

        - partition 1: 16GB swap partition, created while troubleshooting the crashes (to make sure it was not due to lack of memory). The crashes happen both with and without it.
        - partition 2: 400GB EXT4 partition, used to serve VM/container images via NFS.
        - partition 3: 50GB write cache for the ZFS pool.

    - Bays 2 through 5: 4x 4TB Seagate IronWolf (ST4000VN008-2DR1) in RAID-Z2, ashift=12, ZSTD compression (although most of the data is compressed with LZ4).

     

    My remote NAS is a Helios4, also running OMV, with the Rsync plugin in server mode. On the Helios64 I have 4 Rsync jobs for 4 different shared folders, 3 sending and 1 receiving. They all are scheduled to run on Sundays at 3 AM. The reason for starting all 4 jobs at the same time is to maximise the bandwidth usage, since the backup happens over the internet (Wireguard running on the router, not on the H64), and the latency from where I am to where the Helios4 is (Brazil) is quite high. The transfers are incremental, so not a lot of data gets sent with each run, but I guess Rsync still needs to read all the files on the source to see what has changed, causing the high IO load.

    The data on these synced folders is very diverse in size and number of files. In total it's about 1.7TB.

     

    Another way I can reproduce the crash is by running a ZFS scrub.

     

    And also as I mentioned in the previous post, in one occasion, the NAS crashed when generating simultaneous IO from multiple VMs, which sit on an EXT4 partition on the SSD, so it doesn't seem limited to the ZFS pool.

  6. Sorry if I'm hijacking this thread, but the title fits the problem, and I didn't want to create yet another thread about stability issues.

    I'm also having similar problems with my Helios64. It will consistently crash and reboot when I/O is stressed. The two most common scenarios where this happens is when running an Rsync job that I use to sync my H64 with my remote backup NAS (a Helios4 that still works like a champ), and when performing a ZFS scrub. In one isolated occasion, I was also able to trigger a crash by executing an Ansible playbook that made several VMs generate I/O activity simultaneously (the VMs disks are served by the H64).

    Since the Rsync crash is consistent and predictable, I managed to capture the crash output from the serial console in two occasions and under different settings, listed below:

     

    Kernel 5.10.16, performance governor, frequency locked to 1.2GHz

    Spoiler

    [227720.764931] Insufficient stack space to handle exception!
    [227720.764934] ESR: 0x96000047 -- DABT (current EL)
    [227720.764935] FAR: 0xffff800011e77fd0
    [227720.764936] Task stack:     [0xffff800011e78000..0xffff800011e7c000]
    [227720.764938] IRQ stack:      [0xffff800011c10000..0xffff800011c14000]
    [227720.764939] Overflow stack: [0xffff0000f77b22b0..0xffff0000f77b32b0]
    [227720.764941] CPU: 5 PID: 38 Comm: ksoftirqd/5 Tainted: P         C OE     5.10.16-rockchip64 #21.02.2
    [227720.764942] Hardware name: Helios64 (DT)
    [227720.764943] pstate: 40000085 (nZcv daIf -PAN -UAO -TCO BTYPE=--)
    [227720.764944] pc : kfree+0x4/0x480
    [227720.764945] lr : __free_slab+0xac/0x1a8
    [227720.764946] sp : ffff800011e78060
    [227720.764948] x29: ffff800011e78060 x28: 0000000000200001
    [227720.764951] x27: fffffe00012d3c80 x26: ffff0000534f2f80
    [227720.764954] x25: ffff000040001700 x24: fffffffffffff000
    [227720.764957] x23: 0000000000000000 x22: 0000000000200000
    [227720.764959] x21: 0000000000000001 x20: ffff000040000c00
    [227720.764962] x19: fffffe00012d3c80 x18: 0000000000000000
    [227720.764964] x17: 0000000000000000 x16: 0000000000000000
    [227720.764967] x15: 00000bd82b5ac7bf x14: 0000000000000001
    [227720.764969] x13: 00000000f0000000 x12: 0000000000000001
    [227720.764972] x11: 00000000f0000000 x10: 0000000000000001
    [227720.764975] x9 : fffffe0001d823c8 x8 : 0000000000000001
    [227720.764978] x7 : ffff000040000c20 x6 : 0000000000000000
    [227720.764980] x5 : 0000000000000000 x4 : ffff000040001700
    [227720.764983] x3 : 0000000000000020 x2 : ffff0000403b9d00
    [227720.764985] x1 : 1ffff00000000000 x0 : ffff0000534f0f00
    [227720.764988] Kernel panic - not syncing: kernel stack overflow
    [227720.764990] CPU: 5 PID: 38 Comm: ksoftirqd/5 Tainted: P         C OE     5.10.16-rockchip64 #21.02.2
    [227720.764990] Hardware name: Helios64 (DT)
    [227720.764992] Call trace:
    [227720.764993]  dump_backtrace+0x0/0x200
    [227720.764994]  show_stack+0x18/0x68
    [227720.764995]  dump_stack+0xcc/0x124
    [227720.764996]  panic+0x174/0x374
    [227720.764997]  nmi_panic+0x64/0x98
    [227720.764997]  handle_bad_stack+0x124/0x150
    [227720.764999]  __bad_stack+0x90/0x94
    [227720.765000]  kfree+0x4/0x480
    [227720.765001]  discard_slab+0x70/0xb0
    [227720.765002]  __slab_free+0x394/0x3e0
    [227720.765003]  kfree+0x470/0x480
    [227720.765004]  __free_slab+0xac/0x1a8
    [227720.765005]  discard_slab+0x70/0xb0
    [227720.765006]  __slab_free+0x394/0x3e0
    [227720.765007]  kfree+0x470/0x480
    [227720.765008]  __free_slab+0xac/0x1a8
    [227720.765009]  discard_slab+0x70/0xb0
    [227720.765010]  __slab_free+0x394/0x3e0
    [227720.765011]  kfree+0x470/0x480
    [227720.765012]  __free_slab+0xac/0x1a8
    [227720.765013]  discard_slab+0x70/0xb0
    [227720.765014]  __slab_free+0x394/0x3e0
    [227720.765015]  kfree+0x470/0x480
    [227720.765016]  __free_slab+0xac/0x1a8
    [227720.765017]  discard_slab+0x70/0xb0
    [227720.765018]  __slab_free+0x394/0x3e0
    [227720.765020]  kfree+0x470/0x480
    [227720.765020]  __free_slab+0xac/0x1a8
    [227720.765022]  discard_slab+0x70/0xb0
    [227720.765023]  __slab_free+0x394/0x3e0
    [227720.765024]  kfree+0x470/0x480
    [227720.765025]  __free_slab+0xac/0x1a8
    [227720.765026]  discard_slab+0x70/0xb0
    [227720.765027]  __slab_free+0x394/0x3e0
    [227720.765028]  kfree+0x470/0x480
    [227720.765029]  __free_slab+0xac/0x1a8
    [227720.765030]  discard_slab+0x70/0xb0
    [227720.765031]  __slab_free+0x394/0x3e0
    [227720.765032]  kfree+0x470/0x480
    [227720.765033]  __free_slab+0xac/0x1a8
    [227720.765034]  discard_slab+0x70/0xb0
    [227720.765036]  __slab_free+0x394/0x3e0
    [227720.765037]  kfree+0x470/0x480
    [227720.765038]  __free_slab+0xac/0x1a8
    [227720.765039]  discard_slab+0x70/0xb0
    [227720.765040]  __slab_free+0x394/0x3e0
    [227720.765041]  kfree+0x470/0x480
    [227720.765042]  __free_slab+0xac/0x1a8
    [227720.765043]  discard_slab+0x70/0xb0
    [227720.765044]  __slab_free+0x394/0x3e0
    [227720.765045]  kfree+0x470/0x480
    [227720.765046]  __free_slab+0xac/0x1a8
    [227720.765047]  discard_slab+0x70/0xb0
    [227720.765048]  __slab_free+0x394/0x3e0
    [227720.765049]  kfree+0x470/0x480
    [227720.765051]  __free_slab+0xac/0x1a8
    [227720.765051]  discard_slab+0x70/0xb0
    [227720.765053]  __slab_free+0x394/0x3e0
    [227720.765054]  kfree+0x470/0x480
    [227720.765055]  __free_slab+0xac/0x1a8
    [227720.765056]  discard_slab+0x70/0xb0
    [227720.765057]  __slab_free+0x394/0x3e0
    [227720.765058]  kfree+0x470/0x480
    [227720.765059]  __free_slab+0xac/0x1a8
    [227720.765060]  discard_slab+0x70/0xb0
    [227720.765060]  __slab_free+0x394/0x3e0
    [227720.765061]  kfree+0x470/0x480
    [227720.765062]  __free_slab+0xac/0x1a8
    [227720.765063]  discard_slab+0x70/0xb0
    [227720.765064]  __slab_free+0x394/0x3e0
    [227720.765065]  kfree+0x470/0x480
    [227720.765066]  __free_slab+0xac/0x1a8
    [227720.765067]  discard_slab+0x70/0xb0
    [227720.765068]  __slab_free+0x394/0x3e0
    [227720.765069]  kfree+0x470/0x480
    [227720.765070]  __free_slab+0xac/0x1a8
    [227720.765071]  discard_slab+0x70/0xb0
    [227720.765072]  __slab_free+0x394/0x3e0
    [227720.765073]  kfree+0x470/0x480
    [227720.765074]  __free_slab+0xac/0x1a8
    [227720.765076]  discard_slab+0x70/0xb0
    [227720.765077]  __slab_free+0x394/0x3e0
    [227720.765078]  kfree+0x470/0x480
    [227720.765079]  __free_slab+0xac/0x1a8
    [227720.765080]  discard_slab+0x70/0xb0
    [227720.765081]  __slab_free+0x394/0x3e0
    [227720.765082]  kfree+0x470/0x480
    [227720.765083]  __free_slab+0xac/0x1a8
    [227720.765084]  discard_slab+0x70/0xb0
    [227720.765085]  __slab_free+0x394/0x3e0
    [227720.765086]  kfree+0x470/0x480
    [227720.765087]  __free_slab+0xac/0x1a8
    [227720.765088]  discard_slab+0x70/0xb0
    [227720.765089]  __slab_free+0x394/0x3e0
    [227720.765090]  kfree+0x470/0x480
    [227720.765092]  __free_slab+0xac/0x1a8
    [227720.765093]  discard_slab+0x70/0xb0
    [227720.765094]  __slab_free+0x394/0x3e0
    [227720.765095]  kfree+0x470/0x480
    [227720.765096]  __free_slab+0xac/0x1a8
    [227720.765097]  discard_slab+0x70/0xb0
    [227720.765098]  __slab_free+0x394/0x3e0
    [227720.765099]  kfree+0x470/0x480
    [227720.765100]  __free_slab+0xac/0x1a8
    [227720.765101]  discard_slab+0x70/0xb0
    [227720.765102]  __slab_free+0x394/0x3e0
    [227720.765103]  kfree+0x470/0x480
    [227720.765104]  __free_slab+0xac/0x1a8
    [227720.765105]  discard_slab+0x70/0xb0
    [227720.765107]  __slab_free+0x394/0x3e0
    [227720.765107]  kfree+0x470/0x480
    [227720.765108]  __free_slab+0xac/0x1a8
    [227720.765109]  discard_slab+0x70/0xb0
    [227720.765110]  __slab_free+0x394/0x3e0
    [227720.765111]  kfree+0x470/0x480
    [227720.765112]  __free_slab+0xac/0x1a8
    [227720.765114]  discard_slab+0x70/0xb0
    [227720.765114]  __slab_free+0x394/0x3e0
    [227720.765115]  kfree+0x470/0x480
    [227720.765116]  __free_slab+0xac/0x1a8
    [227720.765118]  discard_slab+0x70/0xb0
    [227720.765119]  __slab_free+0x394/0x3e0
    [227720.765120]  kfree+0x470/0x480
    [227720.765121]  __free_slab+0xac/0x1a8
    [227720.765122]  discard_slab+0x70/0xb0
    [227720.765123]  __slab_free+0x394/0x3e0
    [227720.765124]  kfree+0x470/0x480
    [227720.765125]  __free_slab+0xac/0x1a8
    [227720.765127]  discard_slab+0x70/0xb0
    [227720.765128]  __slab_free+0x394/0x3e0
    [227720.765129]  kfree+0x470/0x480
    [227720.765130]  __free_slab+0xac/0x1a8
    [227720.765131]  discard_slab+0x70/0xb0
    [227720.765132]  __slab_free+0x394/0x3e0
    [227720.765133]  kfree+0x470/0x480
    [227720.765134]  __free_slab+0xac/0x1a8
    [227720.765135]  discard_slab+0x70/0xb0
    [227720.765136]  __slab_free+0x394/0x3e0
    [227720.765137]  kfree+0x470/0x480
    [227720.765139]  __free_slab+0xac/0x1a8
    [227720.765139]  discard_slab+0x70/0xb0
    [227720.765141]  __slab_free+0x394/0x3e0
    [227720.765142]  kfree+0x470/0x480
    [227720.765143]  __free_slab+0xac/0x1a8
    [227720.765144]  discard_slab+0x70/0xb0
    [227720.765145]  unfreeze_partials.isra.93+0x1d0/0x250
    [227720.765146]  put_cpu_partial+0x18c/0x230
    [227720.765147]  __slab_free+0x2cc/0x3e0
    [227720.765148]  kfree+0x470/0x480
    [227720.765149]  __d_free_external+0x20/0x40
    [227720.765151]  rcu_core+0x25c/0x820
    [227720.765151]  rcu_core_si+0x10/0x20
    [227720.765153]  efi_header_end+0x160/0x41c
    [227720.765153]  run_ksoftirqd+0x58/0x70
    [227720.765155]  smpboot_thread_fn+0x200/0x238
    [227720.765156]  kthread+0x140/0x150
    [227720.765156]  ret_from_fork+0x10/0x34
    [227720.765177] SMP: stopping secondary CPUs
    [227720.765178] Kernel Offset: disabled
    [227720.765179] CPU features: 0x0240022,6100200c
    [227720.765180] Memory Limit: none

     

    [New boot sequence starts...]

     

     

    Kernel 5.10.21, ondemand governor, dynamic stock frequencies (settings recommended in this and other stability threads):

    Spoiler

    [96770.340605] Insufficient stack space to handle exception!
    [96770.340607] ESR: 0x96000047 -- DABT (current EL)
    [96770.340609] FAR: 0xffff800011e4ffb0
    [96770.340610] Task stack:     [0xffff800011e50000..0xffff800011e54000]
    [96770.340612] IRQ stack:      [0xffff800011c08000..0xffff800011c0c000]
    [96770.340613] Overflow stack: [0xffff0000f77922b0..0xffff0000f77932b0]
    [96770.340615] CPU: 4 PID: 33 Comm: ksoftirqd/4 Tainted: P         C OE     5.10.21-rockchip64 #21.02.3
    [96770.340616] Hardware name: Helios64 (DT)
    [96770.340617] pstate: 80000085 (Nzcv daIf -PAN -UAO -TCO BTYPE=--)
    [96770.340619] pc : __slab_free+0x4/0x3e0
    [96770.340620] lr : kfree+0x370/0x398
    [96770.340621] sp : ffff800011e50070
    [96770.340622] x29: ffff800011e50070 x28: 0000000000100001
    [96770.340626] x27: fffffe000129dfc0 x26: ffff00005277f100
    [96770.340629] x25: 0080000000000000 x24: ffff000040001700
    [96770.340631] x23: 00000000019eed04 x22: ffff00009970af80
    [96770.340634] x21: ffff8000102c18c4 x20: ffff8000118b9948
    [96770.340636] x19: fffffe000245c280 x18: 0000000000000000
    [96770.340639] x17: 00000ff4399e0f64 x16: 00004fda04bdf8a0
    [96770.340642] x15: 000005d80eb36a80 x14: 00000000000001a6
    [96770.340644] x13: 00000000f0000000 x12: 0000000000000001
    [96770.340647] x11: 00000000f0000000 x10: 0000000000000001
    [96770.340650] x9 : fffffe00031e4e88 x8 : 0000000000000001
    [96770.340652] x7 : ffff000040000ba0 x6 : ffff000000000000
    [96770.340655] x5 : ffff8000102c18c4 x4 : 0000000000000001
    [96770.340657] x3 : ffff00009970af80 x2 : ffff00009970af80
    [96770.340660] x1 : fffffe000245c280 x0 : ffff000040001700
    [96770.340663] Kernel panic - not syncing: kernel stack overflow
    [96770.340664] CPU: 4 PID: 33 Comm: ksoftirqd/4 Tainted: P         C OE     5.10.21-rockchip64 #21.02.3
    [96770.340666] Hardware name: Helios64 (DT)
    [96770.340667] Call trace:
    [96770.340668]  dump_backtrace+0x0/0x200
    [96770.340669]  show_stack+0x18/0x68
    [96770.340670]  dump_stack+0xcc/0x124
    [96770.340671]  panic+0x174/0x374
    [96770.340672]  nmi_panic+0x64/0x98
    [96770.340673]  handle_bad_stack+0x124/0x150
    [96770.340675]  __bad_stack+0x90/0x94
    [96770.340676]  __slab_free+0x4/0x3e0
    [96770.340677]  __free_slab+0xac/0x1a8
    [96770.340678]  discard_slab+0x70/0xb0
    [96770.340679]  __slab_free+0x398/0x3e0
    [96770.340680]  kfree+0x370/0x398
    [96770.340681]  __free_slab+0xac/0x1a8
    [96770.340682]  discard_slab+0x70/0xb0
    [96770.340683]  __slab_free+0x398/0x3e0
    [96770.340684]  kfree+0x370/0x398
    [96770.340685]  __free_slab+0xac/0x1a8
    [96770.340686]  discard_slab+0x70/0xb0
    [96770.340687]  __slab_free+0x398/0x3e0
    [96770.340688]  kfree+0x370/0x398
    [96770.340689]  __free_slab+0xac/0x1a8
    [96770.340690]  discard_slab+0x70/0xb0
    [96770.340691]  __slab_free+0x398/0x3e0
    [96770.340692]  kfree+0x370/0x398
    [96770.340693]  __free_slab+0xac/0x1a8
    [96770.340694]  discard_slab+0x70/0xb0
    [96770.340694]  __slab_free+0x398/0x3e0
    [96770.340696]  kfree+0x370/0x398
    [96770.340697]  __free_slab+0xac/0x1a8
    [96770.340697]  discard_slab+0x70/0xb0
    [96770.340698]  __slab_free+0x398/0x3e0
    [96770.340700]  kfree+0x370/0x398
    [96770.340701]  __free_slab+0xac/0x1a8
    [96770.340702]  discard_slab+0x70/0xb0
    [96770.340703]  __slab_free+0x398/0x3e0
    [96770.340704]  kfree+0x370/0x398
    [96770.340705]  __free_slab+0xac/0x1a8
    [96770.340706]  discard_slab+0x70/0xb0
    [96770.340708]  __slab_free+0x398/0x3e0
    [96770.340708]  kfree+0x370/0x398
    [96770.340710]  __free_slab+0xac/0x1a8
    [96770.340711]  discard_slab+0x70/0xb0
    [96770.340712]  __slab_free+0x398/0x3e0
    [96770.340713]  kfree+0x370/0x398
    [96770.340714]  __free_slab+0xac/0x1a8
    [96770.340715]  discard_slab+0x70/0xb0
    [96770.340716]  __slab_free+0x398/0x3e0
    [96770.340717]  kfree+0x370/0x398
    [96770.340718]  __free_slab+0xac/0x1a8
    [96770.340719]  discard_slab+0x70/0xb0
    [96770.340720]  __slab_free+0x398/0x3e0
    [96770.340721]  kfree+0x370/0x398
    [96770.340722]  __free_slab+0xac/0x1a8
    [96770.340724]  discard_slab+0x70/0xb0
    [96770.340725]  __slab_free+0x398/0x3e0
    [96770.340726]  kfree+0x370/0x398
    [96770.340727]  __free_slab+0xac/0x1a8
    [96770.340728]  discard_slab+0x70/0xb0
    [96770.340729]  __slab_free+0x398/0x3e0
    [96770.340730]  kfree+0x370/0x398
    [96770.340731]  __free_slab+0xac/0x1a8
    [96770.340732]  discard_slab+0x70/0xb0
    [96770.340733]  __slab_free+0x398/0x3e0
    [96770.340734]  kfree+0x370/0x398
    [96770.340735]  __free_slab+0xac/0x1a8
    [96770.340736]  discard_slab+0x70/0xb0
    [96770.340737]  __slab_free+0x398/0x3e0
    [96770.340738]  kfree+0x370/0x398
    [96770.340739]  __free_slab+0xac/0x1a8
    [96770.340740]  discard_slab+0x70/0xb0
    [96770.340741]  __slab_free+0x398/0x3e0
    [96770.340743]  kfree+0x370/0x398
    [96770.340743]  __free_slab+0xac/0x1a8
    [96770.340744]  discard_slab+0x70/0xb0
    [96770.340746]  __slab_free+0x398/0x3e0
    [96770.340747]  kfree+0x370/0x398
    [96770.340748]  __free_slab+0xac/0x1a8
    [96770.340748]  discard_slab+0x70/0xb0
    [96770.340749]  __slab_free+0x398/0x3e0
    [96770.340750]  kfree+0x370/0x398
    [96770.340751]  __free_slab+0xac/0x1a8
    [96770.340753]  discard_slab+0x70/0xb0
    [96770.340753]  __slab_free+0x398/0x3e0
    [96770.340755]  kfree+0x370/0x398
    [96770.340755]  __free_slab+0xac/0x1a8
    [96770.340757]  discard_slab+0x70/0xb0
    [96770.340757]  __slab_free+0x398/0x3e0
    [96770.340759]  kfree+0x370/0x398
    [96770.340760]  __free_slab+0xac/0x1a8
    [96770.340760]  discard_slab+0x70/0xb0
    [96770.340762]  __slab_free+0x398/0x3e0
    [96770.340763]  kfree+0x370/0x398
    [96770.340764]  __free_slab+0xac/0x1a8
    [96770.340765]  discard_slab+0x70/0xb0
    [96770.340766]  __slab_free+0x398/0x3e0
    [96770.340767]  kfree+0x370/0x398
    [96770.340768]  __free_slab+0xac/0x1a8
    [96770.340769]  discard_slab+0x70/0xb0
    [96770.340770]  __slab_free+0x398/0x3e0
    [96770.340771]  kfree+0x370/0x398
    [96770.340772]  __free_slab+0xac/0x1a8
    [96770.340773]  discard_slab+0x70/0xb0
    [96770.340774]  __slab_free+0x398/0x3e0
    [96770.340775]  kfree+0x370/0x398
    [96770.340776]  __free_slab+0xac/0x1a8
    [96770.340777]  discard_slab+0x70/0xb0
    [96770.340778]  __slab_free+0x398/0x3e0
    [96770.340779]  kfree+0x370/0x398
    [96770.340780]  __free_slab+0xac/0x1a8
    [96770.340781]  discard_slab+0x70/0xb0
    [96770.340782]  __slab_free+0x398/0x3e0
    [96770.340783]  kfree+0x370/0x398
    [96770.340784]  __free_slab+0xac/0x1a8
    [96770.340785]  discard_slab+0x70/0xb0
    [96770.340786]  __slab_free+0x398/0x3e0
    [96770.340787]  kfree+0x370/0x398
    [96770.340788]  __free_slab+0xac/0x1a8
    [96770.340789]  discard_slab+0x70/0xb0
    [96770.340790]  __slab_free+0x398/0x3e0
    [96770.340791]  kfree+0x370/0x398
    [96770.340792]  __free_slab+0xac/0x1a8
    [96770.340792]  discard_slab+0x70/0xb0
    [96770.340793]  __slab_free+0x398/0x3e0
    [96770.340795]  kfree+0x370/0x398
    [96770.340795]  __free_slab+0xac/0x1a8
    [96770.340797]  discard_slab+0x70/0xb0
    [96770.340798]  __slab_free+0x398/0x3e0
    [96770.340799]  kfree+0x370/0x398
    [96770.340800]  __free_slab+0xac/0x1a8
    [96770.340801]  discard_slab+0x70/0xb0
    [96770.340802]  __slab_free+0x398/0x3e0
    [96770.340803]  kfree+0x370/0x398
    [96770.340804]  __free_slab+0xac/0x1a8
    [96770.340806]  discard_slab+0x70/0xb0
    [96770.340807]  __slab_free+0x398/0x3e0
    [96770.340808]  kfree+0x370/0x398
    [96770.340809]  __free_slab+0xac/0x1a8
    [96770.340810]  discard_slab+0x70/0xb0
    [96770.340811]  __slab_free+0x398/0x3e0
    [96770.340813]  kfree+0x370/0x398
    [96770.340814]  __free_slab+0xac/0x1a8
    [96770.340815]  discard_slab+0x70/0xb0
    [96770.340816]  __slab_free+0x398/0x3e0
    [96770.340817]  kfree+0x370/0x398
    [96770.340818]  __free_slab+0xac/0x1a8
    [96770.340819]  discard_slab+0x70/0xb0
    [96770.340820]  __slab_free+0x398/0x3e0
    [96770.340821]  kfree+0x370/0x398
    [96770.340822]  __free_slab+0xac/0x1a8
    [96770.340823]  discard_slab+0x70/0xb0
    [96770.340824]  unfreeze_partials.isra.93+0x1d0/0x250
    [96770.340826]  put_cpu_partial+0x18c/0x228
    [96770.340827]  __slab_free+0x2d0/0x3e0
    [96770.340827]  kfree+0x370/0x398
    [96770.340829]  __d_free_external+0x20/0x40
    [96770.340830]  rcu_core+0x25c/0x820
    [96770.340831]  rcu_core_si+0x10/0x20
    [96770.340832]  efi_header_end+0x160/0x41c
    [96770.340833]  run_ksoftirqd+0x58/0x70
    [96770.340834]  smpboot_thread_fn+0x200/0x238
    [96770.340835]  kthread+0x140/0x150
    [96770.340836]  ret_from_fork+0x10/0x34
    [96770.340855] SMP: stopping secondary CPUs
    [96770.340857] Kernel Offset: disabled
    [96770.340858] CPU features: 0x0240022,6100200c
    [96770.340859] Memory Limit: none

     

    [New boot sequence starts...]

     

    Happy to help with more information or tests.

  7. Hi all.

    Some time ago, in the release notes of a past Armbian version, I remember reading that the network LED should now be enabled by default and associated with eth0.

    Despite always keeping my Helios64 as up-to-date as possible, mine never lit up, but I didn't think much of it. Today I decided to investigate why, and I'm not sure where to go from here.

     

    First things first, the LED works, it's not a hardware issue. I can echo 1 and 0 to the LED path (/sys/class/leds/helios64:blue:net/brightness) and it lights on and off as expected.

     

    From what I can tell, this LED, along with the heartbeat one, is enabled by the Systemd unit helios64-heartbeat-led.service. When I check for the status of the unit, I get the following output:

    ● helios64-heartbeat-led.service - Enable heartbeat & network activity led on Helios64
       Loaded: loaded (/etc/systemd/system/helios64-heartbeat-led.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Sat 2021-03-13 00:30:52 GMT; 8min ago
      Process: 3740 ExecStart=/usr/bin/bash -c echo heartbeat | tee /sys/class/leds/helios64\:\:status/trigger (code=exited, status=0/SUCCESS)
      Process: 3744 ExecStart=/usr/bin/bash -c echo netdev | tee /sys/class/leds/helios64\:blue\:net/trigger (code=exited, status=1/FAILURE)
     Main PID: 3744 (code=exited, status=1/FAILURE)
    
    Mar 13 00:30:52 nas.lan systemd[1]: Starting Enable heartbeat & network activity led on Helios64...
    Mar 13 00:30:52 nas.lan bash[3740]: heartbeat
    Mar 13 00:30:52 nas.lan bash[3744]: netdev
    Mar 13 00:30:52 nas.lan bash[3744]: tee: '/sys/class/leds/helios64:blue:net/trigger': Invalid argument
    Mar 13 00:30:52 nas.lan systemd[1]: helios64-heartbeat-led.service: Main process exited, code=exited, status=1/FAILURE
    Mar 13 00:30:52 nas.lan systemd[1]: helios64-heartbeat-led.service: Failed with result 'exit-code'.
    Mar 13 00:30:52 nas.lan systemd[1]: Failed to start Enable heartbeat & network activity led on Helios64.

     

    Looking into the unit's source, I can see the line where this fails:

    ExecStart=bash -c 'echo netdev | tee /sys/class/leds/helios64\\:blue\\:net/trigger'

     

    And sure enough, there is no "netdev" inside /sys/class/leds/helios64:blue:net/trigger:

    $ cat /sys/class/leds/helios64:blue:net/trigger
    [none] usb-gadget usb-host kbd-scrolllock kbd-numlock kbd-capslock kbd-kanalock kbd-shiftlock kbd-altgrlock kbd-ctrllock kbd-altlock kbd-shiftllock kbd-shiftrlock kbd-ctrlllock kbd-ctrlrlock usbport mmc1 mmc2 disk-activity disk-read disk-write ide-disk mtd nand-disk heartbeat cpu cpu0 cpu1 cpu2 cpu3 cpu4 cpu5 activity default-on panic stmmac-0:00:link stmmac-0:00:1Gbps stmmac-0:00:100Mbps stmmac-0:00:10Mbps rc-feedback gpio-charger-online tcpm-source-psy-4-0022-online

     

    I'm not sure why it's not there or how does that get populated.

    I'm running Armbian Buster 21.02.3 with LK 5.10.21-rockchip64. Any help will be much appreciated.

  8. 4 hours ago, RamsDeep said:

    Is there a How-To for deploying ZFS anywhere?

    I am a beginner and figuring this out as I go.

    Installing the module from OMV does not appear to work for me currently

     

    Thanks

    Use kernel 5.8 and follow the instructions on the thread above.

  9. A little advice for anyone planning to use nand-sata-install to install on the eMMC and has already installed and configured OMV:

    nand-sata-install will break OMV, but it's easy to fix if you know what's happening. It will skip /srv when copying the root to avoid copying stuff from other mounted filesystems, but OMV 5 stores part of its config in there (salt and pillar folders) and will throw a fit if they're not there when you boot from the eMMC. Simply copy these folders back from the microSD using your preferred method.

    If you have NFS shares set in OMV, make sure to add the entry /export/* to the file /usr/lib/nand-sata-install/exclude.txt BEFORE running nand-sata-install, or it will try to copy the content of your NFS shares to the eMMC.

    Lastly, if you're using ZFS, which by default mounts to /<pool_name>, make sure to add its mountpoint to /usr/lib/nand-sata-install/exclude.txt before nand-sata-install as well.

  10. @RaSca Are your disks powered from the USB port itself, or do they have their own power supplies? Is your USB hub powered externally?

    To me it sounds like the disks are trying to draw more power than the board can provide, and thus failing.

    For the record, I have a 14TB WD Elements (powered by a 12V external PSU) connected to my H64 and running flawlessly, reaching the max speeds the disk can provide (~200MB/s reads).

  11. Idk if these questions are within the scope of this thread, but I'm curious about a few things regarding thermals on the Helios64, if someone from Kobol could clarify:

    - What is the expected temperature range for this SoC?
    - How does the heatsink interface with the SoC? Pads, paste?

    - Does the heatsink also make contact with any other chips around the CPU?

    - Would it benefit in any way from a "re-paste" with high performance thermal paste?

     

    I could, of course, just disassemble mine and find out for myself, but I figured I would ask first.

  12. @KiSMI had the same kind of issue with the legacy image. It seems anything that puts a constant load on the system eventually causes a kernel crash, and sometimes a reboot.

    The current image (kernel 5.8) however is rock solid. I've been using it now for 2 days and really stressing both the CPU and disks on my Helios64 without a single problem.

  13. 8 minutes ago, jbergler said:

     

    In the meantime, I built a zfs 0.8.5 module for the 5.8.13 kernel using a focal container.

    First install this package with dpkg -i kmod-zfs-5.8.13-rockchip64_0.8.5-1_arm64.deb and then install the zfs utils with apt install -t buster-backports zfsutils-linux

     

    Legend! Thank you very much, I'll give this a try asap.

  14. 17 hours ago, gprovost said:

    @antsu Maybe you want to create a dedicated thread "ZFS on Helios64" since it's a big topic on its own ? It will useful for later and this will avoid too many cross discussion here.

    I agree. Apologies for sort of derailing the thread and my sincere thanks to those who tried to help. I'll keep an eye on the other thread I mentioned in my previous posts, since it is already essentially the same topic.

  15. 6 minutes ago, flower said:

     

    if its the disks fault you might see it in smartlog

    smartctl --all /dev/sda

     

    Nothing wrong as far as I can tell:

    sudo smartctl --all /dev/sde
    smartctl 6.6 2017-11-05 r4594 [aarch64-linux-4.4.213-rk3399] (local build)
    Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
    
    === START OF INFORMATION SECTION ===
    Model Family:     Seagate IronWolf
    Device Model:     ST4000VN008-2DR166
    Serial Number:    xxxxxx
    LU WWN Device Id: xxxxxx
    Firmware Version: SC60
    User Capacity:    4,000,787,030,016 bytes [4.00 TB]
    Sector Sizes:     512 bytes logical, 4096 bytes physical
    Rotation Rate:    5980 rpm
    Form Factor:      3.5 inches
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   ACS-3 T13/2161-D revision 5
    SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
    Local Time is:    Wed Oct  7 18:33:40 2020 BST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    General SMART Values:
    Offline data collection status:  (0x82)	Offline data collection activity
    					was completed without error.
    					Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0)	The previous self-test routine completed
    					without error or no self-test has ever 
    					been run.
    Total time to complete Offline 
    data collection: 		(  591) seconds.
    Offline data collection
    capabilities: 			 (0x7b) SMART execute Offline immediate.
    					Auto Offline data collection on/off support.
    					Suspend Offline collection upon new
    					command.
    					Offline surface scan supported.
    					Self-test supported.
    					Conveyance Self-test supported.
    					Selective Self-test supported.
    SMART capabilities:            (0x0003)	Saves SMART data before entering
    					power-saving mode.
    					Supports SMART auto save timer.
    Error logging capability:        (0x01)	Error logging supported.
    					General Purpose Logging supported.
    Short self-test routine 
    recommended polling time: 	 (   1) minutes.
    Extended self-test routine
    recommended polling time: 	 ( 664) minutes.
    Conveyance self-test routine
    recommended polling time: 	 (   2) minutes.
    SCT capabilities: 	       (0x50bd)	SCT Status supported.
    					SCT Error Recovery Control supported.
    					SCT Feature Control supported.
    					SCT Data Table supported.
    
    SMART Attributes Data Structure revision number: 10
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x000f   080   064   044    Pre-fail  Always       -       97064976
      3 Spin_Up_Time            0x0003   094   093   000    Pre-fail  Always       -       0
      4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       132
      5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x000f   093   060   045    Pre-fail  Always       -       2082420238
      9 Power_On_Hours          0x0032   083   083   000    Old_age   Always       -       14904 (177 35 0)
     10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       132
    184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
    187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
    188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       2
    189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
    190 Airflow_Temperature_Cel 0x0022   071   053   040    Old_age   Always       -       29 (Min/Max 28/29)
    191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
    192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       47
    193 Load_Cycle_Count        0x0032   088   088   000    Old_age   Always       -       24785
    194 Temperature_Celsius     0x0022   029   047   000    Old_age   Always       -       29 (0 16 0 0 0)
    197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
    240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       14694 (50 14 0)
    241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       50906738288
    242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       157149639538
    
    SMART Error Log Version: 1
    No Errors Logged
    
    SMART Self-test log structure revision number 1
    Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
    # 1  Short offline       Completed without error       00%     14904         -
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.
    

     

  16. Spoiler

     

    15 hours ago, aprayoga said:

    So with Armbian 20.08.8 LEGACY,  the system still crashed? Let us know if you managed to get the log and a way to reproduce the crash.

    I managed to catch the error this time, although I suspect because I caught it almost immediately after it happened, the system did not have a chance to become completely unresponsive.

    As you will see in the log below, the disk in the 5th bay (sde) became "disconnected" after the error, which makes me wonder if a software fault caused the disk to "soft-disconnect" or if the disks' hardware is failing after hours of intense activity and triggered the error (in which case I'm just wasting everyone's time). After a forced reboot the disk appears to be working fine and resilvered in seconds, so I'm really not sure.

     

    Please see below:

     

     Steps to reproduce:

    
    - Flashed Armbian_20.08.8_Helios64_buster_legacy_4.4.213.img.xz and went through initial setup.
    - sudo apt-get update && sudo apt-get dist-upgrade -y
    - sudo apt-get install linux-headers-legacy-rk3399
    - sudo apt-get install zfs-dkms -t buster-backports
    - installed OMV through armbian-config
    - logged in to OMV web interface and enabled the omv-extras repo
    - still in OMV, went to Plugins and installed openmediavault-zfs (regular, not armhf)
    - enabled SMART monitoring for all disks (not sure if relevant)
    - From the newly-added "ZFS" section in OMV, imported my ZFS pool (4x4TB RAID-Z2).
    - Created some shared folders and shared them via NFS and Samba
    - Started a scrub on the ZFS pool, this puts some medium/high load on the CPU and disks.
    - Have two clients (Proxmox) accesssing a couple of VM disks via NFS.

     

     

    Error log  after approx. 6 hours (from serial terminal):

    
    [13483.617705] Internal error: Oops: 96000004 [#1] SMP
    [13483.619387] Modules linked in: zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) softdog r8152 af_packet snd_soc_rockchip_hdmi_dp midgard_kbase sg cdn_dp uio_pdrv_g
    enirq gpio_beeper uio nfsd auth_rpcgss nfs_acl lockd lm75 grace sunrpc ip_tables x_tables raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md
    _mod
    [13483.627827] CPU: 5 PID: 203 Comm: irq/259-ahci4 Tainted: P           O    4.4.213-rk3399 #3
    [13483.629885] Hardware name: Helios64 (DT)
    [13483.631530] task: ffffffc0e1612a00 task.stack: ffffffc0e17f0000
    [13483.633360] PC is at irq_thread_fn+0x44/0x74
    [13483.635044] LR is at irq_thread_fn+0x30/0x74
    [13483.636698] pc : [<ffffff8008113e68>] lr : [<ffffff8008113e54>] pstate: 60000145
    [13483.638635] sp : ffffffc0e17f3d50
    [13483.640208] x29: ffffffc0e17f3d50 x28: 0000000000000001 
    [13483.641982] x27: ffffffc0e1612a00 x26: 0000000000000001 
    [13483.643735] x25: ffffffc0e1612a00 x24: ffffffc0ed84e440 
    [13483.645474] x23: ffffff8009409201 x22: ffffff8008113fd0 
    [13483.647188] x21: 0000000000000001 x20: ffffffc0ed84e400 
    [13483.648889] x19: ffffe2c0eadc1200 x18: 0000000030d00800 
    [13483.650573] x17: 00000000b995214c x16: 0000000000000050 
    [13483.652244] x15: 000000000000000e x14: 0000000000000007 
    [13483.653916] x13: 0000000000000001 x12: 0000000000000028 
    [13483.655568] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f 
    [13483.657222] x9 : 656824ff6462716e x8 : 7f7f7fffffffffff 
    [13483.658867] x7 : 6567fffefefefefe x6 : 00000040eec83000 
    [13483.660510] x5 : ffffff800867ea2c x4 : 0000000000000001 
    [13483.662146] x3 : ffffffbdc1891410 x2 : ffffe2c0eadc1294 
    [13483.663795] x1 : ffffffc0e1612a00 x0 : 0000000000000001 
    [13483.665445] 
    [13483.665445] PC: 0xffffff8008113de8:
    [13483.668159] 3de8  37800080 36880060 aa1303e0 94000aea aa1603e0 942a6c05 f9401e60 f9403c01
    [13483.670116] 3e08  b4000061 aa1503e0 d63f0020 a94153f3 a9425bf5 a8c37bfd d65f03c0 a9bd7bfd
    [13483.672079] 3e28  910003fd a90153f3 f90013f5 aa0103f4 aa0003f3 aa1e03e0 d503201f b9403a80
    [13483.674046] 3e48  f9400681 f9401282 d63f0040 2a0003f5 7100041f 540000e1 91025262 f9800051
    [13483.676011] 3e68  885f7c40 11000400 88017c40 35ffffa1 aa1403e1 aa1303e0 97ffffae 2a1503e0
    [13483.678002] 3e88  a94153f3 f94013f5 a8c37bfd d65f03c0 a9bd7bfd 910003fd a90153f3 f90013f5
    [13483.680014] 3ea8  aa0103f4 aa0003f3 aa1e03e0 d503201f d5384101 b9401020 11080000 b9001020
    [13483.682056] 3ec8  b9403a80 f9400681 f9401282 d63f0040 2a0003f5 7100041f 540000e1 91025262
    [13483.684116] 
    [13483.684116] LR: 0xffffff8008113dd4:
    [13483.687088] 3dd4  f9406e62 8a200040 f9006e60 b50000c0 b9400020 37800080 36880060 aa1303e0
    [13483.689173] 3df4  94000aea aa1603e0 942a6c05 f9401e60 f9403c01 b4000061 aa1503e0 d63f0020
    [13483.691253] 3e14  a94153f3 a9425bf5 a8c37bfd d65f03c0 a9bd7bfd 910003fd a90153f3 f90013f5
    [13483.693334] 3e34  aa0103f4 aa0003f3 aa1e03e0 d503201f b9403a80 f9400681 f9401282 d63f0040
    [13483.695424] 3e54  2a0003f5 7100041f 540000e1 91025262 f9800051 885f7c40 11000400 88017c40
    [13483.697554] 3e74  35ffffa1 aa1403e1 aa1303e0 97ffffae 2a1503e0 a94153f3 f94013f5 a8c37bfd
    [13483.699665] 3e94  d65f03c0 a9bd7bfd 910003fd a90153f3 f90013f5 aa0103f4 aa0003f3 aa1e03e0
    [13483.701788] 3eb4  d503201f d5384101 b9401020 11080000 b9001020 b9403a80 f9400681 f9401282
    [13483.703917] 
    [13483.703917] SP: 0xffffffc0e17f3cd0:
    [13483.706981] 3cd0  08113fd0 ffffff80 09409201 ffffff80 ed84e440 ffffffc0 e1612a00 ffffffc0
    [13483.709064] 3cf0  00000001 00000000 e1612a00 ffffffc0 00000001 00000000 e17f3d50 ffffffc0
    [13483.711114] 3d10  08113e54 ffffff80 e17f3d50 ffffffc0 08113e68 ffffff80 60000145 00000000
    [13483.713134] 3d30  eadc1200 ffffe2c0 ed84e400 ffffffc0 ffffffff ffffffff 08113fd0 ffffff80
    [13483.715127] 3d50  e17f3d80 ffffffc0 081141a8 ffffff80 eadc1200 ffffe2c0 08113e24 ffffff80
    [13483.717124] 3d70  ed84e400 ffffffc0 08114128 ffffff80 e17f3e00 ffffffc0 080d6d80 ffffff80
    [13483.719118] 3d90  ed858280 ffffffc0 e1612a00 ffffffc0 08e216d2 ffffff80 ed84e400 ffffffc0
    [13483.721093] 3db0  08114070 ffffff80 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.723082] 
    [13483.723082] X1: 0xffffffc0e1612980:
    [13483.725898] 2980  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.727901] 29a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.729921] 29c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.731907] 29e0  00000000 00000000 00000000 00000000 00000000 00000000 e1612a00 ffffffc0
    [13483.733899] 2a00  00000012 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
    [13483.735883] 2a20  e17f0000 ffffffc0 00000003 00208040 00000000 00000000 00000000 00000000
    [13483.737875] 2a40  00000001 00000005 0000007e 00000000 00324a49 00000001 e1b10000 ffffffc0
    [13483.739857] 2a60  00000005 00000001 00000031 00000078 00000031 00000032 08bce9c0 ffffff80
    [13483.741870] 
    [13483.741870] X3: 0xffffffbdc1891390:
    [13483.744738] 1390  00000000 00000000 ffffffff 00000001 00000100 dead0000 00000200 dead0000
    [13483.746778] 13b0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.748830] 13d0  00000000 00000000 ffffffff 00000001 00000100 dead0000 00000200 dead0000
    [13483.750868] 13f0  00000000 00000000 00000000 00000000 00004080 00000000 00000000 00000000
    [13483.752919] 1410  62451380 ffffffc0 8015000d 00000001 c0fd7400 ffffffbd 00000007 00000007
    [13483.754964] 1430  edda7500 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.757019] 1450  00000001 00000000 ffffffff 00000000 c1891401 ffffffbd 00000001 00000001
    [13483.759087] 1470  00000000 00000000 00000000 00000000 00010028 00000000 dd7bc708 ffffffc0
    [13483.761155] 
    [13483.761155] X5: 0xffffff800867e9ac:
    [13483.764040] e9ac  7900fa74 7900fe60 17fffff4 7940fe60 11000400 12003c00 7900fe60 7100281f
    [13483.766075] e9cc  54fffdc9 34000061 7900f261 d5033e9f 7940f260 17ffffea a9be7bfd 910003fd
    [13483.768104] e9ec  a90153f3 aa0103f3 aa0003f4 aa1e03e0 d503201f f9407e61 b4000081 d0007c40
    [13483.770135] ea0c  f9428000 97ed96c5 f9400680 f9408a61 97ed96c2 f9400280 aa1303e1 97ed96bf
    [13483.772161] ea2c  a94153f3 a8c27bfd d65f03c0 a9bd7bfd 910003fd a90153f3 f90013f5 aa0003f3
    [13483.774187] ea4c  b00067f4 aa1e03e0 9131e294 d503201f 9101e280 f9408675 9414ba34 f9408660
    [13483.776210] ea6c  b9413801 340000a1 f940a013 b9401260 35000100 d4210000 39482261 91006280
    [13483.778266] ea8c  91012293 f27e003f 9a930013 17fffff8 51000400 b9001260 35000140 f9400260
    [13483.780315] 
    [13483.780315] X20: 0xffffffc0ed84e380:
    [13483.783241] e380  0000000c 00000000 e1612fa8 ffffffc0 00000000 00000000 00000000 00000000
    [13483.785290] e3a0  e1612588 ffffffc0 080d4414 ffffff80 000000cb 00000000 092fdf88 ffffff80
    [13483.787306] e3c0  00000000 00000000 f7ed1718 ffffffc0 ed84e3d0 ffffffc0 ed84e3d0 ffffffc0
    [13483.789351] e3e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.791360] e400  086b5924 ffffff80 e1608000 ffffffc0 00000000 00000000 00000000 00000000
    [13483.793372] e420  086b8974 ffffff80 e1612a00 ffffffc0 00000000 00000000 00000103 00000000
    [13483.795337] e440  00000001 00000000 00000000 00000000 ed8580a8 ffffffc0 ed85a540 ffffffc0
    [13483.797327] e460  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.799292] 
    [13483.799292] X22: 0xffffff8008113f50:
    [13483.802078] 3f50  f9400e60 f90017e0 aa1403e0 942a6bad d5384100 9100a3e1 97ff3a9e a94153f3
    [13483.804053] 3f70  a8c37bfd d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f
    [13483.806048] 3f90  91038262 f9800051 885f7c40 51000400 8801fc40 35ffffa1 d5033bbf 350000c0
    [13483.808054] 3fb0  9103a260 d2800003 52800022 52800061 97ffa91f f9400bf3 a8c27bfd d65f03c0
    [13483.810073] 3fd0  a9be7bfd 910003fd a90153f3 aa1e03e0 d503201f d5384114 b9402e80 37100160
    [13483.812094] 3ff0  d00097a0 91080400 39400401 35000081 d4210000 52800021 39000401 a94153f3
    [13483.814110] 4010  a8c27bfd d65f03c0 aa1403e0 97ff0d20 aa0003f3 b9454a82 911c6281 d0006880
    [13483.816154] 4030  913ef400 b9403a63 94020060 b9403a60 97fff98f aa0003f4 91010261 52800000
    [13483.818192] 
    [13483.818192] X23: 0xffffff8009409181:
    [13483.821156] 9180  09223c58 ffffff80 00000000 00000000 ed93fdb0 ffffffc0 ed93fdb0 ffffffc0
    [13483.823255] 91a0  081787a0 ffffff80 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.825382] 91c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.827473] 91e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.829568] 9200  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.831632] 9220  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.833680] 9240  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.835700] 9260  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.837699] 9280  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.839683] 
    [13483.839683] X24: 0xffffffc0ed84e3c0:
    [13483.842519] e3c0  00000000 00000000 f7ed1718 ffffffc0 ed84e3d0 ffffffc0 ed84e3d0 ffffffc0
    [13483.844531] e3e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.846546] e400  086b5924 ffffff80 e1608000 ffffffc0 00000000 00000000 00000000 00000000
    [13483.848552] e420  086b8974 ffffff80 e1612a00 ffffffc0 00000000 00000000 00000103 00000000
    [13483.850546] e440  00000001 00000000 00000000 00000000 ed8580a8 ffffffc0 ed85a540 ffffffc0
    [13483.852534] e460  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.854528] e480  0a400000 ffffff80 0a501000 ffffff80 00000004 00000000 eaf04d99 ffffffc0
    [13483.856510] e4a0  eae07f98 ffffffc0 eb95ed18 ffffffc0 a2634d30 ffffffc0 eaf04030 ffffffc0
    [13483.858484] 
    [13483.858484] X25: 0xffffffc0e1612980:
    [13483.861247] 2980  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.863204] 29a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.865140] 29c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.867055] 29e0  00000000 00000000 00000000 00000000 00000000 00000000 e1612a00 ffffffc0
    [13483.868958] 2a00  00000012 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
    [13483.870827] 2a20  e17f0000 ffffffc0 00000003 00208040 00000000 00000000 00000000 00000000
    [13483.872696] 2a40  00000001 00000005 0000007e 00000000 00324a49 00000001 e1b10000 ffffffc0
    [13483.874557] 2a60  00000005 00000001 00000031 00000078 00000031 00000032 08bce9c0 ffffff80
    [13483.876404] 
    [13483.876404] X27: 0xffffffc0e1612980:
    [13483.878960] 2980  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.880835] 29a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.882702] 29c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.884538] 29e0  00000000 00000000 00000000 00000000 00000000 00000000 e1612a00 ffffffc0
    [13483.886378] 2a00  00000012 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
    [13483.888197] 2a20  e17f0000 ffffffc0 00000003 00208040 00000000 00000000 00000000 00000000
    [13483.890017] 2a40  00000001 00000005 0000007e 00000000 00324a49 00000001 e1b10000 ffffffc0
    [13483.891823] 2a60  00000005 00000001 00000031 00000078 00000031 00000032 08bce9c0 ffffff80
    [13483.893636] 
    [13483.893636] X29: 0xffffffc0e17f3cd0:
    [13483.896040] 3cd0  08113fd0 ffffff80 09409201 ffffff80 ed84e440 ffffffc0 e1612a00 ffffffc0
    [13483.897831] 3cf0  00000001 00000000 e1612a00 ffffffc0 00000001 00000000 e17f3d50 ffffffc0
    [13483.899604] 3d10  08113e54 ffffff80 e17f3d50 ffffffc0 08113e68 ffffff80 60000145 00000000
    [13483.901394] 3d30  eadc1200 ffffe2c0 ed84e400 ffffffc0 ffffffff ffffffff 08113fd0 ffffff80
    [13483.903162] 3d50  e17f3d80 ffffffc0 081141a8 ffffff80 eadc1200 ffffe2c0 08113e24 ffffff80
    [13483.904923] 3d70  ed84e400 ffffffc0 08114128 ffffff80 e17f3e00 ffffffc0 080d6d80 ffffff80
    [13483.906675] 3d90  ed858280 ffffffc0 e1612a00 ffffffc0 08e216d2 ffffff80 ed84e400 ffffffc0
    [13483.908411] 3db0  08114070 ffffff80 00000000 00000000 00000000 00000000 00000000 00000000
    [13483.910140] 
    [13483.911218] Process irq/259-ahci4 (pid: 203, stack limit = 0xffffffc0e17f0000)
    [13483.912847] Stack: (0xffffffc0e17f3d50 to 0xffffffc0e17f4000)
    [13483.914375] 3d40:                                   ffffffc0e17f3d80 ffffff80081141a8
    [13483.916130] 3d60: ffffe2c0eadc1200 ffffff8008113e24 ffffffc0ed84e400 ffffff8008114128
    [13483.917912] 3d80: ffffffc0e17f3e00 ffffff80080d6d80 ffffffc0ed858280 ffffffc0e1612a00
    [13483.919697] 3da0: ffffff8008e216d2 ffffffc0ed84e400 ffffff8008114070 0000000000000000
    [13483.921510] 3dc0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13483.923305] 3de0: ffffffc0ed858280 ffffffc0e1612a00 0000000000000000 ffffff8008113fd0
    [13483.925110] 3e00: 0000000000000000 ffffff8008082ef0 ffffff80080d6ca0 ffffffc0ed858280
    [13483.926905] 3e20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13483.928696] 3e40: ffffffc0e17f3e60 0000000000000000 ffffffc0ee71c600 ffffffc0ed84e400
    [13483.930514] 3e60: ffffffc000000000 dead4ead00000000 ffffffc0ffffffff ffffffffffffffff
    [13483.932333] 3e80: ffffffc0e17f3e80 ffffffc0e17f3e80 0000000000000000 dead4ead00000000
    [13483.934153] 3ea0: 00000000ffffffff ffffffffffffffff ffffffc0e17f3eb0 ffffffc0e17f3eb0
    [13483.935970] 3ec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13483.937801] 3ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13483.939617] 3f00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13483.941400] 3f20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13483.943172] 3f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13483.944912] 3f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13483.946651] 3f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13483.948394] 3fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13483.950125] 3fc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000
    [13483.951843] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13483.953555] Call trace:
    [13483.954772] Exception stack(0xffffffc0e17f3b80 to 0xffffffc0e17f3cb0)
    [13483.956364] 3b80: ffffe2c0eadc1200 0000008000000000 ffffffc0e17f3d50 ffffff8008113e68
    [13483.958081] 3ba0: ffffffc062451380 0000000000001d4c ffffffc062451080 0000000000000005
    [13483.959808] 3bc0: ffffffc0e17f3bf0 ffffff80084c2ed0 ffffffc0e17f3c10 ffffff80092b3000
    [13483.961522] 3be0: ffffff80092e6100 0000000000000004 ffffffc0e17f3c20 ffffff8008081160
    [13483.963249] 3c00: ffffffc0e17f3c20 ffffff8008081064 0000000000000000 ffffff800808100c
    [13483.964997] 3c20: 0000000000000001 ffffffc0e1612a00 ffffe2c0eadc1294 ffffffbdc1891410
    [13483.966763] 3c40: 0000000000000001 ffffff800867ea2c 00000040eec83000 6567fffefefefefe
    [13483.968527] 3c60: 7f7f7fffffffffff 656824ff6462716e 7f7f7f7f7f7f7f7f 0101010101010101
    [13483.970309] 3c80: 0000000000000028 0000000000000001 0000000000000007 000000000000000e
    [13483.972097] 3ca0: 0000000000000050 00000000b995214c
    [13483.973621] [<ffffff8008113e68>] irq_thread_fn+0x44/0x74
    [13483.975183] [<ffffff80081141a8>] irq_thread+0x138/0x1a0
    [13483.976737] [<ffffff80080d6d80>] kthread+0xe0/0xf0
    [13483.978256] [<ffffff8008082ef0>] ret_from_fork+0x10/0x20
    [13483.979789] Code: 7100041f 540000e1 91025262 f9800051 (885f7c40) 
    [13483.981388] ---[ end trace 14e9e3e474676f58 ]---
    [13483.984979] Unable to handle kernel paging request at virtual address ffffffffffffffc8
    [13483.986806] pgd = ffffffc0e1721000
    [13483.988191] [ffffffffffffffc8] *pgd=00000000ebcb8003, *pud=00000000ebcb8003, *pmd=0000000000000000
    [13483.990180] Internal error: Oops: 96000005 [#2] SMP
    [13483.991735] Modules linked in: zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) softdog r8152 af_packet snd_soc_rockchip_hdmi_dp midgard_kbase sg cdn_dp uio_pdrv_g
    enirq gpio_beeper uio nfsd auth_rpcgss nfs_acl lockd lm75 grace sunrpc ip_tables x_tables raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md
    _mod
    [13483.999941] CPU: 4 PID: 203 Comm: irq/259-ahci4 Tainted: P      D    O    4.4.213-rk3399 #3
    [13484.001913] Hardware name: Helios64 (DT)
    [13484.003495] task: ffffffc0e1612a00 task.stack: ffffffc0e17f0000
    [13484.005270] PC is at kthread_data+0x24/0x2c
    [13484.006888] LR is at irq_thread_dtor+0x50/0xa0
    [13484.008517] pc : [<ffffff80080d74c0>] lr : [<ffffff8008114020>] pstate: 60000145
    [13484.010427] sp : ffffffc0e17f3a00
    [13484.011995] x29: ffffffc0e17f3a00 x28: ffffffc0e1612a00 
    [13484.013766] x27: ffffffc0e1612a00 x26: 0000000000000001 
    [13484.015514] x25: 0000000000000000 x24: ffffff80092e60c0 
    [13484.017253] x23: ffffff8008e1ca7e x22: ffffffc0e1613200 
    [13484.018975] x21: ffffff80094438a0 x20: ffffffc0e1612a00 
    [13484.020686] x19: ffffffc0e1612a00 x18: 0000000030d00800 
    [13484.022405] x17: 0000000000000000 x16: 0000000000000000 
    [13484.024131] x15: 0000000000000000 x14: ffffffc0ee6db800 
    [13484.025868] x13: 00000040eec6f000 x12: 0000000034d5d91d 
    [13484.027616] x11: 0000000000000000 x10: 0000000000000a30 
    [13484.029370] x9 : ffffffc0e17f3970 x8 : ffffffc0e1613490 
    [13484.031151] x7 : 0000000000000004 x6 : 0000000000668ae2 
    [13484.032935] x5 : 0000004b6e4f92ab x4 : 0000000000000000 
    [13484.034723] x3 : ffffffc0e16131b8 x2 : ffffffc0e1613200 
    [13484.036530] x1 : ffffff8008113fd0 x0 : 0000000000000000 
    [13484.038313] 
    [13484.038313] PC: 0xffffff80080d7440:
    [13484.041302] 7440  f85b8000 53030c14 721d001f 540000e0 52800014 2a1403e0 a94153f3 f94013f5
    [13484.043397] 7460  a8c37bfd d65f03c0 52800040 aa1503e1 941021c0 d5384100 eb00027f 54fffea0
    [13484.045500] 7480  aa1303e0 94003111 910062a0 942b5492 17fffff1 128004b4 17ffffef a9be7bfd
    [13484.047617] 74a0  910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f f9430a60 f9400bf3 a8c27bfd
    [13484.049733] 74c0  f85c8000 d65f03c0 a9bd7bfd 910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f
    [13484.051831] 74e0  9100a3e0 d2800102 f90017ff f9430a61 d100e021 940308b2 f9400bf3 f94017e0
    [13484.053944] 7500  a8c37bfd d65f03c0 a9bf7bfd 910003fd aa1e03e0 d503201f 12800000 a8c17bfd
    [13484.056045] 7520  d65f03c0 a9be7bfd 910003fd a90153f3 aa0003f3 aa0103f4 aa1e03e0 d503201f
    [13484.058156] 
    [13484.058156] LR: 0xffffff8008113fa0:
    [13484.061204] 3fa0  8801fc40 35ffffa1 d5033bbf 350000c0 9103a260 d2800003 52800022 52800061
    [13484.063345] 3fc0  97ffa91f f9400bf3 a8c27bfd d65f03c0 a9be7bfd 910003fd a90153f3 aa1e03e0
    [13484.065506] 3fe0  d503201f d5384114 b9402e80 37100160 d00097a0 91080400 39400401 35000081
    [13484.067674] 4000  d4210000 52800021 39000401 a94153f3 a8c27bfd d65f03c0 aa1403e0 97ff0d20
    [13484.069845] 4020  aa0003f3 b9454a82 911c6281 d0006880 913ef400 b9403a63 94020060 b9403a60
    [13484.072024] 4040  97fff98f aa0003f4 91010261 52800000 940f2ee4 34000060 aa1403e0 97ffffc7
    [13484.074201] 4060  aa1303e1 aa1403e0 97ffff34 17ffffe8 a9b87bfd 910003fd a90153f3 a9025bf5
    [13484.076378] 4080  a90363f7 a9046bf9 a90573fb aa0003f5 aa1e03e0 d503201f b9403aa0 97fff978
    [13484.078549] 
    [13484.078549] SP: 0xffffffc0e17f3980:
    [13484.081667] 3980  e1613200 ffffffc0 08e1ca7e ffffff80 092e60c0 ffffff80 00000000 00000000
    [13484.083767] 39a0  00000001 00000000 e1612a00 ffffffc0 e1612a00 ffffffc0 e17f3a00 ffffffc0
    [13484.085845] 39c0  08114020 ffffff80 e17f3a00 ffffffc0 080d74c0 ffffff80 60000145 00000000
    [13484.087882] 39e0  e17f3a00 ffffffc0 08114020 ffffff80 00000000 00000080 080d7b50 ffffff80
    [13484.089912] 3a00  e17f3a20 ffffffc0 080d4fa0 ffffff80 e1612a00 ffffffc0 00000000 00000000
    [13484.091927] 3a20  e17f3a50 ffffffc0 080bd3e0 ffffff80 e1612a00 ffffffc0 00000000 00000000
    [13484.093925] 3a40  e1613000 ffffffc0 e1613200 ffffffc0 e17f3ad0 ffffffc0 08088430 ffffff80
    [13484.095911] 3a60  e17f3c20 ffffffc0 e1612a00 ffffffc0 00000001 00000000 092f2e30 ffffff80
    [13484.097910] 
    [13484.097910] X1: 0xffffff8008113f50:
    [13484.100691] 3f50  f9400e60 f90017e0 aa1403e0 942a6bad d5384100 9100a3e1 97ff3a9e a94153f3
    [13484.102683] 3f70  a8c37bfd d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f
    [13484.104695] 3f90  91038262 f9800051 885f7c40 51000400 8801fc40 35ffffa1 d5033bbf 350000c0
    [13484.106727] 3fb0  9103a260 d2800003 52800022 52800061 97ffa91f f9400bf3 a8c27bfd d65f03c0
    [13484.108776] 3fd0  a9be7bfd 910003fd a90153f3 aa1e03e0 d503201f d5384114 b9402e80 37100160
    [13484.110853] 3ff0  d00097a0 91080400 39400401 35000081 d4210000 52800021 39000401 a94153f3
    [13484.112960] 4010  a8c27bfd d65f03c0 aa1403e0 97ff0d20 aa0003f3 b9454a82 911c6281 d0006880
    [13484.115081] 4030  913ef400 b9403a63 94020060 b9403a60 97fff98f aa0003f4 91010261 52800000
    [13484.117219] 
    [13484.117219] X2: 0xffffffc0e1613180:
    [13484.120334] 3180  00000000 00000000 00000000 00000000 e1613190 ffffffc0 e1613190 ffffffc0
    [13484.122505] 31a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.124677] 31c0  00000000 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
    [13484.126845] 31e0  00000000 00000000 2fa82fa8 dead4ead ffffffff 00000000 ffffffff ffffffff
    [13484.129033] 3200  31823182 dead4ead ffffffff 00000000 ffffffff ffffffff 00000000 00000000
    [13484.131212] 3220  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.133407] 3240  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.135585] 3260  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.137736] 
    [13484.137736] X3: 0xffffffc0e1613138:
    [13484.140844] 3138  e1613138 ffffffc0 e1613138 ffffffc0 00000000 00000000 00000000 00000000
    [13484.142994] 3158  00000000 00000000 00000000 00000000 ed86b600 ffffffc0 ed866600 ffffffc0
    [13484.145134] 3178  00000000 00000000 00000000 00000000 00000000 00000000 e1613190 ffffffc0
    [13484.147231] 3198  e1613190 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.149343] 31b8  00000000 00000000 00000000 00000000 ffffffff ffffffff 00000000 00000000
    [13484.151417] 31d8  00000000 00000000 00000000 00000000 2fa82fa8 dead4ead ffffffff 00000000
    [13484.153488] 31f8  ffffffff ffffffff 31823182 dead4ead ffffffff 00000000 ffffffff ffffffff
    [13484.155574] 3218  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.157650] 
    [13484.157650] X8: 0xffffffc0e1613410:
    [13484.160604] 3410  00000000 00000000 00000000 00000000 00000000 00000001 00000000 00000000
    [13484.162651] 3430  ee76f000 ffffffc0 e1612a00 ffffffc0 ee76f000 ffffffc0 092b3000 ffffff80
    [13484.164691] 3450  eb6230c0 ffffffc0 08bce9c0 ffffff80 f7f3f200 ffffffc0 00000001 00000000
    [13484.166724] 3470  08bce000 ffffff80 e1613090 ffffffc0 e17f3970 ffffffc0 e17f3970 ffffffc0
    [13484.168748] 3490  08084f9c ffffff80 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.170777] 34b0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.172780] 34d0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.174771] 34f0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.176729] 
    [13484.176729] X9: 0xffffffc0e17f38f0:
    [13484.179499] 38f0  00000000 00000000 6e4f92ab 0000004b 00668ae2 00000000 00000004 00000000
    [13484.181472] 3910  e1613490 ffffffc0 e17f3970 ffffffc0 00000a30 00000000 00000000 00000000
    [13484.183440] 3930  34d5d91d 00000000 eec6f000 00000040 ee6db800 ffffffc0 00000000 00000000
    [13484.185434] 3950  00000000 00000000 00000000 00000000 30d00800 00000000 e1612a00 ffffffc0
    [13484.187430] 3970  e1612a00 ffffffc0 094438a0 ffffff80 e1613200 ffffffc0 08e1ca7e ffffff80
    [13484.189431] 3990  092e60c0 ffffff80 00000000 00000000 00000001 00000000 e1612a00 ffffffc0
    [13484.191409] 39b0  e1612a00 ffffffc0 e17f3a00 ffffffc0 08114020 ffffff80 e17f3a00 ffffffc0
    [13484.193413] 39d0  080d74c0 ffffff80 60000145 00000000 e17f3a00 ffffffc0 08114020 ffffff80
    [13484.195414] 
    [13484.195414] X14: 0xffffffc0ee6db780:
    [13484.198217] b780  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.200191] b7a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.202144] b7c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.204060] b7e0  00000000 00000000 00000000 00000000 08be2d58 ffffff80 024200ca 00000000
    [13484.205959] b800  00000010 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
    [13484.207845] b820  ee6f0000 ffffffc0 00000002 00200040 00000000 00000000 00000000 00000000
    [13484.209725] b840  00000000 00000004 00000030 00000000 00324a96 00000001 ead73800 ffffffc0
    [13484.211604] b860  00000004 00000001 00000078 00000078 00000078 00000000 08bce6e0 ffffff80
    [13484.213469] 
    [13484.213469] X19: 0xffffffc0e1612980:
    [13484.216058] 2980  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.217955] 29a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.219818] 29c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.221673] 29e0  00000000 00000000 00000000 00000000 00000000 00000000 e1612a00 ffffffc0
    [13484.223529] 2a00  00000012 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
    [13484.225410] 2a20  e17f0000 ffffffc0 00000003 00208044 00000000 00000000 00000000 00000000
    [13484.227255] 2a40  00000001 00000004 00000080 00000000 00324a49 00000001 e17c3800 ffffffc0
    [13484.229126] 2a60  00000004 00000001 00000031 00000078 00000031 00000032 08bce9c0 ffffff80
    [13484.230971] 
    [13484.230971] X20: 0xffffffc0e1612980:
    [13484.233509] 2980  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.235374] 29a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.237240] 29c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.239084] 29e0  00000000 00000000 00000000 00000000 00000000 00000000 e1612a00 ffffffc0
    [13484.240905] 2a00  00000012 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
    [13484.242725] 2a20  e17f0000 ffffffc0 00000003 00208044 00000000 00000000 00000000 00000000
    [13484.244508] 2a40  00000001 00000004 00000080 00000000 00324a49 00000001 e17c3800 ffffffc0
    [13484.246318] 2a60  00000004 00000001 00000031 00000078 00000031 00000032 08bce9c0 ffffff80
    [13484.248097] 
    [13484.248097] X21: 0xffffff8009443820:
    [13484.250532] 3820  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.252337] 3840  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.254114] 3860  00000000 00000000 00000000 00000000 ee580f40 ffffffc0 ee580fc0 ffffffc0
    [13484.255895] 3880  ee580f00 ffffffc0 ee580f80 ffffffc0 00000000 00000000 f7ec9f00 ffffffc0
    [13484.257717] 38a0  00000000 00000000 00000000 00000000 ededa500 ffffffc0 00000001 00000000
    [13484.259551] 38c0  ee5f0e00 ffffffc0 ee403600 ffffffc0 edda6080 ffffffc0 00000000 00000000
    [13484.261388] 38e0  ee401c00 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.263222] 3900  ee4f7600 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.265039] 
    [13484.265039] X22: 0xffffffc0e1613180:
    [13484.267469] 3180  00000000 00000000 00000000 00000000 e1613190 ffffffc0 e1613190 ffffffc0
    [13484.269299] 31a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.271095] 31c0  00000000 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
    [13484.272891] 31e0  00000000 00000000 2fa82fa8 dead4ead ffffffff 00000000 ffffffff ffffffff
    [13484.274676] 3200  31823182 dead4ead ffffffff 00000000 ffffffff ffffffff 00000000 00000000
    [13484.276473] 3220  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.278260] 3240  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.280033] 3260  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.281790] 
    [13484.281790] X23: 0xffffff8008e1c9fe:
    [13484.284179] c9fc  206b7374 7025203d 43000a29 206c6c61 63617274 000a3a65 65637845 6f697470
    [13484.285986] ca1c  7473206e 006b6361 6e493001 6e726574 65206c61 726f7272 7325203a 7825203a
    [13484.287798] ca3c  25235b20 53205d64 000a504d 72503001 7365636f 2e252073 2820732a 3a646970
    [13484.289641] ca5c  2c642520 61747320 6c206b63 74696d69 30203d20 29702578 7453000a 3a6b6361
    [13484.291492] ca7c  30010020 74614600 65206c61 70656378 6e6f6974 206e6920 65746e69 70757272
    [13484.293366] ca9c  61460074 206c6174 65637865 6f697470 6f4f006e 2d207370 47554220 25360100
    [13484.295217] cabc  64255b73 75203a5d 6665646e 64656e69 736e6920 63757274 6e6f6974 6370203a
    [13484.297108] cadc  0a70253d 00360100 73706f4f 75202d20 6665646e 64656e69 736e6920 63757274
    [13484.298982] cafc  6e6f6974 25360100 64255b73 73203a5d 61637379 25206c6c 01000a64 64614232
    [13484.300891] 
    [13484.300891] X24: 0xffffff80092e6040:
    [13484.303589] 6040  f547a000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.305562] 6060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.307494] 6080  00000000 dead4ead ffffffff 00000000 ffffffff ffffffff 00000000 00000000
    [13484.309438] 60a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.311372] 60c0  00000000 deaf1eed ffffffff 00000000 ffffffff ffffffff 00000000 00000000
    [13484.313311] 60e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.315236] 6100  080bec20 ffffff80 081249c8 ffffff80 08a14824 ffffff80 08a17fe8 ffffff80
    [13484.317170] 6120  084c2e48 ffffff80 084c3528 ffffff80 080bed14 ffffff80 080f777c ffffff80
    [13484.319096] 
    [13484.319096] X27: 0xffffffc0e1612980:
    [13484.321870] 2980  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.323851] 29a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.325829] 29c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.327774] 29e0  00000000 00000000 00000000 00000000 00000000 00000000 e1612a00 ffffffc0
    [13484.329713] 2a00  00000012 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
    [13484.331633] 2a20  e17f0000 ffffffc0 00000003 00208044 00000000 00000000 00000000 00000000
    [13484.333579] 2a40  00000001 00000004 00000080 00000000 00324a49 00000001 e17c3800 ffffffc0
    [13484.335492] 2a60  00000004 00000001 00000031 00000078 00000031 00000032 08bce9c0 ffffff80
    [13484.337442] 
    [13484.337442] X28: 0xffffffc0e1612980:
    [13484.340141] 2980  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.342086] 29a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.344004] 29c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [13484.345912] 29e0  00000000 00000000 00000000 00000000 00000000 00000000 e1612a00 ffffffc0
    [13484.347821] 2a00  00000012 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
    [13484.349725] 2a20  e17f0000 ffffffc0 00000003 00208044 00000000 00000000 00000000 00000000
    [13484.351612] 2a40  00000001 00000004 00000080 00000000 00324a49 00000001 e17c3800 ffffffc0
    [13484.353511] 2a60  00000004 00000001 00000031 00000078 00000031 00000032 08bce9c0 ffffff80
    [13484.355391] 
    [13484.355391] X29: 0xffffffc0e17f3980:
    [13484.358041] 3980  e1613200 ffffffc0 08e1ca7e ffffff80 092e60c0 ffffff80 00000000 00000000
    [13484.359940] 39a0  00000001 00000000 e1612a00 ffffffc0 e1612a00 ffffffc0 e17f3a00 ffffffc0
    [13484.361854] 39c0  08114020 ffffff80 e17f3a00 ffffffc0 080d74c0 ffffff80 60000145 00000000
    [13484.363782] 39e0  e17f3a00 ffffffc0 08114020 ffffff80 00000000 00000080 080d7b50 ffffff80
    [13484.365741] 3a00  e17f3a20 ffffffc0 080d4fa0 ffffff80 e1612a00 ffffffc0 00000000 00000000
    [13484.367681] 3a20  e17f3a50 ffffffc0 080bd3e0 ffffff80 e1612a00 ffffffc0 00000000 00000000
    [13484.369630] 3a40  e1613000 ffffffc0 e1613200 ffffffc0 e17f3ad0 ffffffc0 08088430 ffffff80
    [13484.371535] 3a60  e17f3c20 ffffffc0 e1612a00 ffffffc0 00000001 00000000 092f2e30 ffffff80
    [13484.373441] 
    [13484.374654] Process irq/259-ahci4 (pid: 203, stack limit = 0xffffffc0e17f0000)
    [13484.376398] Stack: (0xffffffc0e17f3a00 to 0xffffffc0e17f4000)
    [13484.378028] 3a00: ffffffc0e17f3a20 ffffff80080d4fa0 ffffffc0e1612a00 0000000000000000
    [13484.379836] 3a20: ffffffc0e17f3a50 ffffff80080bd3e0 ffffffc0e1612a00 0000000000000000
    [13484.381619] 3a40: ffffffc0e1613000 ffffffc0e1613200 ffffffc0e17f3ad0 ffffff8008088430
    [13484.383389] 3a60: ffffffc0e17f3c20 ffffffc0e1612a00 0000000000000001 ffffff80092f2e30
    [13484.385134] 3a80: ffffff8008e1ca7e 0000000000000140 ffffffc0e1612a00 0000000000000001
    [13484.386869] 3aa0: ffffffc0e1612a00 ffffffc0e1612a00 ffffffc0e17f3ad0 ffffff800808841c
    [13484.388617] 3ac0: ffffffc0e17f3c20 ffffffc0e1612a00 ffffffc0e17f3b10 ffffff8008093544
    [13484.390372] 3ae0: ffffe2c0eadc1294 ffffffc0e17f3c20 0000000096000004 ffffffc0eb6230c0
    [13484.392112] 3b00: 0000000060000145 0000000000000025 ffffffc0e17f3b40 ffffff8008093968
    [13484.393886] 3b20: ffffffc0e17f3c20 ffffe2c0eadc1294 0000000096000004 ffffffc0eb6230c0
    [13484.395649] 3b40: ffffffc0e17f3b70 ffffff8008080b08 0000000096000004 ffffe2c0eadc1294
    [13484.397429] 3b60: ffffffc0e17f3c20 ffffff8008bc7070 ffffffc0e17f3d50 ffffff80080825cc
    [13484.399202] 3b80: ffffe2c0eadc1200 0000008000000000 ffffffc0e17f3d50 ffffff8008113e68
    [13484.400967] 3ba0: ffffffc062451380 0000000000001d4c ffffffc062451080 0000000000000005
    [13484.402730] 3bc0: ffffffc0e17f3bf0 ffffff80084c2ed0 ffffffc0e17f3c10 ffffff80092b3000
    [13484.404484] 3be0: ffffff80092e6100 0000000000000004 ffffffc0e17f3c20 ffffff8008081160
    [13484.406256] 3c00: ffffffc0e17f3c20 ffffff8008081064 0000000000000000 ffffff800808100c
    [13484.408000] 3c20: 0000000000000001 ffffffc0e1612a00 ffffe2c0eadc1294 ffffffbdc1891410
    [13484.409788] 3c40: 0000000000000001 ffffff800867ea2c 00000040eec83000 6567fffefefefefe
    [13484.411571] 3c60: 7f7f7fffffffffff 656824ff6462716e 7f7f7f7f7f7f7f7f 0101010101010101
    [13484.413384] 3c80: 0000000000000028 0000000000000001 0000000000000007 000000000000000e
    [13484.415211] 3ca0: 0000000000000050 00000000b995214c 0000000030d00800 ffffe2c0eadc1200
    [13484.417037] 3cc0: ffffffc0ed84e400 0000000000000001 ffffff8008113fd0 ffffff8009409201
    [13484.418854] 3ce0: ffffffc0ed84e440 ffffffc0e1612a00 0000000000000001 ffffffc0e1612a00
    [13484.420684] 3d00: 0000000000000001 ffffffc0e17f3d50 ffffff8008113e54 ffffffc0e17f3d50
    [13484.422517] 3d20: ffffff8008113e68 0000000060000145 ffffe2c0eadc1200 ffffffc0ed84e400
    [13484.424347] 3d40: ffffffffffffffff ffffff8008113fd0 ffffffc0e17f3d80 ffffff80081141a8
    [13484.426181] 3d60: ffffe2c0eadc1200 ffffff8008113e24 ffffffc0ed84e400 ffffff8008114128
    [13484.428012] 3d80: ffffffc0e17f3e00 ffffff80080d6d80 ffffffc0ed858280 ffffffc0e1612a00
    [13484.429871] 3da0: ffffff8008e216d2 ffffffc0ed84e400 ffffff8008114070 0000000000000000
    [13484.431730] 3dc0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13484.433601] 3de0: ffffffc0ed858280 ffffffc0e1612a00 0000000000000000 ffffff8008113fd0
    [13484.435468] 3e00: 0000000000000000 ffffff8008082ef0 ffffff80080d6ca0 ffffffc0ed858280
    [13484.437342] 3e20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13484.439220] 3e40: ffffffc0e17f3e60 0000000000000000 ffffffc0ee71c600 ffffffc0ed84e400
    [13484.441081] 3e60: ffffffc000000000 dead4ead00000000 ffffffc0ffffffff ffffffffffffffff
    [13484.442954] 3e80: ffffffc0e17f3e80 ffffffc0e17f3e80 0000000000000001 dead4ead00010001
    [13484.444822] 3ea0: 00000000ffffffff ffffffffffffffff ffffffc0e17f3eb0 ffffffc0e17f3eb0
    [13484.446698] 3ec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13484.448571] 3ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13484.450445] 3f00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13484.452279] 3f20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13484.454141] 3f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13484.455964] 3f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13484.457776] 3f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13484.459550] 3fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13484.461296] 3fc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000
    [13484.463020] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    [13484.464703] Call trace:
    [13484.465921] Exception stack(0xffffffc0e17f3830 to 0xffffffc0e17f3960)
    [13484.467513] 3820:                                   ffffffc0e1612a00 0000008000000000
    [13484.469248] 3840: ffffffc0e17f3a00 ffffff80080d74c0 0000000000000000 ffffffc0e17f3a70
    [13484.470983] 3860: ffffffc0edf40000 ffffffc0e17f38b0 ffffffc0e17f38a0 ffffff8008baec70
    [13484.472722] 3880: ffffffc0e17f38a0 ffffff8008baee00 00000000000001c0 000000000001d7de
    [13484.474478] 38a0: ffffffc0e17f3910 ffffff80080fb598 ffffffc0f7f3f3d0 ffffffc0e1612ca8
    [13484.476226] 38c0: 0000000000003221 ffffffc0f7f3f200 0000000000000000 ffffff8008113fd0
    [13484.477981] 38e0: ffffffc0e1613200 ffffffc0e16131b8 0000000000000000 0000004b6e4f92ab
    [13484.479717] 3900: 0000000000668ae2 0000000000000004 ffffffc0e1613490 ffffffc0e17f3970
    [13484.481483] 3920: 0000000000000a30 0000000000000000 0000000034d5d91d 00000040eec6f000
    [13484.483240] 3940: ffffffc0ee6db800 0000000000000000 0000000000000000 0000000000000000
    [13484.485003] [<ffffff80080d74c0>] kthread_data+0x24/0x2c
    [13484.486527] [<ffffff80080d4fa0>] task_work_run+0xa8/0xcc
    [13484.488055] [<ffffff80080bd3e0>] do_exit+0x420/0x8d8
    [13484.489545] [<ffffff8008088430>] die+0x1a4/0x1b8
    [13484.490991] [<ffffff8008093544>] __do_kernel_fault.part.0+0x78/0x88
    [13484.492584] [<ffffff8008093968>] do_translation_fault+0xb8/0xc8
    [13484.494143] [<ffffff8008080b08>] do_mem_abort+0x54/0xac
    [13484.495622] Exception stack(0xffffffc0e17f3b80 to 0xffffffc0e17f3cb0)
    [13484.497210] 3b80: ffffe2c0eadc1200 0000008000000000 ffffffc0e17f3d50 ffffff8008113e68
    [13484.498949] 3ba0: ffffffc062451380 0000000000001d4c ffffffc062451080 0000000000000005
    [13484.500681] 3bc0: ffffffc0e17f3bf0 ffffff80084c2ed0 ffffffc0e17f3c10 ffffff80092b3000
    [13484.502423] 3be0: ffffff80092e6100 0000000000000004 ffffffc0e17f3c20 ffffff8008081160
    [13484.504148] 3c00: ffffffc0e17f3c20 ffffff8008081064 0000000000000000 ffffff800808100c
    [13484.505891] 3c20: 0000000000000001 ffffffc0e1612a00 ffffe2c0eadc1294 ffffffbdc1891410
    [13484.507619] 3c40: 0000000000000001 ffffff800867ea2c 00000040eec83000 6567fffefefefefe
    [13484.509357] 3c60: 7f7f7fffffffffff 656824ff6462716e 7f7f7f7f7f7f7f7f 0101010101010101
    [13484.511105] 3c80: 0000000000000028 0000000000000001 0000000000000007 000000000000000e
    [13484.512870] 3ca0: 0000000000000050 00000000b995214c
    [13484.514353] [<ffffff80080825cc>] el1_da+0x24/0x84
    [13484.515810] [<ffffff80081141a8>] irq_thread+0x138/0x1a0
    [13484.517312] [<ffffff80080d6d80>] kthread+0xe0/0xf0
    [13484.518766] [<ffffff8008082ef0>] ret_from_fork+0x10/0x20
    [13484.520256] Code: d503201f f9430a60 f9400bf3 a8c27bfd (f85c8000) 
    [13484.521839] ---[ end trace 14e9e3e474676f59 ]---
    [13484.525820] Fixing recursive fault but reboot is needed!
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.617705] Internal error: Oops: 96000004 [#1] SMP
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.911218] Process irq/259-ahci4 (pid: 203, stack limit = 0xffffffc0e17f0000)
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.912847] Stack: (0xffffffc0e17f3d50 to 0xffffffc0e17f4000)
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.914375] 3d40:                                   ffffffc0e17f3d80 ffffff80081141a8
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.916130] 3d60: ffffe2c0eadc1200 ffffff8008113e24 ffffffc0ed84e400 ffffff8008114128
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.917912] 3d80: ffffffc0e17f3e00 ffffff80080d6d80 ffffffc0ed858280 ffffffc0e1612a00
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.919697] 3da0: ffffff8008e216d2 ffffffc0ed84e400 ffffff8008114070 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.921510] 3dc0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.923305] 3de0: ffffffc0ed858280 ffffffc0e1612a00 0000000000000000 ffffff8008113fd0
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.925110] 3e00: 0000000000000000 ffffff8008082ef0 ffffff80080d6ca0 ffffffc0ed858280
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.926905] 3e20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.928696] 3e40: ffffffc0e17f3e60 0000000000000000 ffffffc0ee71c600 ffffffc0ed84e400
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.930514] 3e60: ffffffc000000000 dead4ead00000000 ffffffc0ffffffff ffffffffffffffff
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.932333] 3e80: ffffffc0e17f3e80 ffffffc0e17f3e80 0000000000000000 dead4ead00000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.934153] 3ea0: 00000000ffffffff ffffffffffffffff ffffffc0e17f3eb0 ffffffc0e17f3eb0
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.935970] 3ec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.937801] 3ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.939617] 3f00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.941400] 3f20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.943172] 3f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.944912] 3f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.946651] 3f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.948394] 3fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.950125] 3fc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.951843] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.979789] Code: 7100041f 540000e1 91025262 f9800051 (885f7c40) 
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13483.990180] Internal error: Oops: 96000005 [#2] SMP
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.374654] Process irq/259-ahci4 (pid: 203, stack limit = 0xffffffc0e17f0000)
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.376398] Stack: (0xffffffc0e17f3a00 to 0xffffffc0e17f4000)
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.378028] 3a00: ffffffc0e17f3a20 ffffff80080d4fa0 ffffffc0e1612a00 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.379836] 3a20: ffffffc0e17f3a50 ffffff80080bd3e0 ffffffc0e1612a00 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.381619] 3a40: ffffffc0e1613000 ffffffc0e1613200 ffffffc0e17f3ad0 ffffff8008088430
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.383389] 3a60: ffffffc0e17f3c20 ffffffc0e1612a00 0000000000000001 ffffff80092f2e30
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.385134] 3a80: ffffff8008e1ca7e 0000000000000140 ffffffc0e1612a00 0000000000000001
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.386869] 3aa0: ffffffc0e1612a00 ffffffc0e1612a00 ffffffc0e17f3ad0 ffffff800808841c
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.388617] 3ac0: ffffffc0e17f3c20 ffffffc0e1612a00 ffffffc0e17f3b10 ffffff8008093544
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.390372] 3ae0: ffffe2c0eadc1294 ffffffc0e17f3c20 0000000096000004 ffffffc0eb6230c0
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.392112] 3b00: 0000000060000145 0000000000000025 ffffffc0e17f3b40 ffffff8008093968
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.393886] 3b20: ffffffc0e17f3c20 ffffe2c0eadc1294 0000000096000004 ffffffc0eb6230c0
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.395649] 3b40: ffffffc0e17f3b70 ffffff8008080b08 0000000096000004 ffffe2c0eadc1294
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.397429] 3b60: ffffffc0e17f3c20 ffffff8008bc7070 ffffffc0e17f3d50 ffffff80080825cc
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.399202] 3b80: ffffe2c0eadc1200 0000008000000000 ffffffc0e17f3d50 ffffff8008113e68
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.400967] 3ba0: ffffffc062451380 0000000000001d4c ffffffc062451080 0000000000000005
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.402730] 3bc0: ffffffc0e17f3bf0 ffffff80084c2ed0 ffffffc0e17f3c10 ffffff80092b3000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.404484] 3be0: ffffff80092e6100 0000000000000004 ffffffc0e17f3c20 ffffff8008081160
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.406256] 3c00: ffffffc0e17f3c20 ffffff8008081064 0000000000000000 ffffff800808100c
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.408000] 3c20: 0000000000000001 ffffffc0e1612a00 ffffe2c0eadc1294 ffffffbdc1891410
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.409788] 3c40: 0000000000000001 ffffff800867ea2c 00000040eec83000 6567fffefefefefe
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.411571] 3c60: 7f7f7fffffffffff 656824ff6462716e 7f7f7f7f7f7f7f7f 0101010101010101
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.413384] 3c80: 0000000000000028 0000000000000001 0000000000000007 000000000000000e
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.415211] 3ca0: 0000000000000050 00000000b995214c 0000000030d00800 ffffe2c0eadc1200
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.417037] 3cc0: ffffffc0ed84e400 0000000000000001 ffffff8008113fd0 ffffff8009409201
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.418854] 3ce0: ffffffc0ed84e440 ffffffc0e1612a00 0000000000000001 ffffffc0e1612a00
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.420684] 3d00: 0000000000000001 ffffffc0e17f3d50 ffffff8008113e54 ffffffc0e17f3d50
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.422517] 3d20: ffffff8008113e68 0000000060000145 ffffe2c0eadc1200 ffffffc0ed84e400
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.424347] 3d40: ffffffffffffffff ffffff8008113fd0 ffffffc0e17f3d80 ffffff80081141a8
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.426181] 3d60: ffffe2c0eadc1200 ffffff8008113e24 ffffffc0ed84e400 ffffff8008114128
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.428012] 3d80: ffffffc0e17f3e00 ffffff80080d6d80 ffffffc0ed858280 ffffffc0e1612a00
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.429871] 3da0: ffffff8008e216d2 ffffffc0ed84e400 ffffff8008114070 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.431730] 3dc0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.433601] 3de0: ffffffc0ed858280 ffffffc0e1612a00 0000000000000000 ffffff8008113fd0
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.435468] 3e00: 0000000000000000 ffffff8008082ef0 ffffff80080d6ca0 ffffffc0ed858280
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.437342] 3e20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.439220] 3e40: ffffffc0e17f3e60 0000000000000000 ffffffc0ee71c600 ffffffc0ed84e400
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.441081] 3e60: ffffffc000000000 dead4ead00000000 ffffffc0ffffffff ffffffffffffffff
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.442954] 3e80: ffffffc0e17f3e80 ffffffc0e17f3e80 0000000000000001 dead4ead00010001
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.444822] 3ea0: 00000000ffffffff ffffffffffffffff ffffffc0e17f3eb0 ffffffc0e17f3eb0
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.446698] 3ec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.448571] 3ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.450445] 3f00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.452279] 3f20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.454141] 3f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.455964] 3f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.457776] 3f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.459550] 3fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.461296] 3fc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.463020] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
    
    Message from syslogd@localhost at Oct  7 17:41:03 ...
     kernel:[13484.520256] Code: d503201f f9430a60 f9400bf3 a8c27bfd (f85c8000) 
    [13514.580264] ata5.00: exception Emask 0x0 SAct 0x1c00 SErr 0x0 action 0x6 frozen
    [13514.583877] ata5.00: failed command: WRITE FPDMA QUEUED
    [13514.587339] ata5.00: cmd 61/08:50:10:82:00/00:00:0c:01:00/40 tag 10 ncq 4096 out
    [13514.587339]          res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
    [13514.594777] ata5.00: status: { DRDY }
    [13514.598247] ata5.00: failed command: WRITE FPDMA QUEUED
    [13514.601932] ata5.00: cmd 61/18:58:40:8c:05/00:00:0c:01:00/40 tag 11 ncq 12288 out
    [13514.601932]          res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
    [13514.609896] ata5.00: status: { DRDY }
    [13514.613629] ata5.00: failed command: WRITE FPDMA QUEUED
    [13514.617568] ata5.00: cmd 61/18:60:38:d4:01/00:00:10:01:00/40 tag 12 ncq 12288 out
    [13514.617568]          res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
    [13514.625792] ata5.00: status: { DRDY }
    [13514.629550] ata5: hard resetting link
    [13515.124053] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    [13520.127922] ata5.00: qc timeout (cmd 0xec)
    [13520.131747] ata5.00: failed to IDENTIFY (I/O error, err_mask=0x4)
    [13520.135765] ata5.00: revalidation failed (errno=-5)
    [13520.139617] ata5: hard resetting link
    [13520.631960] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
    [13530.635648] ata5.00: qc timeout (cmd 0xec)
    [13530.639451] ata5.00: failed to IDENTIFY (I/O error, err_mask=0x4)
    [13530.643448] ata5.00: revalidation failed (errno=-5)
    [13530.647276] ata5: limiting SATA link speed to 3.0 Gbps
    [13530.651134] ata5: hard resetting link
    [13531.143648] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
    [13561.146868] ata5.00: qc timeout (cmd 0xec)
    [13561.150716] ata5.00: failed to IDENTIFY (I/O error, err_mask=0x4)
    [13561.154765] ata5.00: revalidation failed (errno=-5)
    [13561.158690] ata5.00: disabled
    [13561.162424] ata5: hard resetting link
    [13561.658842] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
    [13561.662914] sd 4:0:0:0: [sde] tag#10 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
    [13561.667160] sd 4:0:0:0: [sde] tag#10 Sense Key : 0x5 [current] [descriptor] 
    [13561.671328] sd 4:0:0:0: [sde] tag#10 ASC=0x21 ASCQ=0x4 
    [13561.675285] sd 4:0:0:0: [sde] tag#10 CDB: opcode=0x8a 8a 00 00 00 00 01 0c 00 82 10 00 00 00 08 00 00
    [13561.679675] blk_update_request: I/O error, dev sde, sector 4496327184
    [13561.683825] zio pool=storage vdev=/dev/disk/by-id/wwn-0x5000c500b27f7291-part1 error=5 type=2 offset=2302118469632 size=4096 flags=180880
    [13561.692231] sd 4:0:0:0: [sde] tag#11 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
    [13561.696717] sd 4:0:0:0: [sde] tag#11 Sense Key : 0x5 [current] [descriptor] 
    [13561.701146] sd 4:0:0:0: [sde] tag#11 ASC=0x21 ASCQ=0x4 
    [13561.705398] sd 4:0:0:0: [sde] tag#11 CDB: opcode=0x8a 8a 00 00 00 00 01 0c 05 8c 40 00 00 00 18 00 00
    [13561.710091] blk_update_request: I/O error, dev sde, sector 4496657472
    [13561.714557] zio pool=storage vdev=/dev/disk/by-id/wwn-0x5000c500b27f7291-part1 error=5 type=2 offset=2302287577088 size=12288 flags=40080c80
    [13561.723709] sd 4:0:0:0: [sde] tag#12 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
    [13561.728582] sd 4:0:0:0: [sde] tag#12 Sense Key : 0x5 [current] [descriptor] 
    [13561.733326] sd 4:0:0:0: [sde] tag#12 ASC=0x21 ASCQ=0x4 
    [13561.737866] sd 4:0:0:0: [sde] tag#12 CDB: opcode=0x8a 8a 00 00 00 00 01 10 01 d4 38 00 00 00 18 00 00
    [13561.742813] blk_update_request: I/O error, dev sde, sector 4563522616
    [13561.747537] zio pool=storage vdev=/dev/disk/by-id/wwn-0x5000c500b27f7291-part1 error=5 type=2 offset=2336522530816 size=12288 flags=40080c80
    [13561.757162] ata5: EH complete
    [13561.761717] sd 4:0:0:0: [sde] tag#15 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00
    [13561.766846] sd 4:0:0:0: [sde] tag#15 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 77 80 00 00 00 08 00 00
    [13561.766937] sd 4:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13561.766941] sd 4:0:0:0: [sde] Sense not available.
    [13561.767036] sd 4:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13561.767038] sd 4:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13561.767039] sd 4:0:0:0: [sde] Sense not available.
    [13561.767040] sd 4:0:0:0: [sde] Sense not available.
    [13561.767121] sd 4:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13561.767123] sd 4:0:0:0: [sde] Sense not available.
    [13561.767223] sd 4:0:0:0: [sde] Write Protect is on
    [13561.767226] sd 4:0:0:0: [sde] Mode Sense: 00 7e 4f e1
    [13561.767300] sd 4:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13561.767305] sd 4:0:0:0: [sde] Sense not available.
    [13561.767407] sd 4:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13561.767412] sd 4:0:0:0: [sde] Sense not available.
    [13561.839323] blk_update_request: I/O error, dev sde, sector 7814018944
    [13561.845407] sd 4:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13561.850634] sd 4:0:0:0: [sde] Sense not available.
    [13561.855552] sd 4:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13561.857845] sd 4:0:0:0: [sde] Sense not available.
    [13561.858865] zio pool=storage vdev=/dev/disk/by-id/wwn-0x5000c500b27f7291-part1 error=5 type=2 offset=2336522563584 size=4096 flags=180880
    [13561.858939] zio pool=storage vdev=/dev/disk/by-id/wwn-0x5000c500b27f7291-part1 error=5 type=2 offset=2578942078976 size=16384 flags=40080c80
    [13561.942955] sd 4:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13561.948749] sd 4:0:0:0: [sde] Sense not available.
    [13561.956531] sd 4:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13561.962294] sd 4:0:0:0: [sde] Sense not available.
    [13561.970630] sd 4:0:0:0: [sde] Write Protect is off
    [13561.975979] sd 4:0:0:0: [sde] Mode Sense: 2e 2e 2f 2e
    [13562.280549] sd 4:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13562.283602] sd 4:0:0:0: [sde] Sense not available.
    [13562.286855] sd 4:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13562.289411] sd 4:0:0:0: [sde] Sense not available.
    [13562.292547] sd 4:0:0:0: [sde] Write Protect is on
    [13562.294645] sd 4:0:0:0: [sde] Mode Sense: 40 41 9b ea
    [13562.316505] sd 4:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13562.321986] sd 4:0:0:0: [sde] Sense not available.
    [13562.329699] sd 4:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13562.334988] sd 4:0:0:0: [sde] Sense not available.
    [13562.340015] sd 4:0:0:0: [sde] Write Protect is off
    [13562.344783] sd 4:0:0:0: [sde] Mode Sense: 2e 2e 2f 2e
    [13562.349781] sd 4:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13562.354848] sd 4:0:0:0: [sde] Sense not available.
    [13562.359647] sd 4:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13562.364700] sd 4:0:0:0: [sde] Sense not available.
    [13562.369750] sd 4:0:0:0: [sde] Write Protect is on
    [13562.371633] sd 4:0:0:0: [sde] Mode Sense: ac 64 9c ce
    [13562.377498] sd 4:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13562.379765] sd 4:0:0:0: [sde] Sense not available.
    [13562.381761] sd 4:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13562.383965] sd 4:0:0:0: [sde] Sense not available.
    [13562.385942] sd 4:0:0:0: [sde] Write Protect is off
    [13562.387832] sd 4:0:0:0: [sde] Mode Sense: 00 c0 dd 56
    [13562.389949] sd 4:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13562.392326] sd 4:0:0:0: [sde] Sense not available.
    [13562.394865] sd 4:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=0x04 driverbyte=0x00
    [13562.397209] sd 4:0:0:0: [sde] Sense not available.
    

     

     

    Boot log (just in case):

    
    DDR Version 1.24 20191016
    In
    channel 0
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 1
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 0 training pass!
    channel 1 training pass!
    change freq to 416MHz 0,1
    Channel 0: LPDDR4,416MHz
    Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
    Channel 1: LPDDR4,416MHz
    Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
    256B stride
    channel 0
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 1
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 0 training pass!
    channel 1 training pass!
    channel 0, cs 0, advanced training done
    channel 1, cs 0, advanced training done
    change freq to 856MHz 1,0
    ch 0 ddrconfig = 0x101, ddrsize = 0x40
    ch 1 ddrconfig = 0x101, ddrsize = 0x40
    pmugrf_os_reg[2] = 0x32C1F2C1, stride = 0xD
    ddr_set_rate to 328MHZ
    ddr_set_rate to 666MHZ
    ddr_set_rate to 928MHZ
    channel 0, cs 0, advanced training done
    channel 1, cs 0, advanced training done
    ddr_set_rate to 416MHZ, ctl_index 0
    ddr_set_rate to 856MHZ, ctl_index 1
    support 416 856 328 666 928 MHz, current 856MHz
    OUT
    Boot1: 2019-03-14, version: 1.19
    CPUId = 0x0
    ChipType = 0x10, 254
    SdmmcInit=2 0
    BootCapSize=100000
    UserCapSize=14910MB
    FwPartOffset=2000 , 100000
    mmc0:cmd5,20
    SdmmcInit=0 0
    BootCapSize=0
    UserCapSize=30436MB
    FwPartOffset=2000 , 0
    StorageInit ok = 215428
    SecureMode = 0
    SecureInit read PBA: 0x4
    SecureInit read PBA: 0x404
    SecureInit read PBA: 0x804
    SecureInit read PBA: 0xc04
    SecureInit read PBA: 0x1004
    SecureInit read PBA: 0x1404
    SecureInit read PBA: 0x1804
    SecureInit read PBA: 0x1c04
    SecureInit ret = 0, SecureMode = 0
    atags_set_bootdev: ret:(0)
    GPT 0x3380ec0 signature is wrong
    recovery gpt...
    GPT 0x3380ec0 signature is wrong
    recovery gpt fail!
    LoadTrust Addr:0x4000
    No find bl30.bin
    No find bl32.bin
    Load uboot, ReadLba = 2000
    Load OK, addr=0x200000, size=0xded90
    RunBL31 0x40000
    NOTICE:  BL31: v1.3(debug):42583b6
    NOTICE:  BL31: Built : 07:55:13, Oct 15 2019
    NOTICE:  BL31: Rockchip release version: v1.1
    INFO:    GICv3 with legacy support detected. ARM GICV3 driver initialized in EL3
    INFO:    Using opteed sec cpu_context!
    INFO:    boot cpu mask: 0
    INFO:    plat_rockchip_pmu_init(1190): pd status 3e
    INFO:    BL31: Initializing runtime services
    WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
    ERROR:   Error initializing runtime service opteed_fast
    INFO:    BL31: Preparing for EL3 exit to normal world
    INFO:    Entry point address = 0x200000
    INFO:    SPSR = 0x3c9
    
    
    U-Boot 2020.07-armbian (Oct 05 2020 - 15:53:18 +0200)
    
    SoC: Rockchip rk3399
    Reset cause: POR
    DRAM:  3.9 GiB
    PMIC:  RK808 
    SF: Detected w25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
    MMC:   mmc@fe320000: 1, sdhci@fe330000: 0
    Loading Environment from MMC... unable to select a mode
    *** Warning - No block device, using default environment
    
    In:    serial
    Out:   serial
    Err:   serial
    Model: Helios64
    Revision: 1.2 - 4GB non ECC
    Net:   eth0: ethernet@fe300000
    scanning bus for devices...
    Hit any key to stop autoboot:  0 
    switch to partitions #0, OK
    mmc1 is current device
    Scanning mmc 1:1...
    Found U-Boot script /boot/boot.scr
    3185 bytes read in 5 ms (622.1 KiB/s)
    ## Executing script at 00500000
    Boot script loaded from mmc 1
    117 bytes read in 5 ms (22.5 KiB/s)
    8511739 bytes read in 365 ms (22.2 MiB/s)
    20703240 bytes read in 879 ms (22.5 MiB/s)
    101917 bytes read in 12 ms (8.1 MiB/s)
    2698 bytes read in 9 ms (292 KiB/s)
    Applying kernel provided DT fixup script (rockchip-fixup.scr)
    ## Executing script at 09000000
    ## Loading init Ramdisk from Legacy Image at 06000000 ...
       Image Name:   uInitrd
       Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
       Data Size:    8511675 Bytes = 8.1 MiB
       Load Address: 00000000
       Entry Point:  00000000
       Verifying Checksum ... OK
    ## Flattened Device Tree blob at 01f00000
       Booting using the fdt blob at 0x1f00000
       Loading Ramdisk to f56c6000, end f5ee40bb ... OK
    ERROR: reserving fdt memory region failed (addr=0 size=0)
       Loading Device Tree to 00000000f5644000, end 00000000f56c5fff ... OK
    
    Starting kernel ...
    

     

     

     

  17. 5 hours ago, lyuyhn said:

    To fellow zfs users, I just saw that the 0.8.5 release is out with many compatibility patches, we should wait until this version lands in the repos to use it on the Helios64 ;)

    Just tried building 0.8.5 on the Buster current image.  Unfortunately is still fails with the "__stack_chk_guard" errors.

    This topic and this GitHub issue (by the same author) have more info about the possible cause for this.

     

    ----------------------

     

    @aprayoga I will try to reproduce the crash on legacy again and post the results.

  18. 7 hours ago, Igor said:

    This one works "even less" (if that makes sense) since it's missing this patch and fails at an earlier stage in the compilation process. (With the patch applied it fails at the same point I mentioned in my previous post).

     

    Unrelated, but I should also note that today, after configuring my Helios64 with the legacy Armbian image and letting it run for a few hours with a moderate load, it crashed again with a kernel panic. Unfortunately I wasn't able to capture the panic messages as I had iftop open on the serial console at the time and it all mixed together beautifully.

  19. @IgorI had the same error with the package you posted as I had when manually compiling from source (which again, seems to be related to the GCC version):
     

      MODPOST /var/lib/dkms/zfs/0.8.4/build/module/Module.symvers
    ERROR: modpost: "__stack_chk_guard" [/var/lib/dkms/zfs/0.8.4/build/module/zfs/zfs.ko] undefined!
    ERROR: modpost: "__stack_chk_guard" [/var/lib/dkms/zfs/0.8.4/build/module/zcommon/zcommon.ko] undefined!
    ERROR: modpost: "__stack_chk_guard" [/var/lib/dkms/zfs/0.8.4/build/module/unicode/zunicode.ko] undefined!
    ERROR: modpost: "__stack_chk_guard" [/var/lib/dkms/zfs/0.8.4/build/module/spl/spl.ko] undefined!
    ERROR: modpost: "__stack_chk_guard" [/var/lib/dkms/zfs/0.8.4/build/module/nvpair/znvpair.ko] undefined!
    ERROR: modpost: "__stack_chk_guard" [/var/lib/dkms/zfs/0.8.4/build/module/lua/zlua.ko] undefined!
    ERROR: modpost: "__stack_chk_guard" [/var/lib/dkms/zfs/0.8.4/build/module/icp/icp.ko] undefined!
    ERROR: modpost: "__stack_chk_guard" [/var/lib/dkms/zfs/0.8.4/build/module/avl/zavl.ko] undefined!
    make[4]: *** [scripts/Makefile.modpost:111: /var/lib/dkms/zfs/0.8.4/build/module/Module.symvers] Error 1
    make[4]: *** Deleting file '/var/lib/dkms/zfs/0.8.4/build/module/Module.symvers'
    make[3]: *** [Makefile:1665: modules] Error 2
    make[3]: Leaving directory '/usr/src/linux-headers-5.8.13-rockchip64'
    make[2]: *** [Makefile:30: modules] Error 2
    make[2]: Leaving directory '/var/lib/dkms/zfs/0.8.4/build/module'
    make[1]: *** [Makefile:808: all-recursive] Error 1
    make[1]: Leaving directory '/var/lib/dkms/zfs/0.8.4/build'
    make: *** [Makefile:677: all] Error 2

     

    Just to test it, I've added the Debian testing repo to the Buster image and installed build-essential from there, which contains GCC 10.  With GCC10 your package installs successfully, but then the testing repo breaks OMV compatibility.

    Guess I'll just stay in legacy for the time being.

  20. 4 hours ago, Igor said:

     

    *** ZFS Version: zfs-0.8.3-1ubuntu12.4
    *** Compatible Kernels: 2.6.32 - 5.4

    For 5.8.y you need at least 0.8.4 from sourceshttps://github.com/openzfs/zfs/issues

     

    DKMS on Debian / Ubuntu is simply too old. Install attached .deb package (i tested this on Ubuntu Focal) and it will work:
     

      Reveal hidden contents
    
    
    sudo armbian-config main=Software selection=Headers_install
    dpkg -i zfs-dkms_0.8.4-1ubuntu11_all.deb 
    (Reading database ... 66592 files and directories currently installed.)
    Preparing to unpack zfs-dkms_0.8.4-1ubuntu11_all.deb ...
    
    ------------------------------
    Deleting module version: 0.8.3
    completely from the DKMS tree.
    ------------------------------
    Done.
    Unpacking zfs-dkms (0.8.4-1ubuntu11) over (0.8.3-1ubuntu12.4) ...
    Setting up zfs-dkms (0.8.4-1ubuntu11) ...
    Loading new zfs-0.8.4 DKMS files...
    Building for 5.8.13-rockchip64
    Building initial module for 5.8.13-rockchip64
    Done.
    
    zavl.ko:
    Running module version sanity check.
     - Original module
       - No original module exists within this kernel
     - Installation
       - Installing to /lib/modules/5.8.13-rockchip64/updates/dkms/
    
    znvpair.ko:
    Running module version sanity check.
     - Original module
       - No original module exists within this kernel
     - Installation
       - Installing to /lib/modules/5.8.13-rockchip64/updates/dkms/
    
    zunicode.ko:
    Running module version sanity check.
     - Original module
       - No original module exists within this kernel
     - Installation
       - Installing to /lib/modules/5.8.13-rockchip64/updates/dkms/
    
    zcommon.ko:
    Running module version sanity check.
     - Original module
       - No original module exists within this kernel
     - Installation
       - Installing to /lib/modules/5.8.13-rockchip64/updates/dkms/
    
    zfs.ko:
    Running module version sanity check.
     - Original module
       - No original module exists within this kernel
     - Installation
       - Installing to /lib/modules/5.8.13-rockchip64/updates/dkms/
    
    icp.ko:
    Running module version sanity check.
     - Original module
       - No original module exists within this kernel
     - Installation
       - Installing to /lib/modules/5.8.13-rockchip64/updates/dkms/
    
    zlua.ko:
    Running module version sanity check.
     - Original module
       - No original module exists within this kernel
     - Installation
       - Installing to /lib/modules/5.8.13-rockchip64/updates/dkms/
    
    spl.ko:
    Running module version sanity check.
     - Original module
       - No original module exists within this kernel
     - Installation
       - Installing to /lib/modules/5.8.13-rockchip64/updates/dkms/
    
    depmod.....
    
    DKMS: install completed.
    Processing triggers for initramfs-tools (0.136ubuntu6.3) ...
    update-initramfs: Generating /boot/initrd.img-5.8.13-rockchip64
    update-initramfs: Converting to u-boot format
    
    modprobe zfs
    [319920.214635] ZFS: Loaded module v0.8.4-1ubuntu11, ZFS pool version 5000, ZFS filesystem version 5

     

     

    zfs-dkms_0.8.4-1ubuntu11_all.deb 1.81 MB · 2 downloads

    Thank you. To be clear, I did try to compile the latest ZFS from source, but ran into some errors that my internet searches suggest have to do with the GCC version in Debian Buster. I was able to get it to build successfully with the Debian Bullseye image (which as far as I can tell uses GCC 10), but then there's no OMV...

    I'll give that package a shot, thanks again.

×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines