Helios64 - freeze whatever the kernel is.

[SR-G]

As stated in my comments here (but it seems no one is reading this blog post comments anymore ...) : https://blog.kobol.io/2020/10/27/helios64-software-issue/

 

- With 5.8.17 i have kernel panics.

- With 5.8.14 i have freeze.

 

So what to do in order to have a stable situation ?!

 

 

Quote

I installed armbian 20.08.21 with kernel 5.8.17 (and neither 5.8.14 nor 5.8.16 ), and i've just encountered the whole system being frozen (during a MDADM RAID5 array creation / after ~20h of rebuild ...), hence the need to power-down the NAS (first led was blinking red).

Linux helios64 5.8.17-rockchip64 #20.08.21 SMP PREEMPT Sat Oct 31 08:22:59 CET 2020 aarch64 GNU/Linux

I don't have any logs (as by default journalctl is persisted in ram) so i have no clue about what has happened (nothing in kern.log), but could this be linked to the issue described in this post ?
Or was this identified kernel crash only happening on 5.8.16 and is now corrected with 5.8.17 ?
So should i stay on 5.8.17, or is it adviced to revert to 5.8.14 ?

 

Quote

And today i just got (but without immediate freeze and without blinkind red light / but i still had to reboot after a few minutes due to weird behaviors, and even if ssh sessions were still online) (still with 5.8.17)

[Sat Nov 28 01:08:46 2020] Unable to handle kernel paging request at virtual address bfff800010c4132c
[Sat Nov 28 01:08:46 2020] Mem abort info:
[Sat Nov 28 01:08:46 2020] ESR = 0x86000004
[Sat Nov 28 01:08:46 2020] EC = 0x21: IABT (current EL), IL = 32 bits
[Sat Nov 28 01:08:46 2020] SET = 0, FnV = 0
[Sat Nov 28 01:08:46 2020] EA = 0, S1PTW = 0
[Sat Nov 28 01:08:46 2020] [bfff800010c4132c] address between user and kernel address ranges
[Sat Nov 28 01:08:46 2020] Internal error: Oops: 86000004 [#1] PREEMPT SMP
[Sat Nov 28 01:08:46 2020] Modules linked in: rfkill governor_performance snd_soc_hdmi_codec r8152 snd_soc_rockchip_i2s hantro_vpu(C) rockchip_rga rockchip_vdec(C) snd_soc_core rockchipdrm v4l2_h264 snd_pcm_dmaengine videobuf2_vmalloc videobuf2_dma_sg videobuf2_dma_contig snd_pcm dw_mipi_dsi v4l2_mem2mem dw_hdmi videobuf2_memops snd_timer analogix_dp videobuf2_v4l2 snd zstd panfrost videobuf2_common fusb302 pwm_fan leds_pwm drm_kms_helper soundcore gpio_charger tcpm videodev gpu_sched cec typec mc rc_core sg drm drm_panel_orientation_quirks cpufreq_dt gpio_beeper zram lm75 ip_tables x_tables autofs4 raid10 raid1 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx md_mod realtek dwmac_rk stmmac_platform stmmac mdio_xpcs adc_keys
[Sat Nov 28 01:08:46 2020] CPU: 4 PID: 7457 Comm: sshd Tainted: G C 5.9.10-rockchip64 #20.11
[Sat Nov 28 01:08:46 2020] Hardware name: Helios64 (DT)
[Sat Nov 28 01:08:46 2020] pstate: 00000005 (nzcv daif -PAN -UAO BTYPE=--)
[Sat Nov 28 01:08:46 2020] pc : 0xbfff800010c4132c
[Sat Nov 28 01:08:46 2020] lr : simple_copy_to_iter+0x34/0x68
[Sat Nov 28 01:08:46 2020] sp : ffff8000132eba30
[Sat Nov 28 01:08:46 2020] x29: ffff8000132eba30 x28: 0000000000000000
[Sat Nov 28 01:08:46 2020] x27: ffff000043d97d00 x26: 00000000000005a8
[Sat Nov 28 01:08:46 2020] x25: 0000000000001228 x24: 00000000000005a8
[Sat Nov 28 01:08:46 2020] x23: ffff8000132ebcc8 x22: 0000000000000001
[Sat Nov 28 01:08:46 2020] x21: ffff8000132ebcc8 x20: ffff0000a0aa0882
[Sat Nov 28 01:08:46 2020] x19: 00000000000005a8 x18: 0000000000000000
[Sat Nov 28 01:08:46 2020] x17: 0000000000000000 x16: 0000000000000000
[Sat Nov 28 01:08:46 2020] x15: 0000000000000000 x14: 34d9ba8f70dd5728
[Sat Nov 28 01:08:46 2020] x13: abbcee5b1190622f x12: a4a3849f1551ce47
[Sat Nov 28 01:08:46 2020] x11: 9ce64aa033a6ddcb x10: 0000000000000882
[Sat Nov 28 01:08:46 2020] x9 : 908fce41edbdd9d6 x8 : 00000000f7e00000
[Sat Nov 28 01:08:46 2020] x7 : 0000000000000018 x6 : ffff800011a84510
[Sat Nov 28 01:08:46 2020] x5 : ffff800011a84510 x4 : 0000000000000000
[Sat Nov 28 01:08:46 2020] x3 : ffff800010010000 x2 : ffff80000de00000
[Sat Nov 28 01:08:46 2020] x1 : ffff000002210000 x0 : ffff000003040000
[Sat Nov 28 01:08:46 2020] Call trace:
[Sat Nov 28 01:08:46 2020] 0xbfff800010c4132c
[Sat Nov 28 01:08:46 2020] __skb_datagram_iter+0x144/0x240
[Sat Nov 28 01:08:46 2020] skb_copy_datagram_iter+0x50/0x110
[Sat Nov 28 01:08:46 2020] tcp_recvmsg+0x590/0x950
[Sat Nov 28 01:08:46 2020] inet_recvmsg+0x50/0x120
[Sat Nov 28 01:08:46 2020] sock_recvmsg+0x4c/0x60
[Sat Nov 28 01:08:46 2020] sock_read_iter+0x88/0xe0
[Sat Nov 28 01:08:46 2020] new_sync_read+0x16c/0x180
[Sat Nov 28 01:08:46 2020] vfs_read+0x148/0x1d8
[Sat Nov 28 01:08:46 2020] ksys_read+0xe0/0xf8
[Sat Nov 28 01:08:46 2020] __arm64_sys_read+0x1c/0x28
[Sat Nov 28 01:08:46 2020] el0_svc_common.constprop.0+0x70/0x188
[Sat Nov 28 01:08:46 2020] do_el0_svc+0x24/0x90
[Sat Nov 28 01:08:46 2020] el0_sync_handler+0x90/0x198
[Sat Nov 28 01:08:46 2020] el0_sync+0x158/0x180
[Sat Nov 28 01:08:46 2020] Code: bad PC value
[Sat Nov 28 01:08:46 2020] ---[ end trace 82cecfd63ab60ea2 ]---

 

Quote

So i switched back to 5.8.14 ... and i also have a lot of issues and kernel panic !!!

 

> Linux helios64 5.8.14-rockchip64 #20.08.10 SMP PREEMPT Tue Oct 13 16:58:01 CEST 2020 aarch64 GNU/Linux
 

From today (board frozen + red light blinking + had to reboot :

Message from syslogd@localhost at Nov 29 14:48:57 ...
kernel:[16987.580304] Internal error: Oops: 96000044 [#1] PREEMPT SMP

Message from syslogd@localhost at Nov 29 14:48:57 ...
kernel:[16987.601169] Code: aa0103e0 f9400422 f85f8403 f9000462 (f9000043)

Message from syslogd@localhost at Nov 29 14:48:58 ...
kernel:[16987.780586] Internal error: Oops: 96000004 [#2] PREEMPT SMP

Message from syslogd@localhost at Nov 29 14:48:58 ...
kernel:[16987.801529] Code: f2fbd5a7 d0008408 f9400f01 aa0103e0 (f9400422)

Message from syslogd@localhost at Nov 29 14:48:58 ...
kernel:[16987.807055] Internal error: Oops: 96000004 [#3] PREEMPT SMP

Message from syslogd@localhost at Nov 29 14:48:58 ...
kernel:[16987.833634] Code: f2fbd5a7 d0008408 f9400f01 aa0103e0 (f9400422)

 

So what to do to have a stable situation ?
Is it only possible ?
With which kernel ?

 

It seems to be "under load" (one freeze during RAID array being built, and several ones (with .14 or .17 kernels) while files were being copied through the 1Gb/s network interface).

- System newly installed and running on fresh SSD card.

- 5*3.5" WD HDD plugged in / RAID5 mdadm array (so no M.2, ...)

- nothing else done on the NAS (nothing has been installed on the OS, no processes in memory outside raid + rsync or scp file copying through SSH)

[SR-G]

Same problem again (freeze + red LED blinking + had to force a power down as everything is frozen)..

Kernel is still 5.8.14

 

Linux helios64 5.8.14-rockchip64 #20.08.10 SMP PREEMPT Tue Oct 13 16:58:01 CEST 2020 aarch64 GNU/Linux

 

In currently opened console : 

 

Message from syslogd@localhost at Nov 29 18:15:51 ...
 kernel:[12092.786060] Internal error: Oops: 96000044 [#1] PREEMPT SMP

Message from syslogd@localhost at Nov 29 18:15:51 ...
 kernel:[12092.806913] Code: aa0103e0 f9400422 f85f8403 f9000462 (f9000043) 

Message from syslogd@localhost at Nov 29 18:45:01 ...
 kernel:[13842.826133] Internal error: Oops: 96000004 [#2] PREEMPT SMP

Message from syslogd@localhost at Nov 29 18:45:01 ...
 kernel:[13842.846908] Code: f2fbd5a7 d0008408 f9400f01 aa0103e0 (f9400422) 

 

dmesg -T : 

[Sun Nov 29 18:15:50 2020] Unable to handle kernel paging request at virtual address fffefe0000356788
[Sun Nov 29 18:15:50 2020] Mem abort info:
[Sun Nov 29 18:15:50 2020]   ESR = 0x96000044
[Sun Nov 29 18:15:50 2020]   EC = 0x25: DABT (current EL), IL = 32 bits
[Sun Nov 29 18:15:50 2020]   SET = 0, FnV = 0
[Sun Nov 29 18:15:50 2020]   EA = 0, S1PTW = 0
[Sun Nov 29 18:15:50 2020] Data abort info:
[Sun Nov 29 18:15:50 2020]   ISV = 0, ISS = 0x00000044
[Sun Nov 29 18:15:50 2020]   CM = 0, WnR = 1
[Sun Nov 29 18:15:50 2020] [fffefe0000356788] address between user and kernel address ranges
[Sun Nov 29 18:15:50 2020] Internal error: Oops: 96000044 [#1] PREEMPT SMP
[Sun Nov 29 18:15:50 2020] Modules linked in: rfkill governor_performance zstd snd_soc_hdmi_codec r8152 snd_soc_rockchip_i2s rockchipdrm hantro_vpu(C) rockchip_vdec(C) dw_mipi_dsi snd_soc_core rockchip_rga dw_hdmi v4l2_h264 snd_pcm_dmaengine videobuf2_dma_contig analogix_dp snd_pcm videobuf2_vmalloc v4l2_mem2mem videobuf2_dma_sg drm_kms_helper videobuf2_memops cec snd_timer videobuf2_v4l2 panfrost gpu_sched rc_core videobuf2_common snd gpio_charger leds_pwm pwm_fan videodev drm soundcore sg mc drm_panel_orientation_quirks fusb30x(C) gpio_beeper zram cpufreq_dt lm75 ip_tables x_tables autofs4 raid10 raid1 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx md_mod realtek dwmac_rk stmmac_platform stmmac mdio_xpcs adc_keys
[Sun Nov 29 18:15:50 2020] CPU: 4 PID: 3173 Comm: rsync Tainted: G         C        5.8.14-rockchip64 #20.08.10
[Sun Nov 29 18:15:50 2020] Hardware name: Helios64 (DT)
[Sun Nov 29 18:15:50 2020] pstate: 80000085 (Nzcv daIf -PAN -UAO BTYPE=--)
[Sun Nov 29 18:15:50 2020] pc : free_pcppages_bulk+0xfc/0x700
[Sun Nov 29 18:15:50 2020] lr : free_unref_page_commit+0x104/0x138
[Sun Nov 29 18:15:50 2020] sp : ffff800014d93b00
[Sun Nov 29 18:15:50 2020] x29: ffff800014d93b00 x28: dead000000000100 
[Sun Nov 29 18:15:50 2020] x27: 000000000000002b x26: 0000000000000014 
[Sun Nov 29 18:15:50 2020] x25: 0000000000000002 x24: ffff0000f77ae4e0 
[Sun Nov 29 18:15:50 2020] x23: ffff0000f77db100 x22: ffff0000f77ae4c0 
[Sun Nov 29 18:15:50 2020] x21: 0000000000000001 x20: ffff800014d93bb8 
[Sun Nov 29 18:15:50 2020] x19: ffff0000f77ae4f0 x18: 0000000000000000 
[Sun Nov 29 18:15:50 2020] x17: 0000000000000000 x16: 0000000000000000 
[Sun Nov 29 18:15:50 2020] x15: 0000000000000000 x14: d6c225a5ddaebbb0 
[Sun Nov 29 18:15:50 2020] x13: d75a64349d17fce9 x12: 619d061f648d8b9a 
[Sun Nov 29 18:15:50 2020] x11: 65ce21deeaf49f2c x10: 77d1a2095a672f3b 
[Sun Nov 29 18:15:50 2020] x9 : c7c5c42229bdccc4 x8 : ffff8000112ed000 
[Sun Nov 29 18:15:50 2020] x7 : dead000000000122 x6 : 0000000000000000 
[Sun Nov 29 18:15:50 2020] x5 : 000000000030f231 x4 : 0000000000000000 
[Sun Nov 29 18:15:50 2020] x3 : ffff0000f77ae4f0 x2 : fffefe0000356788 
[Sun Nov 29 18:15:50 2020] x1 : fffffe00006e3248 x0 : fffffe00006e3240 
[Sun Nov 29 18:15:50 2020] Call trace:
[Sun Nov 29 18:15:50 2020]  free_pcppages_bulk+0xfc/0x700
[Sun Nov 29 18:15:50 2020]  free_unref_page_commit+0x104/0x138
[Sun Nov 29 18:15:50 2020]  free_unref_page+0x5c/0x80
[Sun Nov 29 18:15:50 2020]  __put_page+0x54/0x68
[Sun Nov 29 18:15:50 2020]  anon_pipe_buf_release+0x7c/0x90
[Sun Nov 29 18:15:50 2020]  pipe_read+0x1dc/0x3c0
[Sun Nov 29 18:15:50 2020]  new_sync_read+0x16c/0x180
[Sun Nov 29 18:15:50 2020]  vfs_read+0x19c/0x1c0
[Sun Nov 29 18:15:50 2020]  ksys_read+0xe0/0xf8
[Sun Nov 29 18:15:50 2020]  __arm64_sys_read+0x1c/0x28
[Sun Nov 29 18:15:50 2020]  el0_svc_common.constprop.0+0x70/0x188
[Sun Nov 29 18:15:50 2020]  do_el0_svc+0x24/0x90
[Sun Nov 29 18:15:50 2020]  el0_sync_handler+0x90/0x198
[Sun Nov 29 18:15:50 2020]  el0_sync+0x158/0x180
[Sun Nov 29 18:15:50 2020] Code: aa0103e0 f9400422 f85f8403 f9000462 (f9000043) 
[Sun Nov 29 18:15:50 2020] ---[ end trace b0fe83763d7a5b16 ]---
[Sun Nov 29 18:45:00 2020] Unable to handle kernel paging request at virtual address fffefe0000356790
[Sun Nov 29 18:45:00 2020] Mem abort info:
[Sun Nov 29 18:45:00 2020]   ESR = 0x96000004
[Sun Nov 29 18:45:00 2020]   EC = 0x25: DABT (current EL), IL = 32 bits
[Sun Nov 29 18:45:00 2020]   SET = 0, FnV = 0
[Sun Nov 29 18:45:00 2020]   EA = 0, S1PTW = 0
[Sun Nov 29 18:45:00 2020] Data abort info:
[Sun Nov 29 18:45:00 2020]   ISV = 0, ISS = 0x00000004
[Sun Nov 29 18:45:00 2020]   CM = 0, WnR = 0
[Sun Nov 29 18:45:00 2020] [fffefe0000356790] address between user and kernel address ranges
[Sun Nov 29 18:45:00 2020] Internal error: Oops: 96000004 [#2] PREEMPT SMP
[Sun Nov 29 18:45:00 2020] Modules linked in: rfkill governor_performance zstd snd_soc_hdmi_codec r8152 snd_soc_rockchip_i2s rockchipdrm hantro_vpu(C) rockchip_vdec(C) dw_mipi_dsi snd_soc_core rockchip_rga dw_hdmi v4l2_h264 snd_pcm_dmaengine videobuf2_dma_contig analogix_dp snd_pcm videobuf2_vmalloc v4l2_mem2mem videobuf2_dma_sg drm_kms_helper videobuf2_memops cec snd_timer videobuf2_v4l2 panfrost gpu_sched rc_core videobuf2_common snd gpio_charger leds_pwm pwm_fan videodev drm soundcore sg mc drm_panel_orientation_quirks fusb30x(C) gpio_beeper zram cpufreq_dt lm75 ip_tables x_tables autofs4 raid10 raid1 raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx md_mod realtek dwmac_rk stmmac_platform stmmac mdio_xpcs adc_keys
[Sun Nov 29 18:45:00 2020] CPU: 4 PID: 202 Comm: kswapd0 Tainted: G      D  C        5.8.14-rockchip64 #20.08.10
[Sun Nov 29 18:45:00 2020] Hardware name: Helios64 (DT)
[Sun Nov 29 18:45:00 2020] pstate: 80000085 (Nzcv daIf -PAN -UAO BTYPE=--)
[Sun Nov 29 18:45:00 2020] pc : free_pcppages_bulk+0xf0/0x700
[Sun Nov 29 18:45:00 2020] lr : free_unref_page_commit+0x104/0x138
[Sun Nov 29 18:45:00 2020] sp : ffff800012843800
[Sun Nov 29 18:45:00 2020] x29: ffff800012843800 x28: dead000000000100 
[Sun Nov 29 18:45:00 2020] x27: 0000000000000001 x26: 000000000000003e 
[Sun Nov 29 18:45:00 2020] x25: 0000000000000002 x24: ffff0000f77ae4e0 
[Sun Nov 29 18:45:00 2020] x23: ffff0000f77db100 x22: ffff0000f77ae4c0 
[Sun Nov 29 18:45:00 2020] x21: 0000000000000001 x20: ffff8000128438b8 
[Sun Nov 29 18:45:00 2020] x19: ffff0000f77ae4f0 x18: 0000000000000000 
[Sun Nov 29 18:45:00 2020] x17: 0000000000000000 x16: 0000000000000000 
[Sun Nov 29 18:45:00 2020] x15: 0000000000000000 x14: 0000000000000000 
[Sun Nov 29 18:45:00 2020] x13: 0000000000000000 x12: 0000000000000000 
[Sun Nov 29 18:45:00 2020] x11: ffff8000128438e8 x10: 0000000000000002 
[Sun Nov 29 18:45:00 2020] x9 : 0000000000000001 x8 : ffff8000112ed000 
[Sun Nov 29 18:45:00 2020] x7 : dead000000000122 x6 : 0000000000000000 
[Sun Nov 29 18:45:00 2020] x5 : 000000000030f231 x4 : 0000000000000000 
[Sun Nov 29 18:45:00 2020] x3 : 0000000000000001 x2 : 000000000002b174 
[Sun Nov 29 18:45:00 2020] x1 : fffefe0000356788 x0 : fffefe0000356788 
[Sun Nov 29 18:45:00 2020] Call trace:
[Sun Nov 29 18:45:00 2020]  free_pcppages_bulk+0xf0/0x700
[Sun Nov 29 18:45:00 2020]  free_unref_page_commit+0x104/0x138
[Sun Nov 29 18:45:00 2020]  free_unref_page_list+0x13c/0x238
[Sun Nov 29 18:45:00 2020]  shrink_page_list+0x758/0xdb8
[Sun Nov 29 18:45:00 2020]  shrink_inactive_list+0x1c8/0x450
[Sun Nov 29 18:45:00 2020]  shrink_lruvec+0x320/0x510
[Sun Nov 29 18:45:00 2020]  shrink_node+0x184/0x668
[Sun Nov 29 18:45:00 2020]  balance_pgdat+0x308/0x5e8
[Sun Nov 29 18:45:00 2020]  kswapd+0x1dc/0x520
[Sun Nov 29 18:45:00 2020]  kthread+0x118/0x150
[Sun Nov 29 18:45:00 2020]  ret_from_fork+0x10/0x34
[Sun Nov 29 18:45:00 2020] Code: f2fbd5a7 d0008408 f9400f01 aa0103e0 (f9400422) 
[Sun Nov 29 18:45:00 2020] ---[ end trace b0fe83763d7a5b17 ]---

 

[grek]

Did You tried to jump armbian 20.11 and kernel 5.9. 10. ?

Currently Im using 5.9.10 begin friday without any issues. Before it 5.9.9? and i successfully copied more than 2 TB data. (zfs - raid1)

Also i ran docker and compiled few times ZFS from sources . The load was big but i don't see any issues. 

 

[SR-G]

Not yet ...

Seemed a bit early, and nothing is clearly said about the kernel panic problems being fixed or not ...

 

In the meanwhile i've seen this : https://armbian.atlassian.net/browse/AR-494

 

Quote

This is the reason for Helios64 instabilities...

 

... but it's still not very clear how/if this is related.

[Igor]

4 hours ago, SR-G said:

Seemed a bit early, and nothing is clearly said about the kernel panic problems being fixed or not ...

As @grek already pointed out - use / try with most recent version. I think that kernel freeze was just a temporal workaround and its not valid anymore.

[gprovost]

@SR-G Yes please try will newly available image of Armbian 20.11.1

Can find download link here https://www.armbian.com/helios64/ or https://wiki.kobol.io/download/

 

We will post a new blog post to announce the new build. The past issue was due to some bug in an hardware optimization script. However after we fixed that, there are still other instability reported here and there, and we are still trying to find the root cause.

 

Hopefully LK5.9 will provide a much better experience.

[djurny]

Hi,

I've also experienced almost hourly instabilities when running some load on my Helios64 box. Tried several kernels, each with their own Oops/BUG pattern. See below for an overview:

 

Spoiler

5.8.14-rockchip64 #20.08.10

[30241.791908] kernel BUG at arch/arm64/kernel/traps.c:470!
[30241.792383] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[30241.792870] Modules linked in: xfs dm_crypt algif_skcipher af_alg dm_mod governor_performance rfkill r8152 hantro_vpu(C) rockchip_rga rockchip_vdec(C) panfrost videobuf2_vmalloc v4l2_h264 gpu_sched videobuf2_dma_contig videobuf2_dma_sg leds_pwm v4l2_mem2mem drm pwm_fan videobuf2_memops drm_panel_orientation_quirks gpio_charger videobuf2_v4l2 videobuf2_common videodev mc sg gpio_beeper cpufreq_dt nfsd auth_rpcgss nfs_acl lockd grace sunrpc lm75 ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod uas realtek dwmac_rk stmmac_platform stmmac mdio_xpcs adc_keys
[30241.797910] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C        5.8.14-rockchip64 #20.08.10
[30241.798692] Hardware name: Helios64 (DT)
[30241.799046] pstate: 00000085 (nzcv daIf -PAN -UAO BTYPE=--)
[30241.799547] pc : do_undefinstr+0x2ec/0x310
[30241.799913] lr : do_undefinstr+0x1e0/0x310
[30241.800277] sp : ffff800011ad3d30
[30241.800574] x29: ffff800011ad3d30 x28: ffff0000f6ea6580
[30241.801048] x27: ffff0000f6ea6580 x26: ffff800011ad4000
[30241.801520] x25: ffff800011501d20 x24: 0000000000000000
[30241.801993] x23: 0000000060000085 x22: ffff800010df6118
[30241.802465] x21: ffff800011ad3ee0 x20: ffff0000f6ea6580
[30241.802937] x19: ffff800011ad3da0 x18: 00000000000000fb
[30241.803409] x17: 0000000000000000 x16: 0000000000000200
[30241.803882] x15: 0000000000000000 x14: 00000000000001bb
[30241.804355] x13: 0000000000000205 x12: 0000000000000268
[30241.804827] x11: 0000000000000001 x10: 0000000000000005
[30241.805298] x9 : ffff0000f77c9590 x8 : ffff0000f77c8bc0
[30241.805770] x7 : ffff0000180cac40 x6 : ffff800011ad3d88
[30241.806242] x5 : 00000000d5300000 x4 : ffff800011816118
[30241.806714] x3 : 00000000d5000000 x2 : 0000000000000002
[30241.807186] x1 : ffff0000f6ea6580 x0 : 0000000060000085
[30241.807658] Call trace:
[30241.807883]  do_undefinstr+0x2ec/0x310
[30241.808222]  el1_sync_handler+0x88/0x110
[30241.808574]  el1_sync+0x7c/0x100
[30241.808869]  rcu_dynticks_eqs_enter+0x30/0x70
[30241.809259]  rcu_nmi_exit+0x7c/0x250
[30241.809580]  rcu_irq_exit+0x10/0x20
[30241.809897]  irq_exit+0x4c/0xe0
[30241.810184]  handle_IPI+0x25c/0x3e8
[30241.810501]  gic_handle_irq+0x154/0x158
[30241.810844]  el1_irq+0xb8/0x180
[30241.811128]  arch_cpu_idle+0x28/0x218
[30241.811459]  default_idle_call+0x1c/0x44
[30241.811813]  do_idle+0x210/0x288
[30241.812104]  cpu_startup_entry+0x28/0x68
[30241.812457]  secondary_start_kernel+0x140/0x178
[30241.812865] Code: f9401bf7 17ffff7d a9025bf5 f9001bf7 (d4210000)

 

5.9.10-rockchip64 #20.11

[14692.347605] Unable to handle kernel paging request at virtual address ffdf8000119c4010
[14692.348319] Mem abort info:
[14692.348569]   ESR = 0x96000004
[14692.348843]   EC = 0x25: DABT (current EL), IL = 32 bits
[14692.349311]   SET = 0, FnV = 0
[14692.349582]   EA = 0, S1PTW = 0
[14692.349861] Data abort info:
[14692.350118]   ISV = 0, ISS = 0x00000004
[14692.350457]   CM = 0, WnR = 0
[14692.350722] [ffdf8000119c4010] address between user and kernel address ranges
[14692.351350] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[14692.351843] Modules linked in: xfs dm_crypt algif_skcipher af_alg dm_mod governor_performance rfkill
[14692.356998] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G         C        5.9.10-rockchip64 #20.11
[14692.357754] Hardware name: Helios64 (DT)
[14692.358105] pstate: 00000085 (nzcv daIf -PAN -UAO BTYPE=--)
[14692.358603] pc : ktime_get+0x3c/0xa0
[14692.358924] lr : tick_irq_enter+0x58/0x110
[14692.359287] sp : ffff800011b0bf00
[14692.359582] x29: ffff800011b0bf00 x28: ffff0000f6ead700
[14692.360053] x27: 0000000000000001 x26: 0000000000000000
[14692.360523] x25: ffff0000f6c04300 x24: ffff800011bd3df0
[14692.360992] x23: ffff800011532db0 x22: 00000000006aa93a
[14692.361461] x21: 00000d5cc0701cac x20: ffff800011a7c040
[14692.361930] x19: ffff0000f779af40 x18: 0000000000000000
[14692.362398] x17: 0000000000000000 x16: 0000000000000000
[14692.362867] x15: 000000000000001c x14: 000000000000001d
[14692.363336] x13: 00000000000000d6 x12: 00000000000000ea
[14692.363805] x11: 0000000000000001 x10: 0000000000000a30
[14692.364274] x9 : ffff800011bd3e70 x8 : ffff0000f6eae190
[14692.364743] x7 : 00000000ffffffff x6 : 000000522ff31dbf
[14692.365212] x5 : 00ffffffffffffff x4 : ffff0000f77a89a0
[14692.365680] x3 : ffff80001151f018 x2 : ffff800011849980
[14692.366149] x1 : ffff800011527f40 x0 : ffdf8000119c4010
[14692.366619] Call trace:
[14692.366840]  ktime_get+0x3c/0xa0
[14692.367129]  tick_irq_enter+0x58/0x110
[14692.367465]  irq_enter_rcu+0x44/0x60
[14692.367784]  irq_enter+0x14/0x20
[14692.368074]  __handle_domain_irq+0x70/0x108
[14692.368448]  gic_handle_irq+0x60/0x158
[14692.368783]  el1_irq+0xb8/0x180
[14692.369066]  arch_cpu_idle+0x14/0x20
[14692.369386]  do_idle+0x210/0x260
[14692.369674]  cpu_startup_entry+0x28/0x60
[14692.370024]  secondary_start_kernel+0x148/0x180
[14692.370430] Code: 37000316 d50339bf f9402680 f9403a95 (f9400001)

 

5.8.17-rockchip64 #20.08.21

15:42:37:  816/ 408MHz  1.65  12%   9%   1%   1%   0%   0% 43.9<C2><B0>C  0/5
15:42:42:  816/ 600MHz  1.60  12%   9%   1%   1%   0%   0% 43.9<C2><B0>C  0/5
15:42:47: 1200/ 600MHz  1.55  16%  11%   1%   2%   0%   0% 46.2<C2><B0>C  0/5
15:42:52:  408/ 408MHz  1.50  14%  11%   1%   1%   0%   0% 44.4<C2><B0>C  0/5
15:42:57: 1008/1200MHz  1.46  19%  13%   1%   2%   1%   0% 46.2<C2><B0>C  0/5[ 4561.947127] Unable to handle kernel paging request at virtual address ffdf800011989590
[ 4561.947840] Mem abort info:
[ 4561.948090]   ESR = 0x96000004
[ 4561.948364]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 4561.948833]   SET = 0, FnV = 0
[ 4561.949105]   EA = 0, S1PTW = 0
[ 4561.949385] Data abort info:
[ 4561.949642]   ISV = 0, ISS = 0x00000004
[ 4561.949981]   CM = 0, WnR = 0
[ 4561.950248] [ffdf800011989590] address between user and kernel address ranges
[ 4561.950877] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[ 4561.951370] Modules linked in: xfs dm_crypt algif_skcipher af_alg dm_mod governor_performance rfkill r8152 panfrost gpu_sched leds_pwm drm hantro_vpu(C) gpio_charger rockchip_vdec(C) pwm_fan drm_panel_orientation_quirks v4l2_h264 videobuf2_dma_contig rockchip_rga v4l2_mem2mem videobuf2_vmalloc videobuf2_dma_sg videobuf2_memops videobuf2_v4l2 videobuf2_common videodev mc sg gpio_beeper cpufreq_dt nfsd auth_rpcgss nfs_acl lockd lm75 grace sunrpc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod uas realtek dwmac_rk stmmac_platform stmmac mdio_xpcs adc_keys
[ 4561.956377] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G         C        5.8.17-rockchip64 #20.08.21
[ 4561.957155] Hardware name: Helios64 (DT)
[ 4561.957506] pstate: 00000085 (nzcv daIf -PAN -UAO BTYPE=--)
[ 4561.958010] pc : ktime_get+0x3c/0xa0
[ 4561.958331] lr : tick_sched_timer+0x38/0xb0
[ 4561.958703] sp : ffff800011adbdd0
[ 4561.959000] x29: ffff800011adbdd0 x28: ffff0000f779ba00
[ 4561.959473] x27: ffff0000f779bf08 x26: 0000000000000001
[ 4561.959946] x25: ffff8000114ee018 x24: 0000000000000080
[ 4561.960418] x23: ffff8000114ee018 x22: 00000000001e137e
[ 4561.960890] x21: 000004261f64e325 x20: ffff800011a40080
[ 4561.961361] x19: ffff0000f779bf08 x18: 0000000000000000
[ 4561.961833] x17: 0000000000000000 x16: 0000000000000000
[ 4561.962304] x15: 0000000000000000 x14: 0000000000000000
[ 4561.962776] x13: 000000000000033d x12: 000000000000038d
[ 4561.963247] x11: 0000000000000040 x10: ffff80001182e690
[ 4561.963719] x9 : ffff80001182e688 x8 : ffff0000f6800270
[ 4561.964191] x7 : 0000000000000000 x6 : 0000000000000000
[ 4561.964662] x5 : 0000000000000000 x4 : ffff0000f779bf08
[ 4561.965134] x3 : ffff8000114ee018 x2 : 0000000000010001
[ 4561.965605] x1 : ffff800011501d20 x0 : ffdf800011989590
[ 4561.966078] Call trace:
[ 4561.966303]  ktime_get+0x3c/0xa0
[ 4561.966594]  tick_sched_timer+0x38/0xb0
[ 4561.966939]  __hrtimer_run_queues+0x148/0x3b0
[ 4561.967329]  hrtimer_interrupt+0xf4/0x258
[ 4561.967691]  arch_timer_handler_phys+0x34/0x48
[ 4561.968091]  handle_percpu_devid_irq+0xa0/0x2b8
[ 4561.968496]  generic_handle_irq+0x30/0x48
[ 4561.968855]  __handle_domain_irq+0x94/0x108
[ 4561.969230]  gic_handle_irq+0x60/0x158
[ 4561.969568]  el1_irq+0xb8/0x180
[ 4561.969856]  arch_cpu_idle+0x28/0x218
[ 4561.970186]  default_idle_call+0x1c/0x44
[ 4561.970537]  do_idle+0x210/0x288
[ 4561.970825]  cpu_startup_entry+0x24/0x68
[ 4561.971177]  secondary_start_kernel+0x140/0x178
[ 4561.971583] Code: 37000316 d50339bf f9402680 f9403a95 (f9400001)
[ 4561.972129] ---[ end trace d6ab2e3c9eea65e5 ]---

 

5.9.11-rockchip64 #trunk.2

19:21:50: 1416/ 408MHz  1.78  32%  19%   1%   3%   7%   0% 46.9<C2><B0>C  0/5
19:21:55:  408/1008MHz  1.72  29%  17%   1%   3%   5%   0% 45.0<C2><B0>C  0/5
19:22:01:  408/ 600MHz  1.74  26%  16%   1%   2%   5%   0% 44.4<C2><B0>C  0/5
19:22:06:  600/1416MHz  1.68  26%  15%   1%   3%   5%   0% 44.4<C2><B0>C  0/5
19:22:11: 1008/ 408MHz  1.63  26%  15%   1%   2%   6%   0% 45.6<C2><B0>C  0/5
19:22:16: 1416/1200MHz  1.66  25%  15%   1%   2%   5%   0% 46.9<C2><B0>C  0/5[10141.409339] kernel BUG at arch/arm64/kernel/traps.c:470!
[10141.409827] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[10141.410314] Modules linked in: xfs dm_crypt algif_skcipher af_alg dm_mod governor_performance rfkill r8152 gpio_charger leds_pwm pwm_fan panfrost gpu_sched drm drm_panel_orientation_quirks hantro_vpu(C) rockchi
[10141.415490] CPU: 5 PID: 8086 Comm: kworker/5:0 Tainted: G         C        5.9.11-rockchip64 #trunk.2
[10141.416300] Hardware name: Helios64 (DT)
[10141.416664] Workqueue: events dbs_work_handler
[10141.417064] pstate: 00000085 (nzcv daIf -PAN -UAO BTYPE=--)
[10141.417562] pc : do_undefinstr+0x2ec/0x310
[10141.417928] lr : do_undefinstr+0x1e0/0x310
[10141.418293] sp : ffff80001f90b300
[10141.418590] x29: ffff80001f90b300 x28: ffff0000f6f5ab80
[10141.419063] x27: 00000000000000fa x26: 000000003b9aca00
[10141.419536] x25: 0000000001000001 x24: 0000000000000000
[10141.420008] x23: 0000000000000085 x22: ffff8000100221bc
[10141.420480] x21: ffff80001f90b4b0 x20: ffff0000f6f5ab80
[10141.420951] x19: ffff80001f90b370 x18: ffff0000a72cca00
[10141.421422] x17: 0000000000000000 x16: 0000000000000200
[10141.421893] x15: 000000000000000c x14: 0000003da66903a6
[10141.422365] x13: 000000000000020c x12: 0000000000000215
[10141.422836] x11: 0000000000000001 x10: 0000000000000a30
[10141.423307] x9 : ffff80001f90b440 x8 : ffff0000f6f5b610
[10141.423778] x7 : 0000000000000001 x6 : ffff80001f90b358
[10141.424249] x5 : 00000000d5300000 x4 : ffff800011856258
[10141.424720] x3 : 00000000d5000000 x2 : 0000000000000002
[10141.425191] x1 : ffff0000f6f5ab80 x0 : 0000000000000085
[10141.425664] Call trace:
[10141.425887]  do_undefinstr+0x2ec/0x310
[10141.426224]  el1_sync_handler+0x88/0x110
[10141.426574]  el1_sync+0x7c/0x100
[10141.426865]  save_return_addr+0x4/0x38
[10141.427201]  return_address+0x58/0x90
[10141.427534]  preempt_count_add+0xb8/0x158
[10141.427896]  _raw_spin_lock_irqsave+0x28/0xa0
[10141.428286]  prepare_to_wait_event+0x24/0xe8
[10141.428671]  rk3x_i2c_xfer_common.isra.0+0x3b8/0x480
[10141.429114]  rk3x_i2c_xfer+0x18/0x50
[10141.429434]  __i2c_transfer+0x144/0x650
[10141.429778]  i2c_transfer+0x60/0x128
[10141.430098]  i2c_transfer_buffer_flags+0x5c/0x88
[10141.430513]  regmap_i2c_write+0x20/0x58
[10141.430856]  _regmap_raw_write_impl+0x6f8/0x8c0
[10141.431259]  _regmap_bus_raw_write+0x68/0x88
[10141.431640]  _regmap_write+0x6c/0x160
[10141.431968]  _regmap_update_bits+0xf8/0x110
[10141.432341]  regmap_update_bits_base+0x64/0x98
[10141.432740]  regulator_set_voltage_sel_regmap+0x4c/0x98
[10141.433204]  _regulator_call_set_voltage_sel+0x78/0xd0
[10141.433660]  _regulator_do_set_voltage+0x474/0x5f0
[10141.434087]  regulator_set_voltage_rdev+0xac/0x230
[10141.434513]  regulator_do_balance_voltage+0x280/0x420
[10141.434963]  regulator_balance_voltage+0x50/0x90
[10141.435374]  regulator_set_voltage_unlocked+0x94/0x118
[10141.435830]  regulator_set_voltage+0x50/0x90
[10141.436214]  _set_opp_voltage+0x44/0x110
[10141.436566]  dev_pm_opp_set_rate+0x250/0x618
[10141.436956]  set_target+0x40/0x88 [cpufreq_dt]
[10141.437353]  __cpufreq_driver_target+0x1ec/0x668
[10141.437765]  cs_dbs_update+0x158/0x178
[10141.438102]  dbs_work_handler+0x40/0x80
[10141.438447]  process_one_work+0x1c4/0x470
[10141.438804]  worker_thread+0x4c/0x420
[10141.439134]  kthread+0x118/0x150
[10141.439426]  ret_from_fork+0x10/0x34
[10141.439749] Code: f9401bf7 17ffff7d a9025bf5 f9001bf7 (d4210000)
[10141.440291] ---[ end trace 1ab64c120ad3020b ]---
[10141.440704] note: kworker/5:0[8086] exited with preempt_count 2
[10141.441551] ------------[ cut here ]------------
[10141.441981] WARNING: CPU: 5 PID: 0 at kernel/rcu/tree.c:630 rcu_eqs_enter.isra.0+0x150/0x168
[10141.442725] Modules linked in: xfs dm_crypt algif_skcipher af_alg dm_mod governor_performance rfkill r8152 gpio_charger leds_pwm pwm_fan panfrost gpu_sched drm drm_panel_orientation_quirks hantro_vpu(C) rockchi
[10141.447877] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G      D  C        5.9.11-rockchip64 #trunk.2
[10141.448650] Hardware name: Helios64 (DT)
[10141.449003] pstate: 200003c5 (nzCv DAIF -PAN -UAO BTYPE=--)
[10141.449500] pc : rcu_eqs_enter.isra.0+0x150/0x168
[10141.449920] lr : rcu_eqs_enter.isra.0+0x1c/0x168
[10141.450330] sp : ffff800011bdbee0
[10141.450627] x29: ffff800011bdbee0 x28: 0000000000000000
[10141.451099] x27: ffff0000f6eae580 x26: 0000000000000000
[10141.451571] x25: 0000000000000000 x24: ffff80001129e970
[10141.452043] x23: ffff0000f6eae580 x22: ffff800011533838
[10141.452514] x21: ffff80001184a2a0 x20: ffff800011849980
[10141.452986] x19: ffff8000115358c0 x18: 000000000000000e
[10141.453457] x17: 0000000000000001 x16: 0000000000000019
[10141.453928] x15: 0000000000000004 x14: 000000000000004c
[10141.454399] x13: 00000000000001f7 x12: 000000000000022a
[10141.454870] x11: 0000000000000001 x10: 0000000000000a30
[10141.455342] x9 : ffff800011bdbe70 x8 : ffff0000f6eaf010
[10141.455814] x7 : 00000000ffffffff x6 : 00000038c1df3765
[10141.456285] x5 : 00ffffffffffffff x4 : 00000000004d49de
[10141.456756] x3 : ffff80001151f018 x2 : 4000000000000000
[10141.457227] x1 : 4000000000000002 x0 : ffff0000f77c88c0
[10141.457699] Call trace:
[10141.457923]  rcu_eqs_enter.isra.0+0x150/0x168
[10141.458315]  rcu_idle_enter+0x10/0x20
[10141.458644]  default_idle_call+0x2c/0x1b8
[10141.459004]  do_idle+0x210/0x260
[10141.459295]  cpu_startup_entry+0x24/0x60
[10141.459649]  secondary_start_kernel

 

 

It's not exhaustive; in the end I did the following and the box is now running some load (snapraid scrub on ~12TiB of data) without any issue:

• Enabled daily built kernel, now running Linux kobol0 5.9.11-rockchip64 #trunk.2 SMP PREEMPT Sun Nov 29 00:29:16 CET 2020 aarch64 GNU/Linux.
  Why: Every kernel had their own pattern, either do_undefinstr or XHCI hangup or page fault. Assumed latest greatest has most fixes.
• Enabled the i2c dtb overlays.
  Why: Some of the kernels showed some IRQ related to i2c in the Oops/BUG. Thought I find something in the dtb related to i2c and just enable it to see if that might fix something.
• Moved rootfs from USB stick to SATA SSD in slot4.
  Why: Some of the kernels had a repeated hanging XHCI controller, so I tried to remove some USB devices from the controller, to see if the amount of load on the controller itself might be a vector (, Victor).
• Also removed tlp and set SATA link power management to max_performance (hat tip @gprovost).

It's a weak investigation, as I fiddled with multiple things at once, trying to get things going quickly (I do not have much spare time to spend on this as I would like to). Still, perhaps this will trigger someone or give some more angles to fiddle with for others.

Fingers crossed.

 

Looking good so far:

djurny@kobol0:~$ uname -a
Linux kobol0 5.9.11-rockchip64 #trunk.2 SMP PREEMPT Sun Nov 29 00:29:16 CET 2020 aarch64 GNU/Linux
djurny@kobol0:~$ uptime
 07:26:58 up 2 days, 10:40,  7 users,  load average: 1.73, 1.76, 1.74
djurny@kobol0:~$ 

(The box has been running rdfind, xfs_fsr, snapraid scrub & check for the last 2 days (in that order).)

 

Groetjes,

Edited December 3, 2020 by djurny
Updated current status after 2 days of uptime

[SR-G]

Yes i'm running 5.9.11 since nearly two days and i can confirm that for now it seems way more stable than 5.8.14 / 5.8.17 (these two ones were constantly generating freezes). And without any changes like i2c overlays and still on SD card.

 

The amelioration is probably coming from the resolution of the issue i referenced a couple of posts below : https://armbian.atlassian.net/browse/AR-494

[djurny]

Hi,

A short update, unfortunately kernel has crashed again, but after a couple of days. So there is improvement :-)

No serial console output, as the usb-serial connection on my Pi stopped responding (will open another thread on this, not really Helios64 related though).

Will restart some loading and try a different usb-serial setup, hopefully both will not crash (that often) anymore.

Groetjes,

[djurny]

Hi,

Also here kernel Oops after some load:

[82182.500900] Unable to handle kernel paging request at virtual address ffff800011b14000
[..]
[82182.505753] Internal error: Oops: 96000007 [#1] PREEMPT SMP
[..]
[82182.526948] Call trace:
[82182.527414] x1 : ffff800011532db0 x0 : 00000000ffffffea 
[82182.527921]  gic_handle_irq+0x124/0x158
[82182.528384] Call trace:
[82182.528883]  el1_irq+0xb8/0x180
[82182.529350]  __handle_domain_irq+0xc4/0x108
[82182.529593]  arch_cpu_idle+0x14/0x20
[82182.530058] Code: f822683a a94153f3 a9425bf5 a94363f7 (a9446bf9) 
[82182.530425]  do_idle+0x210/0x260
[82182.530640] ---[ end trace c165b2007f1cb8d2 ]---
[82182.530946]  cpu_startup_entry+0x28/0x60
[82182.531312] Kernel panic - not syncing: Attempted to kill the idle task!
[82182.531657]  rest_init+0xd8/0xe8
[82182.532193] SMP: stopping secondary CPUs
[82182.532503]  arch_call_rest_init+0x10/0x1c
[82182.534913]  start_kernel+0x80c/0x848
[82182.535258] ---[ end trace c165b2007f1cb8d3 ]---
[82182.535692] Kernel Offset: disabled
[82182.536009] CPU features: 0x0240022,2000200c
[82182.536388] Memory Limit: none
[82182.536675] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

 

It appears that the system had been idling for some hours before the page fault occurred.

 

Is there anything I can collect or try to see if this will improve? This oops seems unrelated to system load.

 

Thanks,

Groetjes,

 

Spoiler

[82182.500900] Unable to handle kernel paging request at virtual address ffff800011b14000
[82182.500913] Mem abort info:
[82182.501902]   ESR = 0x96000007
[82182.501910]   EC = 0x25: DABT (current EL), IL = 32 bits
[82182.502669]   SET = 0, FnV = 0
[82182.502676]   EA = 0, S1PTW = 0
[82182.503246] Data abort info:
[82182.503253]   ISV = 0, ISS = 0x00000007
[82182.503869]   CM = 0, WnR = 0
[82182.503878] swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000000361e000
[82182.504750] [ffff800011b14000] pgd=00000000f7fff003, p4d=00000000f7fff003, pud=00000000f7ffe003, pmd=00000000f7ffa003
[82182.505750] pc : gic_handle_irq+0x124/0x158
[82182.505753] Internal error: Oops: 96000007 [#1] PREEMPT SMP
[82182.505760] Modules linked in: xfs dm_crypt algif_skcipher af_alg dm_mod governor_performance rfkill r8152 leds_pwm panfrost gpio_charger pwm_fan gpu_sched drm drm_panel_orientation_quirks rockchip_vdec(C) hantro_vpu(C) rockchip_rga v4l2_h264 videobuf2_dma_contig fusb302 videobuf2_vmalloc v4l2_mem2mem videobuf2_dma_sg tcpm videobuf2_memops typec videobuf2_v4l2 videobuf2_common videodev mc sg gpio_beeper cpufreq_dt nfsd auth_rpcgss nfs_acl lockd grace lm75 sunrpc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod uas realtek dwmac_rk stmmac_platform stmmac mdio_xpcs adc_keys
[82182.506153] lr : gic_handle_irq+0x124/0x158
[82182.506652] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C        5.9.11-rockchip64 #trunk.2
[82182.511858] sp : ffff800011aebfe0
[82182.512228] Hardware name: Helios64 (DT)
[82182.512237] pstate: 60000085 (nZCv daIf -PAN -UAO BTYPE=--)
[82182.513033] x29: ffff800011aebfe0 x28: ffff800011853a80 
[82182.513341] pc : __handle_domain_irq+0xc4/0x108
[82182.513350] lr : __handle_domain_irq+0xa8/0x108
[82182.513716] x27: ffff800011853a80 x26: ffff800011aec000 
[82182.514206] sp : ffff800011b13fc0
[82182.514214] x29: ffff800011b13f80 x28: ffff0000f6eae580 
[82182.514706] x25: ffff800011ae8000 x24: ffff80001129e970 
[82182.515105] x27: 0000000000000001 x26: ffff8000100170c4 
[82182.515536] x23: 0000000040000005 x22: ffff8000100170c4 
[82182.516002] x25: ffff800011bdbf30 x24: 0000ffffffffffff 
[82182.516326] x21: ffff800011843ea0 x20: 000000000000001e 
[82182.516792] x23: ffff800011bdbdf0 x22: ffff800010011b78 
[82182.517291] x19: 000000000000001e x18: 0000000000000010 
[82182.517757] x21: ffff800011bdbf20 x20: 0000000000000000 
[82182.518256] x17: 0000000000000000 x16: 0000000000000000 
[82182.518723] x19: ffff0000f6eae580 x18: 0000000000000001 
[82182.519223] x15: ffff800011853f28 x14: 0720072007200720 
[82182.519690] x17: 0000000000000000 x16: 0000000000000000 
[82182.520188] x13: 0720072007200720 x12: 0720072007200720 
[82182.520655] x15: ffff0000f6eaea28 x14: ffffffffffffffff 
[82182.521154] x11: 0720072007200720 x10: 0720072007200720 
[82182.521620] x13: ffff000016e1dd12 x12: ffff000016e1dd08 
[82182.522120] x9 : 0720072007200720 x8 : 0720072007200720 
[82182.522586] x11: 0000000005f5e100 x10: abcc77118461cefd 
[82182.523085] x7 : 0720072007200720 x6 : ffff0000f6e10f00 
[82182.523551] x9 : 0000000000000002 x8 : 0000000000000002 
[82182.524050] x5 : 0000000000000001 x4 : ffff0000f77141d8 
[82182.524516] x7 : 0000000000000064 x6 : 000001cb50ff49e6 
[82182.525016] x3 : 0000000000000000 x2 : 0000000000000002 
[82182.525483] x5 : 00ffffffffffffff x4 : 0026641396c6f927 
[82182.525981] x1 : ff5f2a3a499be400 x0 : 0000000000000000 
[82182.526448] x3 : ffff80001151f018 x2 : ffff8000e6293000 
[82182.526948] Call trace:
[82182.527414] x1 : ffff800011532db0 x0 : 00000000ffffffea 
[82182.527921]  gic_handle_irq+0x124/0x158
[82182.528384] Call trace:
[82182.528883]  el1_irq+0xb8/0x180
[82182.529350]  __handle_domain_irq+0xc4/0x108
[82182.529593]  arch_cpu_idle+0x14/0x20
[82182.530058] Code: f822683a a94153f3 a9425bf5 a94363f7 (a9446bf9) 
[82182.530425]  do_idle+0x210/0x260
[82182.530640] ---[ end trace c165b2007f1cb8d2 ]---
[82182.530946]  cpu_startup_entry+0x28/0x60
[82182.531312] Kernel panic - not syncing: Attempted to kill the idle task!
[82182.531657]  rest_init+0xd8/0xe8
[82182.532193] SMP: stopping secondary CPUs
[82182.532503]  arch_call_rest_init+0x10/0x1c
[82182.534913]  start_kernel+0x80c/0x848
[82182.535258] ---[ end trace c165b2007f1cb8d3 ]---
[82182.535692] Kernel Offset: disabled
[82182.536009] CPU features: 0x0240022,2000200c
[82182.536388] Memory Limit: none
[82182.536675] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

 

 

 

Spoiler

00:17:32:  408/ 600MHz  2.25  29%   6%   0%   0%  21%   0% 41.7<C2><B0>C  0/5
00:17:38:  408/ 408MHz  2.15  23%  10%   0%   0%  10%   0% 41.7<C2><B0>C  0/5
00:17:43: 1200/1008MHz  2.22  26%   9%   1%   0%  14%   0% 42.2<C2><B0>C  0/5
[..] ## minutes pass
00:18:46:  408/ 408MHz  2.21  27%   6%   0%   1%  18%   0% 42.8<C2><B0>C  0/5
00:18:51:  408/ 600MHz  2.19  28%   7%   0%   0%  18%   0% 42.2<C2><B0>C  0/5
00:18:57: 1800/ 408MHz  2.33  29%  14%   0%   2%  11%   0% 44.4<C2><B0>C  0/5
00:19:04:  408/ 600MHz  2.31  35%   6%   0%   0%  27%   0% 42.8<C2><B0>C  0/5
00:19:12:  600/ 600MHz  2.49  39%   8%   0%   0%  29%   0% 42.8<C2><B0>C  0/5
00:19:18:  408/ 600MHz  2.37  25%   7%   0%   1%  15%   0% 42.2<C2><B0>C  0/5
00:19:23: 1800/ 600MHz  2.42  22%  10%   0%   0%  10%   0% 50.6<C2><B0>C  0/5
Time       big.LITTLE   load %cpu %sys %usr %nice %io %irq   CPU  C.St.
00:19:28:  408/ 408MHz  2.31  12%   9%   0%   1%   0%   0% 42.8<C2><B0>C  0/5
00:19:33:  600/ 408MHz  2.13   4%   2%   0%   1%   0%   0% 42.2<C2><B0>C  0/5
00:19:38:  408/ 408MHz  1.95   2%   1%   0%   0%   0%   0% 41.7<C2><B0>C  0/5
00:19:44:  408/ 408MHz  1.96   2%   1%   0%   0%   0%   0% 42.2<C2><B0>C  0/5
00:19:49:  408/ 408MHz  1.80   2%   1%   0%   0%   0%   0% 41.7<C2><B0>C  0/5
00:19:54:  408/ 408MHz  1.60   2%   1%   0%   0%   0%   0% 41.1<C2><B0>C  0/5
00:19:59:  408/1416MHz  1.71   6%   4%   0%   0%   0%   0% 41.7<C2><B0>C  0/5
00:20:04:  408/ 408MHz  1.57   5%   3%   0%   1%   0%   0% 41.7<C2><B0>C  0/5
00:20:09:  408/ 408MHz  1.45   2%   1%   0%   0%   0%   0% 41.1<C2><B0>C  0/5
00:20:15:  408/ 408MHz  1.33   2%   1%   0%   0%   0%   0% 41.1<C2><B0>C  0/5
00:20:20:  408/ 408MHz  1.30   3%   2%   0%   0%   0%   0% 41.1<C2><B0>C  0/5
00:20:25:  408/ 408MHz  1.20   2%   1%   0%   0%   0%   0% 41.1<C2><B0>C  0/5
00:20:30:  600/ 408MHz  1.18   2%   2%   0%   0%   0%   0% 40.6<C2><B0>C  0/5
00:20:35:  408/ 408MHz  1.09   8%   5%   0%   2%   0%   0% 41.1<C2><B0>C  0/5
00:20:41:  816/ 408MHz  1.00   4%   2%   0%   0%   0%   0% 41.1<C2><B0>C  0/5
Time       big.LITTLE   load %cpu %sys %usr %nice %io %irq   CPU  C.St.
00:20:46:  408/ 408MHz  0.92   2%   1%   0%   0%   0%   0% 41.1<C2><B0>C  0/5
00:20:51:  408/ 408MHz  0.85   2%   1%   0%   0%   0%   0% 40.6<C2><B0>C  0/5
00:20:56:  600/ 600MHz  0.78   3%   2%   0%   0%   0%   0% 41.1<C2><B0>C  0/5
00:21:01:  408/ 408MHz  0.72   2%   1%   0%   0%   0%   0% 41.1<C2><B0>C  0/5
00:21:07:  408/ 408MHz  0.66   8%   5%   0%   2%   0%   0% 41.1<C2><B0>C  0/5
00:21:12:  600/ 408MHz  0.61   3%   1%   0%   0%   0%   0% 41.1<C2><B0>C  0/5
[..] ## hours pass
06:48:06:  408/ 408MHz  0.25   2%   1%   0%   0%   0%   0% 41.1<C2><B0>C  0/5
06:48:11:  408/ 408MHz  0.23   3%   2%   0%   0%   0%   0% 41.7<C2><B0>C  0/5
06:48:17:  408/ 600MHz  0.21   3%   2%   1%   0%   0%   0% 41.1<C2><B0>C  0/5
06:48:22:  408/1008MHz  0.20   3%   2%   0%   0%   0%   0% 41.7<C2><B0>C  0/5
06:48:27:  408/ 600MHz  0.18   8%   5%   0%   1%   0%   0% 41.1<C2><B0>C  0/5
[82182.500900] Unable to handle kernel paging request at virtual address ffff800011b14000

 

 

[gprovost]

@djurny Thanks for the feedback. We will let you know.

 

We are working with Rockchip to get to the root cause of this instability. We are still suspecting wrong DDR timing settings.

[djurny]

Hi,

Short update. After looking at the most recent kernel freezes I experienced on 5.9.13-rockchip64 #trunk.16 and reading through a thread meant for Helios4, I decided to stop using cpufreq governor conservative. Switched to either 'powersave' or 'performance' depending on workload, but not to have the frequency changed on-the-fly. Box has been running smooth since last power cycle.

 

As system is nice and stable now, will do some more maintenance before upgrading to latest advised configuration.

 

Groetjes,

 

See last Oops, mentioning something about trying to set some regulator voltage, triggered by cpufreq-dt module: (Note that system did not fully freeze, parts of the sytem continued service).

Spoiler

[73401.454765] Unable to handle kernel paging request at virtual address ffff80001757f838
[73401.455477] Mem abort info:
[73401.455729]   ESR = 0x96000007
[73401.456005]   EC = 0x25: DABT (current EL), IL = 32 bits
[73401.456476]   SET = 0, FnV = 0
[73401.456750]   EA = 0, S1PTW = 0
[73401.457031] Data abort info:
[73401.457289]   ISV = 0, ISS = 0x00000007
[73401.457630]   CM = 0, WnR = 0
[73401.457898] swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000000361b000
[73401.458490] [ffff80001757f838] pgd=00000000f7fff003, p4d=00000000f7fff003, pud=00000000f7ffe003, pmd=00000000d5751003, pte=0000000000000000
[73401.459600] Internal error: Oops: 96000007 [#1] PREEMPT SMP
[73401.460095] Modules linked in: xfs dm_crypt algif_skcipher af_alg dm_mod governor_performance rfkill panfrost leds_pwm gpu_sched drm pwm_fan drm_panel_orientation_quirks gpio_charger fusb302 tcpm typec rockchip_vdec(C) hantro_vpu(C) rockchip_rga v4l2_h264 videobuf2_dma_contig v4l2_mem2mem videobuf2_vmalloc videobuf2_dma_sg videobuf2_memops videobuf2_v4l2 videobuf2_common videodev mc sg gpio_beeper cpufreq_dt nfsd auth_rpcgss nfs_acl lockd grace lm75 sunrpc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod uas realtek dwmac_rk stmmac_platform stmmac mdio_xpcs adc_keys
[73401.465226] CPU: 5 PID: 25414 Comm: kworker/5:1 Tainted: G         C        5.9.13-rockchip64 #trunk.16
[73401.466053] Hardware name: Helios64 (DT)
[73401.466421] Workqueue: events dbs_work_handler
[73401.466821] pstate: 80000085 (Nzcv daIf -PAN -UAO BTYPE=--)
[73401.467322] pc : save_stack_trace+0x3c/0x70
[73401.467698] lr : unwind_frame+0xc8/0x1a0
[73401.468047] sp : ffff80001557b3a0
[73401.468345] x29: ffff80001557b3a0 x28: ffff0000f5023880 
[73401.468818] x27: 00000000000000fa x26: 0000000001000001 
[73401.469290] x25: ffff80001557b4f0 x24: ffff800011849988 
[73401.469762] x23: ffff800011526780 x22: ffff0000dc0f4880 
[73401.470234] x21: ffff800010022178 x20: ffff80001557b3f0 
[73401.470705] x19: ffff80001557b400 x18: 0000000000000000 
[73401.471177] x17: 0000000000000000 x16: 0000000000000000 
[73401.471648] x15: 0000000000000001 x14: 000000000000001b 
[73401.472120] x13: 000000000000039e x12: 0000000000000002 
[73401.472592] x11: 0000000000000001 x10: 0000000000000a30 
[73401.473064] x9 : ffff80001557b440 x8 : ffff0000dc0f5310 
[73401.473535] x7 : ffff80001557b410 x6 : 0000000000000001 
[73401.474007] x5 : 0000000000000000 x4 : 0000000000000001 
[73401.474479] x3 : ffff80001757f838 x2 : ffff80001557b3d0 
[73401.474950] x1 : ffff80001557b400 x0 : 0000000000000000 
[73401.475423] Call trace:
[73401.475650]  save_stack_trace+0x3c/0x70
[73401.475996]  return_address+0x5c/0x90
[73401.476329]  preempt_count_add+0xb8/0x158
[73401.476690]  _raw_spin_lock_irqsave+0x28/0x2f8
[73401.477090]  lock_timer_base+0x68/0xd0
[73401.477426]  try_to_del_timer_sync+0x34/0x98
[73401.477809]  del_timer_sync+0x38/0x60
[73401.478136]  schedule_timeout+0x1a0/0x370
[73401.478500]  rk3x_i2c_xfer_common.isra.6+0x3dc/0x488
[73401.478942]  rk3x_i2c_xfer+0x18/0x50
[73401.479263]  __i2c_transfer+0x14c/0x670
[73401.479606]  i2c_transfer+0x60/0x128
[73401.479927]  i2c_transfer_buffer_flags+0x5c/0x88
[73401.480341]  regmap_i2c_write+0x20/0x58
[73401.480685]  _regmap_raw_write_impl+0x6fc/0x868
[73401.481088]  _regmap_bus_raw_write+0x68/0x88
[73401.481468]  _regmap_write+0x6c/0x160
[73401.481796]  _regmap_update_bits+0xf8/0x110
[73401.482170]  regmap_update_bits_base+0x64/0x98
[73401.482569]  regulator_set_voltage_sel_regmap+0x48/0x98
[73401.483034]  _regulator_call_set_voltage_sel+0x80/0xd0
[73401.483491]  _regulator_do_set_voltage+0x460/0x5e0
[73401.483919]  regulator_set_voltage_rdev+0xac/0x228
[73401.484346]  regulator_do_balance_voltage+0x29c/0x400
[73401.484796]  regulator_balance_voltage+0x50/0x90
[73401.485209]  regulator_set_voltage_unlocked+0x94/0x128
[73401.485666]  regulator_set_voltage+0x54/0x98
[73401.486051]  _set_opp_voltage+0x44/0x148
[73401.486403]  dev_pm_opp_set_rate+0x274/0x638
[73401.486792]  set_target+0x40/0x88 [cpufreq_dt]
[73401.487190]  __cpufreq_driver_target+0x2b0/0x698
[73401.487603]  cs_dbs_update+0xd8/0x160
[73401.487932]  dbs_work_handler+0x40/0x78
[73401.488279]  process_one_work+0x1ec/0x4b0
[73401.488636]  worker_thread+0x48/0x478
[73401.488966]  kthread+0x140/0x150
[73401.489258]  ret_from_fork+0x10/0x34
[73401.489583] Code: 521b0860 d53cd044 d53cd043 f9800071 (c85f7c65) 
[73401.490125] ---[ end trace f822551f2480a969 ]---
[73401.490538] note: kworker/5:1[25414] exited with preempt_count 1

 

 

After posting this, I configured cpufreq to use schedutil governor and after roughly 3 hours of load, it froze up with one of the other patterns observed before:

Spoiler

[362047.870810] Unable to handle kernel NULL pointer dereference at virtual address 00000000000005dd
[362047.871608] Mem abort info:
[362047.871864]   ESR = 0x96000004
[362047.872146]   EC = 0x25: DABT (current EL), IL = 32 bits
[362047.872622]   SET = 0, FnV = 0
[362047.872901]   EA = 0, S1PTW = 0
[362047.873187] Data abort info:
[362047.873452]   ISV = 0, ISS = 0x00000004
[362047.873800]   CM = 0, WnR = 0
[362047.874074] user pgtable: 4k pages, 48-bit VAs, pgdp=00000000e4faf000
[362047.874648] [00000000000005dd] pgd=0000000000000000, p4d=0000000000000000
[362047.875257] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[362047.875757] Modules linked in: xfs dm_crypt algif_skcipher af_alg dm_mod governor_performance rfkill leds_pwm pwm_fan gpio_charger panfrost gpu_sched drm drm_panel_orientation_quirks rockchip_rga hantro_vpu(C) rockchip_vdec(C) videobuf2_dma_sg v4l2_h264 videobuf2_dma_contig v4l2_mem2mem videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common fusb302 tcpm videodev typec mc sg gpio_beeper cpufreq_dt nfsd lm75 auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod uas realtek dwmac_rk stmmac_platform stmmac mdio_xpcs adc_keys
[362047.880874] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C        5.9.13-rockchip64 #trunk.16
[362047.881667] Hardware name: Helios64 (DT)
[362047.882032] pstate: 40000085 (nZcv daIf -PAN -UAO BTYPE=--)
[362047.882554] pc : preempt_count_add+0x10/0x158
[362047.882957] lr : efi_header_end+0x78/0x3f4
[362047.883332] sp : ffff800011b13e80
[362047.883638] x29: ffff800011b13e80 x28: ffff0000f6eae580
[362047.884124] x27: 0000000000000001 x26: ffff800011532db0
[362047.884609] x25: 0000000000000005 x24: 0000000000000000
[362047.885093] x23: ffff800011bdbdf0 x22: 0000000000000001
[362047.885576] x21: 0000000000000010 x20: ffff8000112a0518
[362047.886060] x19: ffff800011534580 x18: 0000000000000000
[362047.886543] x17: 0000000000000000 x16: 0000000000000000
[362047.887027] x15: 0000000000000006 x14: 0000000000000000
[362047.887509] x13: 00000000000000ed x12: 0000000000000000
[362047.887992] x11: 0000000000000001 x10: 0000000000000a30
[362047.888475] x9 : ffff800011bdbe80 x8 : ffff0000f6eaf010
[362047.888959] x7 : 0000000000436055 x6 : 000007e72f9f3d9c
[362047.889442] x5 : 00ffffffffffffff x4 : ffff8000e6293000
[362047.889925] x3 : 0000000000000001 x2 : ffff8000e6293000
[362047.890408] x1 : ffff800011a58000 x0 : 0000000000000100
[362047.890891] Call trace:
[362047.891129]  preempt_count_add+0x10/0x158
[362047.891500]  efi_header_end+0x78/0x3f4
[362047.891850]  irq_exit+0xb8/0xd8
[362047.892152]  handle_IPI+0x110/0x3c8
[362047.892489]  gic_handle_irq+0x168/0x170
[362047.892850]  el1_irq+0xb8/0x180
[362047.893155]  arch_cpu_idle+0x14/0x20
[362047.893498]  do_idle+0x1fc/0x270
[362047.893809]  cpu_startup_entry+0x24/0x60
[362047.894181]  secondary_start_kernel+0x16c/0x180
[362047.894613] Code: d503233f f000cc61 a9be7bfd 910003fd (b945d821)
[362047.895187] ---[ end trace e70654e451e90e24 ]---
[362047.895618] Kernel panic - not syncing: Attempted to kill the idle task!
[362047.896234] SMP: stopping secondary CPUs
[362047.896609] Kernel Offset: disabled
[362047.896938] CPU features: 0x0240022,2000200c
[362047.897333] Memory Limit: none
[362047.897645] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

 

 

Will change back to powersave and give it some load again.

 

Not sure if there is a correlation here.

 

Groetjes,

[gprovost]

Thanks for the update. Good to hear that with fix cpu frequency system is stable.

 

Yeah the main source of instability is DVFS, we already did some change by tweaking regulator ramp delay : https://github.com/armbian/build/commit/d18323fcc358ddd5bd0836c4853cbc281b6293c6

 

It improved a lot stability but unfortunately doesn't seem to completely solved the problem for everyone. So we are still investigating and getting Rockchip to look at it also.

[Z06Frank]

I just froze my system late last night. I cannot SSH or get into the web interface. I tried to let the battery backup discharge and reboot to no avail.  All lights on the the LEDs sit a solid blue. I had updated Armbian a few days ago to the latest stable build:  "5.9.14-rockchip64 #20.11.4 SMP PREEMPT Tue Dec 15 08:52:20 CET 2020 aarch64 GNU/Linux" to get the eth1 working again. I have latest OMV and Armbian running on the M2 slot. 

 

What I was doing was doing a full data transfer of my media via Ethernet and samba shares from a Windows 10 PC server setup. I let this go overnight and forgot I had hibernation set on the Windows PC. Woke up today to find the Helios 64 in a frozen state....and can't determine what's going on without help here. Thoughts on what to do besides a wipe and restart of the OS/OMV. I'm guess with a larger high speed transfer (i.e. running about 150mb/sec) that I ran into a stability issue and I can't figure out how to read the kernal panic/or logs. 

 

Any ideas what I can try?

[Z06Frank]

18 minutes ago, Z06Frank said:

I just froze my system late last night. I cannot SSH or get into the web interface. I tried to let the battery backup discharge and reboot to no avail.  All lights on the the LEDs sit a solid blue. I had updated Armbian a few days ago to the latest stable build:  "5.9.14-rockchip64 #20.11.4 SMP PREEMPT Tue Dec 15 08:52:20 CET 2020 aarch64 GNU/Linux" to get the eth1 working again. I have latest OMV and Armbian running on the M2 slot. 

 

What I was doing was doing a full data transfer of my media via Ethernet and samba shares from a Windows 10 PC server setup. I let this go overnight and forgot I had hibernation set on the Windows PC. Woke up today to find the Helios 64 in a frozen state....and can't determine what's going on without help here. Thoughts on what to do besides a wipe and restart of the OS/OMV. I'm guess with a larger high speed transfer (i.e. running about 150mb/sec) that I ran into a stability issue and I can't figure out how to read the kernal panic/or logs. 

 

Any ideas what I can try?

Note: I tried the basic reset button; the HHD's spin (and blue LED's flicker) for about 25 sec and it freezes with all blue LED's solid. 

 

[djurny]

3 hours ago, Z06Frank said:

Note: I tried the basic reset button; the HHD's spin (and blue LED's flicker) for about 25 sec and it freezes with all blue LED's solid. 

 

Hi,

Can you connect the USB-C cable, connect to the serial console (1.500.000 baud and no HW/SW flow control) and post the output after a reboot/powercycle? What's on the serial console should point to what is going wrong.

 

Did you by any change update the system and continue without rebooting? I had something similar happen on one of the nanopi R2S boxes, where the update was not successful, resulting in an incomplete /boot folder.

 

Groetjes,

[Z06Frank]

5 hours ago, djurny said:

Hi,

Can you connect the USB-C cable, connect to the serial console (1.500.000 baud and no HW/SW flow control) and post the output after a reboot/powercycle? What's on the serial console should point to what is going wrong.

 

Did you by any change update the system and continue without rebooting? I had something similar happen on one of the nanopi R2S boxes, where the update was not successful, resulting in an incomplete /boot folder.

 

Groetjes,

Here is the serial output......

 

DDR Version 1.24 20191016
In
channel 0
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x10
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 1
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x10
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 0 training pass!
channel 1 training pass!
change freq to 416MHz 0,1
Channel 0: LPDDR4,416MHz
Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
Channel 1: LPDDR4,416MHz
Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
256B stride
channel 0
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x10
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 1
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x10
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 0 training pass!
channel 1 training pass!
channel 0, cs 0, advanced training done
channel 1, cs 0, advanced training done
change freq to 856MHz 1,0
ch 0 ddrconfig = 0x101, ddrsize = 0x40
ch 1 ddrconfig = 0x101, ddrsize = 0x40
pmugrf_os_reg[2] = 0x32C1F2C1, stride = 0xD
ddr_set_rate to 328MHZ
ddr_set_rate to 666MHZ
ddr_set_rate to 928MHZ
channel 0, cs 0, advanced training done
channel 1, cs 0, advanced training done
ddr_set_rate to 416MHZ, ctl_index 0
ddr_set_rate to 856MHZ, ctl_index 1
support 416 856 328 666 928 MHz, current 856MHz
OUT
Boot1: 2019-03-14, version: 1.19
CPUId = 0x0
ChipType = 0x10, 253
SdmmcInit=2 0
BootCapSize=100000
UserCapSize=14910MB
FwPartOffset=2000 , 100000
mmc0:cmd8,20
mmc0:cmd5,20
mmc0:cmd55,20
mmc0:cmd1,20
mmc0:cmd8,20
mmc0:cmd5,20
mmc0:cmd55,20
mmc0:cmd1,20
mmc0:cmd8,20
mmc0:cmd5,20
mmc0:cmd55,20
mmc0:cmd1,20
SdmmcInit=0 1
StorageInit ok = 67768
SecureMode = 0
SecureInit read PBA: 0x4
SecureInit read PBA: 0x404
SecureInit read PBA: 0x804
SecureInit read PBA: 0xc04
SecureInit read PBA: 0x1004
SecureInit read PBA: 0x1404
SecureInit read PBA: 0x1804
SecureInit read PBA: 0x1c04
SecureInit ret = 0, SecureMode = 0
atags_set_bootdev: ret:(0)
GPT 0x3380ec0 signature is wrong
recovery gpt...
GPT 0x3380ec0 signature is wrong
recovery gpt fail!
LoadTrust Addr:0x4000
No find bl30.bin
No find bl32.bin
Load uboot, ReadLba = 2000
Load OK, addr=0x200000, size=0xdd6b0
RunBL31 0x40000
NOTICE:  BL31: v1.3(debug):42583b6
NOTICE:  BL31: Built : 07:55:13, Oct 15 2019
NOTICE:  BL31: Rockchip release version: v1.1
INFO:    GICv3 with legacy support detected. ARM GICV3 driver initialized in EL3
INFO:    Using opteed sec cpu_context!
INFO:    boot cpu mask: 0
INFO:    plat_rockchip_pmu_init(1190): pd status 3e
INFO:    BL31: Initializing runtime services
WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
ERROR:   Error initializing runtime service opteed_fast
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x200000
INFO:    SPSR = 0x3c9

U-Boot 2020.07-armbian (Oct 31 2020 - 08:21:38 +0100)

SoC: Rockchip rk3399
Reset cause: POR
DRAM:  3.9 GiB
PMIC:  RK808
SF: Detected w25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
MMC:   mmc@fe320000: 1, sdhci@fe330000: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Model: Helios64
Revision: 1.2 - 4GB non ECC
Net:   eth0: ethernet@fe300000
scanning bus for devices...
Hit any key to stop autoboot:  0
Card did not respond to voltage select!
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot/boot.scr
3185 bytes read in 18 ms (171.9 KiB/s)
## Executing script at 00500000
Boot script loaded from mmc 0
194 bytes read in 15 ms (11.7 KiB/s)
17971400 bytes read in 1723 ms (9.9 MiB/s)
27507200 bytes read in 2627 ms (10 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
No FDT memory address configured. Please configure
the FDT address via "fdt addr <address>" command.
Aborting!
276 bytes read in 19 ms (13.7 KiB/s)
Applying user provided DT overlay dwc3-0-device.dtbo
No FDT memory address configured. Please configure
the FDT address via "fdt addr <address>" command.
Aborting!
Error applying DT overlays, restoring original DT
## Loading init Ramdisk from Legacy Image at 06000000 ...
   Image Name:   uInitrd
   Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
   Data Size:    17971336 Bytes = 17.1 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
ERROR: Did not find a cmdline Flattened Device Tree
   Loading Ramdisk to f4dc3000, end f5ee6888 ... OK
FDT and ATAGS support not compiled in - hanging
### ERROR ### Please RESET the board ###
 

[djurny]

1 hour ago, Z06Frank said:

Here is the serial output......

[..]
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
No FDT memory address configured. Please configure
the FDT address via "fdt addr <address>" command.
Aborting!
276 bytes read in 19 ms (13.7 KiB/s)
Applying user provided DT overlay dwc3-0-device.dtbo
No FDT memory address configured. Please configure
the FDT address via "fdt addr <address>" command.
Aborting!
Error applying DT overlays, restoring original DT
## Loading init Ramdisk from Legacy Image at 06000000 ...
   Image Name:   uInitrd
   Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
   Data Size:    17971336 Bytes = 17.1 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
ERROR: Did not find a cmdline Flattened Device Tree
   Loading Ramdisk to f4dc3000, end f5ee6888 ... OK
FDT and ATAGS support not compiled in - hanging
### ERROR ### Please RESET the board ###

 

Hi,

Not sure if I can help here, but willing to help.

Can you put the SDcard into another Linux system and list the contents of the /boot folder? Also, please share the contents of the /boot/armbianEnv.txt. Looks like the same thing I had once, after a downgrade of kernel, the DTB files were renamed or not present for unknown reasons.

Groetjes

[Igor]

10 minutes ago, djurny said:

Looks like the same thing I had once, after a downgrade of kernel, the DTB files were renamed or not present for unknown reasons.

 

Problems is known, reason has to be researched. Happens rare, but it does. We should fix this once, but certainly not this year  https://armbian.atlassian.net/browse/AR-572 Not sure for next either due to overwhelming pile of work.  If you guys wants to have perfectly functioning Linux, then you'll need to join and help to get there.

[Z06Frank]

djurny......thanks for your help but I decided to restart from scratch knowing the issue. Back up and running now. I set CPU Governor to "Performance".

 

I read and make some power management changes in Windows 10 (esp ACHI Power Management) to get max performance for SSD/HHDs file transfer and turned off any idling/power savings. I also wanted an option on this tweak to prevent data loss on power loss to hdparm.conf:  http://ubuntuhandbook.org/index.php/2014/01/disable-disk-caching-prevent-data-loss/

[SR-G]

FYI, I'm still encountering a few freezes over time (with 5.9.14) (every 2 weeks, i would say), as far as i can tell, whatever the "CPU" governance configuration is ("conservative" or "performance" have both already generated freezes).

 

So what is the best approach on this ?

Which kernel should be installed ?

It seems that 5.9.14 is at the moment still the latest one (and based on the number of errors i have already encountered here, i have to admit that i'm quite reluctant to test a "dev" and potentially even more unstable branch)

[SR-G]

Situation is worse and worse (i have several freezes per days now - no idea why it has changed, whereas i'm still on the same kernel).

 

What are the updates on this subject ?

[ShadowDance]

@SR-G have you tried limiting min/max CPU speed as well as setting the governor as suggested in https://forum.armbian.com/topic/16664-system-crashes-after-sync-command/?tab=comments#comment-116681 ?

 

I had good luck with only setting governor to performance for a long time, but recently had a system lockup as well. Unfortunately I didn't have serial hooked up at the time so I didn't get any reports. Will keep monitoring (with serial hooked up).

[SR-G]

Yes, as stated in previous message, i tried "conservative" and then "performance" governor mode (which i'm still running on), with same (max) values for both min/max CPU speed, without any real benefits (maybe this is enough for some users, but clearly not in my situation).

 

And i really don't have a lot of things installed : SMB, SSH, docker (with one container : netdata), MDADM, and that's all (no extra containers, no OMV, ...). And no "load" from a CPU point of view, and not a lot of RAID I/O (at least not during most of the crashes).

 

And same, i'm not always in a setup with a usb cable connected (but i would suspect these freezes to be the same ones than the first i encountered and for which some traces are at the beginning of this post).

[tionebrr]

Hello all,
I've suspected that the freezes could be related to throttling, so I restricted the governor to only one frequency (1.8GHz) and I hadn't had a freeze since. Kind of a hammer workaround but well...

[SR-G]

I would be fine with this solution on my side if it was working  (but on my side it's not enough to have same freq. for min and max value (also with 1.8 Ghz, i haven't tryed with lower values) + "performance" or "governor" mode.

[djurny]

3 hours ago, SR-G said:

I would be fine with this solution on my side if it was working  (but on my side it's not enough to have same freq. for min and max value (also with 1.8 Ghz, i haven't tryed with lower values) + "performance" or "governor" mode.

Hi @SR-G,

FYI I have been running Linux kobol0 5.9.13-rockchip64 #trunk.16 SMP PREEMPT Tue Dec 8 21:23:17 CET 2020 aarch64 GNU/Linux for some days now. Still up. Perhaps you can give this one a try?

Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

 

/etc/default/cpufrequtils:

ENABLE=true
MIN_SPEED=408000
MAX_SPEED=1200000
GOVERNOR=powersave

(The box runs idles with 'powersave' and will be set to 'performance' when performing tasks.)

 

I do not know which non-dev kernel version this is? I assume it's 21.02 or 21.03?

Groetjes,

 

Edited January 19, 2021 by djurny
Fixed some Grammer

[Seneca]

I seem to have freezes as well. I stayed on the recommended release with zero updates from the default installation. I'm running Sonarr, Radarr, Jackett, Transmission, Samba, NFS server and Zram swap.

 

Linux muzzy 5.8.14-rockchip64 #20.08.10 SMP PREEMPT Tue Oct 13 16:58:01 CEST 2020 aarch64 aarch64 aarch64 GNU/Linux

I've also changed the cpufreq config to avoid issues. Locked at 1,2ghz min/max, performance govenor.

Quote

Thu 21 Jan 2021 07:25:31 PM CET | Helios64 | 20.08.10 | arm64 | aarch64 | 5.8.14-rockchip64

### dmesg:

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 5.8.14-rockchip64 (root@xeon) (aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 9.2.1 20191025, GNU ld (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 2.33.1.20191209) #20.08.10 SMP PREEMPT Tue Oct 13 16:58:01 CEST 2020
[    0.000000] Machine model: Helios64
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 128 MiB at 0x00000000ecc00000
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000000200000-0x00000000f7ffffff]
[    0.000000] NUMA: NODE_DATA [mem 0xf77db100-0xf77dcfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000200000-0x000000003fffffff]
[    0.000000]   DMA32    [mem 0x0000000040000000-0x00000000f7ffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000200000-0x00000000f7ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000200000-0x00000000f7ffffff]
[    0.000000] On node 0 totalpages: 1015296
[    0.000000]   DMA zone: 4088 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 261632 pages, LIFO batch:63
[    0.000000]   DMA32 zone: 11776 pages used for memmap
[    0.000000]   DMA32 zone: 753664 pages, LIFO batch:63
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.0
[    0.000000] percpu: Embedded 32 pages/cpu s93528 r8192 d29352 u131072
[    0.000000] pcpu-alloc: s93528 r8192 d29352 u131072 alloc=32*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: ARM erratum 845719
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 999432
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: root=UUID=9d58c047-579d-469d-bdee-16cb4f219614 rootwait rootfstype=btrfs console=ttyS2,1500000 console=tty1 consoleblank=0 loglevel=1 ubootpart=2c928d4e-01 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u   cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x3bfff000-0x3ffff000] (64MB)
[    0.000000] Memory: 3740060K/4061184K available (14396K kernel code, 2060K rwdata, 5948K rodata, 4160K init, 574K bss, 190052K reserved, 131072K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=6, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=6.
[    0.000000]     Trampoline variant of Tasks RCU enabled.
[    0.000000]     Rude variant of Tasks RCU enabled.
[    0.000000]     Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=6
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 256 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000fef00000
[    0.000000] ITS [mem 0xfee20000-0xfee3ffff]
[    0.000000] ITS@0x00000000fee20000: allocated 65536 Devices @f6d00000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: using LPI property table @0x00000000f6ca0000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x00000000f6cb0000
[    0.000000] GICv3: GIC: PPI partition interrupt-partition-0[0] { /cpus/cpu@0[0] /cpus/cpu@1[1] /cpus/cpu@2[2] /cpus/cpu@3[3] }
[    0.000000] GICv3: GIC: PPI partition interrupt-partition-1[1] { /cpus/cpu@100[4] /cpus/cpu@101[5] }
[    0.000000] random: get_random_bytes called from start_kernel+0x674/0x83c with crng_init=0
[    0.000000] arch_timer: cp15 timer(s) running at 24.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x588fe9dc0, max_idle_ns: 440795202592 ns
[    0.000007] sched_clock: 56 bits at 24MHz, resolution 41ns, wraps every 4398046511097ns
[    0.001372] Console: colour dummy device 80x25
[    0.001391] printk: console [tty1] enabled
[    0.001528] Calibrating delay loop (skipped), value calculated using timer frequency.. 48.00 BogoMIPS (lpj=96000)
[    0.001543] pid_max: default: 32768 minimum: 301
[    0.001707] LSM: Security Framework initializing
[    0.001747] Yama: becoming mindful.
[    0.001896] AppArmor: AppArmor initialized
[    0.001990] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.002020] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.005294] rcu: Hierarchical SRCU implementation.
[    0.008195] Platform MSI: interrupt-controller@fee20000 domain created
[    0.008715] PCI/MSI: /interrupt-controller@fee00000/interrupt-controller@fee20000 domain created
[    0.010972] EFI services will not be available.
[    0.011740] smp: Bringing up secondary CPUs ...
[    0.012766] Detected VIPT I-cache on CPU1
[    0.012821] GICv3: CPU1: found redistributor 1 region 0:0x00000000fef20000
[    0.012840] GICv3: CPU1: using allocated LPI pending table @0x00000000f6cc0000
[    0.012899] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.014261] Detected VIPT I-cache on CPU2
[    0.014314] GICv3: CPU2: found redistributor 2 region 0:0x00000000fef40000
[    0.014333] GICv3: CPU2: using allocated LPI pending table @0x00000000f6cd0000
[    0.014391] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[    0.015719] Detected VIPT I-cache on CPU3
[    0.015772] GICv3: CPU3: found redistributor 3 region 0:0x00000000fef60000
[    0.015791] GICv3: CPU3: using allocated LPI pending table @0x00000000f6ce0000
[    0.015849] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[    0.017223] CPU features: detected: EL2 vector hardening
[    0.017234] ARM_SMCCC_ARCH_WORKAROUND_1 missing from firmware
[    0.017240] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.017247] Detected PIPT I-cache on CPU4
[    0.017282] GICv3: CPU4: found redistributor 100 region 0:0x00000000fef80000
[    0.017296] GICv3: CPU4: using allocated LPI pending table @0x00000000f6cf0000
[    0.017335] CPU4: Booted secondary processor 0x0000000100 [0x410fd082]
[    0.018522] Detected PIPT I-cache on CPU5
[    0.018560] GICv3: CPU5: found redistributor 101 region 0:0x00000000fefa0000
[    0.018574] GICv3: CPU5: using allocated LPI pending table @0x00000000f6d80000
[    0.018612] CPU5: Booted secondary processor 0x0000000101 [0x410fd082]
[    0.018779] smp: Brought up 1 node, 6 CPUs
[    0.018790] SMP: Total of 6 processors activated.
[    0.018799] CPU features: detected: 32-bit EL0 Support
[    0.018808] CPU features: detected: CRC32 instructions
[    0.018817] CPU features: detected: 32-bit EL1 Support
[    0.046761] CPU: All CPU(s) started at EL2
[    0.046824] alternatives: patching kernel code
[    0.050606] devtmpfs: initialized
[    0.066528] Registered cp15_barrier emulation handler
[    0.066548] Registered setend emulation handler
[    0.067377] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.067444] futex hash table entries: 2048 (order: 5, 131072 bytes, linear)
[    0.071682] xor: measuring software checksum speed
[    0.107962]    8regs     :  2345.000 MB/sec
[    0.148053]    32regs    :  2757.000 MB/sec
[    0.188146]    arm64_neon:  2225.000 MB/sec
[    0.188153] xor: using function: 32regs (2757.000 MB/sec)
[    0.188210] pinctrl core: initialized pinctrl subsystem
[    0.188749] thermal_sys: Registered thermal governor 'fair_share'
[    0.188754] thermal_sys: Registered thermal governor 'bang_bang'
[    0.188760] thermal_sys: Registered thermal governor 'step_wise'
[    0.188765] thermal_sys: Registered thermal governor 'user_space'
[    0.189421] DMI not present or invalid.
[    0.190012] NET: Registered protocol family 16
[    0.193160] DMA: preallocated 2048 KiB GFP_KERNEL pool for atomic allocations
[    0.194635] DMA: preallocated 2048 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.195404] DMA: preallocated 2048 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.195507] audit: initializing netlink subsys (disabled)
[    0.195792] audit: type=2000 audit(0.192:1): state=initialized audit_enabled=0 res=1
[    0.197023] cpuidle: using governor menu
[    0.197335] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.197551] ASID allocator initialised with 65536 entries
[    0.198812] Serial: AMBA PL011 UART driver
[    0.255240] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.255251] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.255258] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.255265] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.261209] cryptd: max_cpu_qlen set to 1000
[    0.336564] raid6: neonx8   gen()  1668 MB/s
[    0.404714] raid6: neonx8   xor()  1226 MB/s
[    0.472891] raid6: neonx4   gen()  1686 MB/s
[    0.541047] raid6: neonx4   xor()  1261 MB/s
[    0.609211] raid6: neonx2   gen()  1525 MB/s
[    0.677375] raid6: neonx2   xor()  1161 MB/s
[    0.745547] raid6: neonx1   gen()  1178 MB/s
[    0.813697] raid6: neonx1   xor()   945 MB/s
[    0.881906] raid6: int64x8  gen()   993 MB/s
[    0.950052] raid6: int64x8  xor()   581 MB/s
[    1.018218] raid6: int64x4  gen()   983 MB/s
[    1.086373] raid6: int64x4  xor()   591 MB/s
[    1.154530] raid6: int64x2  gen()   934 MB/s
[    1.222696] raid6: int64x2  xor()   524 MB/s
[    1.290861] raid6: int64x1  gen()   724 MB/s
[    1.359021] raid6: int64x1  xor()   390 MB/s
[    1.359026] raid6: using algorithm neonx4 gen() 1686 MB/s
[    1.359032] raid6: .... xor() 1261 MB/s, rmw enabled
[    1.359038] raid6: using neon recovery algorithm
[    1.360458] vcc12v_dcin_bkup: supplied by vcc12v_dcin
[    1.360835] vcc12v_hdd: supplied by vcc12v_dcin_bkup
[    1.374150] vcc5v0_perdev: supplied by vcc12v_dcin_bkup
[    1.374662] vcc5v0_hdd: supplied by vcc12v_dcin_bkup
[    1.374943] pcie_power: supplied by vcc5v0_perdev
[    1.375051] vcc5v0_usb: supplied by vcc5v0_perdev
[    1.375268] vcc5v0_sys: supplied by vcc12v_dcin_bkup
[    1.376031] vcc3v3_sys_s3: supplied by vcc5v0_sys
[    1.376086] vcc0v9_s3: supplied by vcc3v3_sys_s3
[    1.376175] avdd_1v8_s0: supplied by vcc3v3_sys_s3
[    1.376243] vcc3v0_sd: supplied by vcc3v3_sys_s3
[    1.377088] usblan_power: supplied by vcc5v0_usb
[    1.377736] iommu: Default domain type: Translated
[    1.380828] vgaarb: loaded
[    1.381357] SCSI subsystem initialized
[    1.381609] libata version 3.00 loaded.
[    1.381871] usbcore: registered new interface driver usbfs
[    1.381915] usbcore: registered new interface driver hub
[    1.382029] usbcore: registered new device driver usb
[    1.382712] pps_core: LinuxPPS API ver. 1 registered
[    1.382718] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.382734] PTP clock support registered
[    1.383045] EDAC MC: Ver: 3.0.0
[    1.384258] NetLabel: Initializing
[    1.384265] NetLabel:  domain hash size = 128
[    1.384270] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    1.384332] NetLabel:  unlabeled traffic allowed by default
[    1.384865] clocksource: Switched to clocksource arch_sys_counter
[    1.487908] VFS: Disk quotas dquot_6.6.0
[    1.487984] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.488834] AppArmor: AppArmor Filesystem Enabled
[    1.499434] NET: Registered protocol family 2
[    1.500260] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[    1.500439] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    1.501009] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)
[    1.501675] TCP: Hash tables configured (established 32768 bind 32768)
[    1.501870] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    1.502033] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    1.502388] NET: Registered protocol family 1
[    1.502418] NET: Registered protocol family 44
[    1.502432] PCI: CLS 0 bytes, default 64
[    1.502672] Trying to unpack rootfs image as initramfs...
[    2.683323] Freeing initrd memory: 15976K
[    2.684553] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
[    2.685056] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    2.685744] kvm [1]: IPA Size Limit: 40bits
[    2.686491] kvm [1]: vgic-v2@fff20000
[    2.686528] kvm [1]: GIC system register CPU interface enabled
[    2.686767] kvm [1]: vgic interrupt IRQ10
[    2.687060] kvm [1]: Hyp mode initialized successfully
[    2.698450] Initialise system trusted keyrings
[    2.698487] Key type blacklist registered
[    2.698686] workingset: timestamp_bits=44 max_order=20 bucket_order=0
[    2.705194] zbud: loaded
[    2.706857] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    2.707779] fuse: init (API version 7.31)
[    2.709259] integrity: Platform Keyring initialized
[    2.773415] Key type asymmetric registered
[    2.773422] Asymmetric key parser 'x509' registered
[    2.773464] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    2.773679] io scheduler mq-deadline registered
[    2.773686] io scheduler kyber registered
[    2.773826] io scheduler bfq registered
[    2.798193] dma-pl330 ff6d0000.dma-controller: Loaded driver for PL330 DMAC-241330
[    2.798206] dma-pl330 ff6d0000.dma-controller:     DBUFF-32x8bytes Num_Chans-6 Num_Peri-12 Num_Events-12
[    2.799699] dma-pl330 ff6e0000.dma-controller: Loaded driver for PL330 DMAC-241330
[    2.799710] dma-pl330 ff6e0000.dma-controller:     DBUFF-128x8bytes Num_Chans-8 Num_Peri-20 Num_Events-16
[    2.803859] pwm-regulator: supplied by regulator-dummy
[    2.805485] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled
[    2.808743] ff180000.serial: ttyS0 at MMIO 0xff180000 (irq = 36, base_baud = 1500000) is a 16550A
[    2.809824] ff1a0000.serial: ttyS2 at MMIO 0xff1a0000 (irq = 37, base_baud = 1500000) is a 16550A
[    2.810010] printk: console [ttyS2] enabled
[    2.811242] Serial: AMBA driver
[    2.812694] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    2.824705] loop: module loaded
[    2.833555] libphy: Fixed MDIO Bus: probed
[    2.833907] tun: Universal TUN/TAP device driver, 1.6
[    2.834990] thunder_xcv, ver 1.0
[    2.835033] thunder_bgx, ver 1.0
[    2.835093] nicpf, ver 1.0
[    2.836227] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    2.836232] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    2.836283] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
[    2.836288] igb: Copyright (c) 2007-2014 Intel Corporation.
[    2.836331] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[    2.836335] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    2.836632] sky2: driver version 1.30
[    2.837447] VFIO - User Level meta-driver version: 0.3
[    2.851776] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.851797] ehci-pci: EHCI PCI platform driver
[    2.851837] ehci-platform: EHCI generic platform driver
[    2.852324] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.852341] ohci-pci: OHCI PCI platform driver
[    2.852384] ohci-platform: OHCI generic platform driver
[    2.853554] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[    2.853587] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
[    2.853783] xhci-hcd xhci-hcd.0.auto: hcc params 0x0220fe64 hci version 0x110 quirks 0x0000000002010010
[    2.853847] xhci-hcd xhci-hcd.0.auto: irq 226, io mem 0xfe900000
[    2.854274] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.08
[    2.854285] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.854293] usb usb1: Product: xHCI Host Controller
[    2.854301] usb usb1: Manufacturer: Linux 5.8.14-rockchip64 xhci-hcd
[    2.854308] usb usb1: SerialNumber: xhci-hcd.0.auto
[    2.854928] hub 1-0:1.0: USB hub found
[    2.854967] hub 1-0:1.0: 1 port detected
[    2.855335] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[    2.855351] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
[    2.855367] xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0 SuperSpeed
[    2.855446] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    2.855582] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.08
[    2.855591] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.855600] usb usb2: Product: xHCI Host Controller
[    2.855607] usb usb2: Manufacturer: Linux 5.8.14-rockchip64 xhci-hcd
[    2.855615] usb usb2: SerialNumber: xhci-hcd.0.auto
[    2.856144] hub 2-0:1.0: USB hub found
[    2.856181] hub 2-0:1.0: 1 port detected
[    2.856654] usbcore: registered new interface driver usb-storage
[    2.859341] mousedev: PS/2 mouse device common for all mice
[    2.860746] i2c /dev entries driver
[    2.863490] pca953x 2-0020: using no AI
[    2.870150] rk808 0-001b: chip id: 0x0
[    2.874150] rk808-regulator rk808-regulator: there is no dvs0 gpio
[    2.874177] rk808-regulator rk808-regulator: there is no dvs1 gpio
[    2.874274] DCDC_REG1: supplied by vcc5v0_sys
[    2.875441] DCDC_REG2: supplied by vcc5v0_sys
[    2.876067] DCDC_REG3: supplied by vcc5v0_sys
[    2.876387] DCDC_REG4: supplied by vcc5v0_sys
[    2.877111] LDO_REG1: supplied by vcc5v0_sys
[    2.877597] LDO_REG2: supplied by vcc5v0_sys
[    2.878110] LDO_REG3: supplied by vcc5v0_sys
[    2.879615] LDO_REG4: supplied by vcc5v0_sys
[    2.879828] vcc_sdio_s0: Bringing 3300000uV into 3000000-3000000uV
[    2.881063] LDO_REG5: supplied by vcc5v0_sys
[    2.881594] LDO_REG6: supplied by vcc5v0_sys
[    2.882779] LDO_REG7: supplied by vcc5v0_sys
[    2.883295] LDO_REG8: supplied by vcc5v0_sys
[    2.884471] SWITCH_REG1: supplied by vcc3v3_sys_s3
[    2.884878] SWITCH_REG2: supplied by vcc3v3_sys_s3
[    2.888733] rk808-rtc rk808-rtc: registered as rtc0
[    2.889305] rk808-rtc rk808-rtc: setting system clock to 2021-01-21T18:25:19 UTC (1611253519)
[    2.891473] fan53555-regulator 0-0040: FAN53555 Option[8] Rev[1] Detected!
[    2.891514] fan53555-reg: supplied by vcc5v0_sys
[    2.892695] vcc1v8_sys_s0: supplied by vcc1v8_sys_s3
[    2.892781] avdd_0v9_s0: supplied by vcc1v8_sys_s3
[    2.894863] fan53555-regulator 0-0041: FAN53555 Option[8] Rev[1] Detected!
[    2.894906] fan53555-reg: supplied by vcc5v0_sys
[    2.905584] sdhci: Secure Digital Host Controller Interface driver
[    2.905591] sdhci: Copyright(c) Pierre Ossman
[    2.905617] Synopsys Designware Multimedia Card Interface Driver
[    2.907405] dwmmc_rockchip fe320000.mmc: IDMAC supports 32-bit address mode.
[    2.907435] dwmmc_rockchip fe320000.mmc: Using internal DMA controller.
[    2.907452] dwmmc_rockchip fe320000.mmc: Version ID is 270a
[    2.907514] dwmmc_rockchip fe320000.mmc: DW MMC controller at irq 28,32 bit host data width,256 deep fifo
[    2.907798] dwmmc_rockchip fe320000.mmc: Got CD GPIO
[    2.920667] mmc_host mmc0: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
[    2.934043] sdhci-pltfm: SDHCI platform and OF driver helper
[    2.935706] mmc1: CQHCI version 5.10
[    2.961483] mmc1: SDHCI controller on fe330000.sdhci [fe330000.sdhci] using ADMA
[    2.965312] ledtrig-cpu: registered to indicate activity on CPUs
[    2.966514] hid: raw HID events driver (C) Jiri Kosina
[    2.966647] usbcore: registered new interface driver usbhid
[    2.966652] usbhid: USB HID core driver
[    2.971609] random: fast init done
[    2.972042] drop_monitor: Initializing network drop monitor service
[    2.972952] NET: Registered protocol family 10
[    3.017334] Segment Routing with IPv6
[    3.017427] NET: Registered protocol family 17
[    3.017673] 8021q: 802.1Q VLAN Support v1.8
[    3.017743] Key type dns_resolver registered
[    3.018306] registered taskstats version 1
[    3.018318] Loading compiled-in X.509 certificates
[    3.021807] Loaded X.509 cert 'Build time autogenerated kernel key: 3c30550356ab525d6d7708f95571a45a475bcefb'
[    3.022015] zswap: loaded using pool lzo/zbud
[    3.022399] Key type ._fscrypt registered
[    3.022405] Key type .fscrypt registered
[    3.022410] Key type fscrypt-provisioning registered
[    3.025494] Btrfs loaded, crc32c=crc32c-generic
[    3.037335] mmc1: Command Queue Engine enabled
[    3.037355] mmc1: new HS200 MMC card at address 0001
[    3.038125] mmcblk1: mmc1:0001 AJTD4R 14.6 GiB
[    3.038368] mmcblk1boot0: mmc1:0001 AJTD4R partition 1 4.00 MiB
[    3.038602] mmcblk1boot1: mmc1:0001 AJTD4R partition 2 4.00 MiB
[    3.038797] mmcblk1rpmb: mmc1:0001 AJTD4R partition 3 4.00 MiB, chardev (243:0)
[    3.040106]  mmcblk1: p1 p2
[    3.047191] Key type encrypted registered
[    3.047215] AppArmor: AppArmor sha1 policy hashing enabled
[    3.153921] rockchip-pcie f8000000.pcie: host bridge /pcie@f8000000 ranges:
[    3.153965] rockchip-pcie f8000000.pcie:      MEM 0x00fa000000..0x00fbdfffff -> 0x00fa000000
[    3.153986] rockchip-pcie f8000000.pcie:       IO 0x00fbe00000..0x00fbefffff -> 0x00fbe00000
[    3.154322] rockchip-pcie f8000000.pcie: PCI host bridge to bus 0000:00
[    3.154333] pci_bus 0000:00: root bus resource [bus 00-1f]
[    3.154343] pci_bus 0000:00: root bus resource [mem 0xfa000000-0xfbdfffff]
[    3.154360] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0xfbe00000-0xfbefffff])
[    3.154402] pci 0000:00:00.0: [1d87:0100] type 01 class 0x060400
[    3.154565] pci 0000:00:00.0: supports D1
[    3.154573] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    3.159810] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    3.160016] pci 0000:01:00.0: [197b:0585] type 00 class 0x010601
[    3.160138] pci 0000:01:00.0: reg 0x10: initial BAR value 0x00000000 invalid
[    3.160147] pci 0000:01:00.0: reg 0x10: [io  size 0x0080]
[    3.160183] pci 0000:01:00.0: reg 0x14: initial BAR value 0x00000000 invalid
[    3.160191] pci 0000:01:00.0: reg 0x14: [io  size 0x0080]
[    3.160226] pci 0000:01:00.0: reg 0x18: initial BAR value 0x00000000 invalid
[    3.160234] pci 0000:01:00.0: reg 0x18: [io  size 0x0080]
[    3.160269] pci 0000:01:00.0: reg 0x1c: initial BAR value 0x00000000 invalid
[    3.160276] pci 0000:01:00.0: reg 0x1c: [io  size 0x0080]
[    3.160310] pci 0000:01:00.0: reg 0x20: initial BAR value 0x00000000 invalid
[    3.160317] pci 0000:01:00.0: reg 0x20: [io  size 0x0080]
[    3.160352] pci 0000:01:00.0: reg 0x24: [mem 0x00000000-0x00001fff]
[    3.160388] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    3.160426] pci 0000:01:00.0: Max Payload Size set to 256 (was 128, max 512)
[    3.160746] pci 0000:01:00.0: PME# supported from D3hot
[    3.160979] pci 0000:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of 15.752 Gb/s with 8.0 GT/s PCIe x2 link)
[    3.166119] pci_bus 0000:01: busn_res: [bus 01-1f] end is updated to 01
[    3.166149] pci 0000:00:00.0: BAR 14: assigned [mem 0xfa000000-0xfa0fffff]
[    3.166164] pci 0000:01:00.0: BAR 6: assigned [mem 0xfa000000-0xfa00ffff pref]
[    3.166175] pci 0000:01:00.0: BAR 5: assigned [mem 0xfa010000-0xfa011fff]
[    3.166195] pci 0000:01:00.0: BAR 0: no space for [io  size 0x0080]
[    3.166204] pci 0000:01:00.0: BAR 0: failed to assign [io  size 0x0080]
[    3.166211] pci 0000:01:00.0: BAR 1: no space for [io  size 0x0080]
[    3.166219] pci 0000:01:00.0: BAR 1: failed to assign [io  size 0x0080]
[    3.166226] pci 0000:01:00.0: BAR 2: no space for [io  size 0x0080]
[    3.166233] pci 0000:01:00.0: BAR 2: failed to assign [io  size 0x0080]
[    3.166240] pci 0000:01:00.0: BAR 3: no space for [io  size 0x0080]
[    3.166247] pci 0000:01:00.0: BAR 3: failed to assign [io  size 0x0080]
[    3.166255] pci 0000:01:00.0: BAR 4: no space for [io  size 0x0080]
[    3.166262] pci 0000:01:00.0: BAR 4: failed to assign [io  size 0x0080]
[    3.166271] pci 0000:00:00.0: PCI bridge to [bus 01]
[    3.166284] pci 0000:00:00.0:   bridge window [mem 0xfa000000-0xfa0fffff]
[    3.166521] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
[    3.166831] pcieport 0000:00:00.0: PME: Signaling with IRQ 237
[    3.167174] pcieport 0000:00:00.0: AER: enabled with IRQ 237
[    3.167523] ahci 0000:01:00.0: version 3.0
[    3.167548] ahci 0000:01:00.0: enabling device (0000 -> 0002)
[    3.168107] ahci 0000:01:00.0: SSS flag set, parallel bus scan disabled
[    3.168209] ahci 0000:01:00.0: AHCI 0001.0301 32 slots 5 ports 6 Gbps 0x1f impl SATA mode
[    3.168222] ahci 0000:01:00.0: flags: 64bit ncq sntf stag pm led clo pmp fbs pio slum part ccc apst boh
[    3.171852] scsi host0: ahci
[    3.172536] scsi host1: ahci
[    3.173132] scsi host2: ahci
[    3.173659] scsi host3: ahci
[    3.174186] scsi host4: ahci
[    3.174410] ata1: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010100 irq 238
[    3.174420] ata2: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010180 irq 239
[    3.174428] ata3: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010200 irq 240
[    3.174436] ata4: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010280 irq 241
[    3.174444] ata5: SATA max UDMA/133 abar m8192@0xfa010000 port 0xfa010300 irq 242
[    3.182999] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[    3.183473] of_cfs_init
[    3.183535] of_cfs_init: OK
[    3.188912] usb 1-1: new high-speed USB device number 2 using xhci-hcd
[    3.338933] usb 1-1: New USB device found, idVendor=2109, idProduct=2815, bcdDevice= 7.04
[    3.338944] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    3.338953] usb 1-1: Product: USB2.0 Hub             
[    3.338960] usb 1-1: Manufacturer: VIA Labs, Inc.         
[    3.399720] hub 1-1:1.0: USB hub found
[    3.399899] hub 1-1:1.0: 4 ports detected
[    3.494438] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci-hcd
[    3.612631] usb 2-1: New USB device found, idVendor=2109, idProduct=0815, bcdDevice= 7.04
[    3.612643] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    3.612651] usb 2-1: Product: USB3.0 Hub             
[    3.612658] usb 2-1: Manufacturer: VIA Labs, Inc.         
[    3.639787] hub 2-1:1.0: USB hub found
[    3.639986] hub 2-1:1.0: 4 ports detected
[    3.648924] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    3.651391] ata1.00: ATA-10: Micron_5210_MTFDDAK7T6QDE,  D2MU801, max UDMA/133
[    3.651402] ata1.00: 15002931888 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    3.653092] ata1.00: configured for UDMA/133
[    3.653765] scsi 0:0:0:0: Direct-Access     ATA      Micron_5210_MTFD U801 PQ: 0 ANSI: 5
[    3.655129] ata1.00: Enabling discard_zeroes_data
[    3.655606] sd 0:0:0:0: [sda] 15002931888 512-byte logical blocks: (7.68 TB/6.99 TiB)
[    3.655619] sd 0:0:0:0: [sda] 4096-byte physical blocks
[    3.655706] sd 0:0:0:0: [sda] Write Protect is off
[    3.655722] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    3.655873] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    3.656790] ata1.00: Enabling discard_zeroes_data
[    3.660434]  sda: sda1
[    3.662525] ata1.00: Enabling discard_zeroes_data
[    3.663173] sd 0:0:0:0: [sda] Attached SCSI disk
[    4.128901] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    4.140795] ata2.00: ATA-8: Corsair Force 3 SSD, 5.05, max UDMA/133
[    4.140809] ata2.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    4.150799] ata2.00: configured for UDMA/133
[    4.151435] scsi 1:0:0:0: Direct-Access     ATA      Corsair Force 3  5.05 PQ: 0 ANSI: 5
[    4.152976] sd 1:0:0:0: [sdb] 468862128 512-byte logical blocks: (240 GB/224 GiB)
[    4.153050] sd 1:0:0:0: [sdb] Write Protect is off
[    4.153065] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    4.153179] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    4.156796]  sdb: sdb1
[    4.159625] sd 1:0:0:0: [sdb] Attached SCSI disk
[    4.280963] usb 2-1.4: new SuperSpeed Gen 1 USB device number 3 using xhci-hcd
[    4.301936] usb 2-1.4: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=30.00
[    4.301946] usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[    4.301955] usb 2-1.4: Product: USB 10/100/1G/2.5G LAN
[    4.301962] usb 2-1.4: Manufacturer: Realtek
[    4.301970] usb 2-1.4: SerialNumber: 000000001
[    4.628900] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    4.640840] ata3.00: ATA-8: Corsair Force 3 SSD, 5.05, max UDMA/133
[    4.640890] ata3.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    4.650843] ata3.00: configured for UDMA/133
[    4.651475] scsi 2:0:0:0: Direct-Access     ATA      Corsair Force 3  5.05 PQ: 0 ANSI: 5
[    4.653076] sd 2:0:0:0: [sdc] 468862128 512-byte logical blocks: (240 GB/224 GiB)
[    4.653147] sd 2:0:0:0: [sdc] Write Protect is off
[    4.653161] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[    4.653275] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    4.656760]  sdc: sdc1
[    4.659406] sd 2:0:0:0: [sdc] Attached SCSI disk
[    5.128897] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    5.140384] ata4.00: ATA-9: INTEL SSDSC2BW180A3L, LF1i, max UDMA/133
[    5.140393] ata4.00: 351651888 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    5.150386] ata4.00: configured for UDMA/133
[    5.150920] scsi 3:0:0:0: Direct-Access     ATA      INTEL SSDSC2BW18 LF1i PQ: 0 ANSI: 5
[    5.152331] sd 3:0:0:0: [sdd] 351651888 512-byte logical blocks: (180 GB/168 GiB)
[    5.152403] sd 3:0:0:0: [sdd] Write Protect is off
[    5.152417] sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[    5.152531] sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    5.154559]  sdd: sdd1
[    5.156457] sd 3:0:0:0: [sdd] Attached SCSI disk
[    5.467352] ata5: SATA link down (SStatus 0 SControl 300)
[    5.470156] Freeing unused kernel memory: 4160K
[    5.470418] Run /init as init process
[    5.470425]   with arguments:
[    5.470430]     /init
[    5.470435]   with environment:
[    5.470440]     HOME=/
[    5.470445]     TERM=linux
[    5.470450]     ubootpart=2c928d4e-01
[    5.470455]     cgroup_enable=memory
[    5.470460]     cgroup_memory=1
[    6.047855] input: adc-keys as /devices/platform/adc-keys/input/input1
[    6.192777] rockchip-vop ff8f0000.vop: Adding to iommu group 2
[    6.199192] rockchip-vop ff900000.vop: Adding to iommu group 3
[    6.313185] rk_gmac-dwmac fe300000.ethernet: IRQ eth_wake_irq not found
[    6.313203] rk_gmac-dwmac fe300000.ethernet: IRQ eth_lpi not found
[    6.313430] rk_gmac-dwmac fe300000.ethernet: PTP uses main clock
[    6.313610] rk_gmac-dwmac fe300000.ethernet: clock input or output? (input).
[    6.313627] rk_gmac-dwmac fe300000.ethernet: TX delay(0x28).
[    6.313640] rk_gmac-dwmac fe300000.ethernet: RX delay(0x20).
[    6.313661] rk_gmac-dwmac fe300000.ethernet: integrated PHY? (no).
[    6.313772] rk_gmac-dwmac fe300000.ethernet: cannot get clock clk_mac_speed
[    6.313782] rk_gmac-dwmac fe300000.ethernet: clock input from PHY
[    6.318814] rk_gmac-dwmac fe300000.ethernet: init for RGMII
[    6.324984] rk_gmac-dwmac fe300000.ethernet: User ID: 0x10, Synopsys ID: 0x35
[    6.325000] rk_gmac-dwmac fe300000.ethernet:     DWMAC1000
[    6.325010] rk_gmac-dwmac fe300000.ethernet: DMA HW capability register supported
[    6.325018] rk_gmac-dwmac fe300000.ethernet: RX Checksum Offload Engine supported
[    6.325025] rk_gmac-dwmac fe300000.ethernet: COE Type 2
[    6.325032] rk_gmac-dwmac fe300000.ethernet: TX Checksum insertion supported
[    6.325039] rk_gmac-dwmac fe300000.ethernet: Wake-Up On Lan supported
[    6.325082] rk_gmac-dwmac fe300000.ethernet: Normal descriptors
[    6.325089] rk_gmac-dwmac fe300000.ethernet: Ring mode enabled
[    6.325096] rk_gmac-dwmac fe300000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    6.468717] libphy: stmmac: probed
[    6.468741] RTL8211F Gigabit Ethernet stmmac-0:00: attached PHY driver [RTL8211F Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:00, irq=POLL)
[    6.468754] RTL8211F Gigabit Ethernet stmmac-0:01: attached PHY driver [RTL8211F Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:01, irq=POLL)
[    6.762442] async_tx: api initialized (async)
[    6.979587] BTRFS: device fsid 8ea91444-ba20-47a0-9932-e6a07c974e97 devid 1 transid 7 /dev/sdd1 scanned by btrfs (354)
[    6.980247] BTRFS: device fsid a587760c-2c3f-40a3-99b6-52e67edeebc1 devid 1 transid 222 /dev/sdc1 scanned by btrfs (354)
[    6.981171] BTRFS: device fsid fccb0287-8cf9-4852-a859-5dc57ecc8a66 devid 1 transid 1907 /dev/sdb1 scanned by btrfs (354)
[    6.981827] BTRFS: device fsid dd84e5f9-e112-4c70-a4a7-b4a785d69490 devid 1 transid 1923 /dev/sda1 scanned by btrfs (354)
[    6.982231] BTRFS: device fsid 9d58c047-579d-469d-bdee-16cb4f219614 devid 1 transid 3681 /dev/mmcblk1p2 scanned by btrfs (354)
[    7.078902] BTRFS info (device mmcblk1p2): disk space caching is enabled
[    7.078915] BTRFS info (device mmcblk1p2): has skinny extents
[    7.089321] BTRFS info (device mmcblk1p2): enabling ssd optimizations
[    7.089334] BTRFS info (device mmcblk1p2): start tree-log replay
[    7.907513] systemd[1]: Inserted module 'autofs4'
[    7.947389] systemd[1]: systemd 245.4-4ubuntu3.2 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
[    7.948819] systemd[1]: Detected architecture arm64.
[    7.983950] systemd[1]: Set hostname to <muzzy>.
[    8.077471] dw-apb-uart ff1a0000.serial: forbid DMA for kernel console
[    8.335846] systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
[    8.443640] systemd[1]: /lib/systemd/system/rpc-statd.service:16: PIDFile= references a path below legacy directory /var/run/, updating /var/run/rpc.statd.pid → /run/rpc.statd.pid; please update the unit file accordingly.
[    8.473390] systemd[1]: /lib/systemd/system/fancontrol.service:11: PIDFile= references a path below legacy directory /var/run/, updating /var/run/fancontrol.pid → /run/fancontrol.pid; please update the unit file accordingly.
[    8.589415] random: systemd: uninitialized urandom read (16 bytes read)
[    8.593498] systemd[1]: Created slice system-modprobe.slice.
[    8.594059] random: systemd: uninitialized urandom read (16 bytes read)
[    8.595576] systemd[1]: Created slice system-serial\x2dgetty.slice.
[    8.595930] random: systemd: uninitialized urandom read (16 bytes read)
[    8.597318] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[    8.599472] systemd[1]: Created slice User and Session Slice.
[    8.599867] systemd[1]: Condition check resulted in Dispatch Password Requests to Console Directory Watch when bootsplash is active being skipped.
[    8.600234] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    8.601569] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    8.602111] systemd[1]: Reached target Local Encrypted Volumes.
[    8.602588] systemd[1]: Reached target Slices.
[    8.602925] systemd[1]: Reached target Swap.
[    8.603241] systemd[1]: Reached target System Time Set.
[    8.630671] systemd[1]: Listening on RPCbind Server Activation Socket.
[    8.631791] systemd[1]: Listening on Syslog Socket.
[    8.632693] systemd[1]: Listening on fsck to fsckd communication Socket.
[    8.633345] systemd[1]: Listening on initctl Compatibility Named Pipe.
[    8.635647] systemd[1]: Listening on Journal Audit Socket.
[    8.636550] systemd[1]: Listening on Journal Socket (/dev/log).
[    8.637671] systemd[1]: Listening on Journal Socket.
[    8.638649] systemd[1]: Listening on udev Control Socket.
[    8.639346] systemd[1]: Listening on udev Kernel Socket.
[    8.644933] systemd[1]: Mounting Huge Pages File System...
[    8.650897] systemd[1]: Mounting POSIX Message Queue File System...
[    8.656758] systemd[1]: Mounting NFSD configuration filesystem...
[    8.663092] systemd[1]: Mounting RPC Pipe File System...
[    8.670823] systemd[1]: Mounting Kernel Debug File System...
[    8.677228] systemd[1]: Mounting Kernel Trace File System...
[    8.678269] systemd[1]: Condition check resulted in Kernel Module supporting RPCSEC_GSS being skipped.
[    8.687374] systemd[1]: Starting Set the console keyboard layout...
[    8.694969] systemd[1]: Starting Create list of static device nodes for the current kernel...
[    8.695685] systemd[1]: Condition check resulted in Load Kernel Module drm being skipped.
[    8.701908] systemd[1]: Started Nameserver information manager.
[    8.703168] systemd[1]: Reached target Network (Pre).
[    8.706159] systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
[    8.706499] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
[    8.713992] systemd[1]: Starting Load Kernel Modules...
[    8.720668] systemd[1]: Starting Remount Root and Kernel File Systems...
[    8.727246] systemd[1]: Starting udev Coldplug all Devices...
[    8.737898] systemd[1]: Mounted Huge Pages File System.
[    8.739404] systemd[1]: Mounted POSIX Message Queue File System.
[    8.740746] systemd[1]: Mounted Kernel Debug File System.
[    8.742102] systemd[1]: Mounted Kernel Trace File System.
[    8.746172] systemd[1]: Finished Create list of static device nodes for the current kernel.
[    8.766939] BTRFS warning (device mmcblk1p2): excessive commit interval 600
[    8.766956] BTRFS info (device mmcblk1p2): use lzo compression, level 0
[    8.766964] BTRFS info (device mmcblk1p2): disk space caching is enabled
[    8.778293] systemd[1]: Finished Remount Root and Kernel File Systems.
[    8.782139] systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
[    8.782342] systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
[    8.787279] systemd[1]: Starting Load/Save Random Seed...
[    8.791017] lm75 2-004c: hwmon2: sensor 'lm75'
[    8.793504] systemd[1]: Starting Create System Users...
[    8.802959] systemd[1]: Finished Load Kernel Modules.
[    8.810368] systemd[1]: Mounting FUSE Control File System...
[    8.816945] systemd[1]: Mounting Kernel Configuration File System...
[    8.829537] systemd[1]: Starting Apply Kernel Variables...
[    8.841887] systemd[1]: Mounted FUSE Control File System.
[    8.848587] systemd[1]: Mounted Kernel Configuration File System.
[    8.852179] RPC: Registered named UNIX socket transport module.
[    8.852190] RPC: Registered udp transport module.
[    8.852196] RPC: Registered tcp transport module.
[    8.852202] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    8.861144] systemd[1]: Mounted RPC Pipe File System.
[    8.867179] systemd[1]: Finished Create System Users.
[    8.876081] systemd[1]: Starting pNFS block layout mapping daemon...
[    8.886365] systemd[1]: Starting Create Static Device Nodes in /dev...
[    8.891727] systemd[1]: Started pNFS block layout mapping daemon.
[    8.907251] systemd[1]: Finished Apply Kernel Variables.
[    8.938372] systemd[1]: Finished Create Static Device Nodes in /dev.
[    8.947989] systemd[1]: Starting udev Kernel Device Manager...
[    8.989660] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    9.000439] systemd[1]: Mounted NFSD configuration filesystem.
[    9.120563] systemd[1]: Started udev Kernel Device Manager.
[    9.144100] systemd[1]: Finished Set the console keyboard layout.
[    9.146098] systemd[1]: Reached target Local File Systems (Pre).
[    9.154427] systemd[1]: Mounting /tmp...
[    9.181966] systemd[1]: Mounted /tmp.
[    9.442101] systemd[1]: Finished udev Coldplug all Devices.
[    9.453170] systemd[1]: Starting Helper to synchronize boot up for ifupdown...
[    9.480809] systemd[1]: Finished Helper to synchronize boot up for ifupdown.
[    9.509623] input: beeper as /devices/platform/beeper/input/input2
[    9.545799] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    9.545929] sd 1:0:0:0: Attached scsi generic sg1 type 0
[    9.546048] sd 2:0:0:0: Attached scsi generic sg2 type 0
[    9.546191] sd 3:0:0:0: Attached scsi generic sg3 type 0
[    9.570181] systemd[1]: Found device /dev/ttyS2.
[    9.636075] mc: Linux media interface: v0.10
[    9.698760] videodev: Linux video capture interface: v2.00
[    9.709392] panfrost ff9a0000.gpu: clock rate = 500000000
[    9.709977] panfrost ff9a0000.gpu: mali-t860 id 0x860 major 0x2 minor 0x0 status 0x0
[    9.709987] panfrost ff9a0000.gpu: features: 00000000,100e77bf, issues: 00000000,24040400
[    9.709994] panfrost ff9a0000.gpu: Features: L2:0x07120206 Shader:0x00000000 Tiler:0x00000809 Mem:0x1 MMU:0x00002830 AS:0xff JS:0x7
[    9.709999] panfrost ff9a0000.gpu: shader_present=0xf l2_present=0x1
[    9.722774] fusb30x: module is from the staging directory, the quality is unknown, you have been warned.
[    9.724710] systemd[1]: Found device /dev/disk/by-uuid/4f6ebe40-b7d9-4919-bcc8-4421b42bedce.
[    9.730400] systemd[1]: Starting File System Check on /dev/disk/by-uuid/4f6ebe40-b7d9-4919-bcc8-4421b42bedce...
[    9.732539] [drm] Initialized panfrost 1.1.0 20180908 for ff9a0000.gpu on minor 0
[    9.763811] fusb302 4-0022: port 0 probe success with role ROLE_MODE_DRP, try_role ROLE_MODE_UFP
[    9.779172] systemd[1]: Started File System Check Daemon to report status.
[    9.781318] rockchip-drm display-subsystem: bound ff8f0000.vop (ops vop_component_ops [rockchipdrm])
[    9.781444] [drm] unsupported AFBC format[3231564e]
[    9.785003] rockchip-rga ff680000.rga: HW Version: 0x03.02
[    9.785238] rockchip-drm display-subsystem: bound ff900000.vop (ops vop_component_ops [rockchipdrm])
[    9.785285] rockchip-rga ff680000.rga: Registered rockchip-rga as /dev/video0
[    9.785546] rockchip-drm display-subsystem: bound fec00000.dp (ops cdn_dp_component_ops [rockchipdrm])
[    9.785565] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    9.785648] rockchip-drm display-subsystem: [drm] Cannot find any crtc or sizes
[    9.786921] [drm] Initialized rockchip 1.0.0 20140818 for display-subsystem on minor 1
[    9.790034] cdn-dp fec00000.dp: [drm:cdn_dp_pd_event_work [rockchipdrm]] Not connected. Disabling cdn
[    9.793525] OF: graph: no port node found in /syscon@ff770000/usb2-phy@e450/otg-port
[    9.798887] xhci-hcd xhci-hcd.3.auto: xHCI Host Controller
[    9.798922] xhci-hcd xhci-hcd.3.auto: new USB bus registered, assigned bus number 3
[    9.799102] xhci-hcd xhci-hcd.3.auto: hcc params 0x0220fe64 hci version 0x110 quirks 0x0000000002010010
[    9.799138] xhci-hcd xhci-hcd.3.auto: irq 225, io mem 0xfe800000
[    9.799592] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.08
[    9.799599] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    9.799604] usb usb3: Product: xHCI Host Controller
[    9.799608] usb usb3: Manufacturer: Linux 5.8.14-rockchip64 xhci-hcd
[    9.799612] usb usb3: SerialNumber: xhci-hcd.3.auto
[    9.800225] ehci-platform fe380000.usb: EHCI Host Controller
[    9.800382] hub 3-0:1.0: USB hub found
[    9.800422] hub 3-0:1.0: 1 port detected
[    9.800764] xhci-hcd xhci-hcd.3.auto: xHCI Host Controller
[    9.800783] xhci-hcd xhci-hcd.3.auto: new USB bus registered, assigned bus number 4
[    9.800800] xhci-hcd xhci-hcd.3.auto: Host supports USB 3.0 SuperSpeed
[    9.800984] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    9.801145] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.08
[    9.801151] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    9.801155] usb usb4: Product: xHCI Host Controller
[    9.801160] usb usb4: Manufacturer: Linux 5.8.14-rockchip64 xhci-hcd
[    9.801164] usb usb4: SerialNumber: xhci-hcd.3.auto
[    9.801978] hub 4-0:1.0: USB hub found
[    9.802025] hub 4-0:1.0: 1 port detected
[    9.802536] ehci-platform fe380000.usb: new USB bus registered, assigned bus number 5
[    9.802780] ehci-platform fe380000.usb: irq 30, io mem 0xfe380000
[    9.808110] rockchip_vdec: module is from the staging directory, the quality is unknown, you have been warned.
[    9.810969] hantro_vpu: module is from the staging directory, the quality is unknown, you have been warned.
[    9.811127] rkvdec ff660000.video-codec: Adding to iommu group 1
[    9.813228] hantro-vpu ff650000.video-codec: Adding to iommu group 0
[    9.816939] ehci-platform fe380000.usb: USB 2.0 started, EHCI 1.00
[    9.818455] hantro-vpu ff650000.video-codec: registered rockchip,rk3399-vpu-enc as /dev/video1
[    9.837933] systemd[1]: Finished File System Check on /dev/disk/by-uuid/4f6ebe40-b7d9-4919-bcc8-4421b42bedce.
[    9.844122] usb usb5: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.08
[    9.844131] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    9.844136] usb usb5: Product: EHCI Host Controller
[    9.844140] usb usb5: Manufacturer: Linux 5.8.14-rockchip64 ehci_hcd
[    9.844145] usb usb5: SerialNumber: fe380000.usb
[    9.847049] hub 5-0:1.0: USB hub found
[    9.847109] hub 5-0:1.0: 1 port detected
[    9.848552] ohci-platform fe3a0000.usb: Generic Platform OHCI controller
[    9.848590] ohci-platform fe3a0000.usb: new USB bus registered, assigned bus number 6
[    9.848744] ohci-platform fe3a0000.usb: irq 31, io mem 0xfe3a0000
[    9.849139] hantro-vpu ff650000.video-codec: registered rockchip,rk3399-vpu-dec as /dev/video3
[    9.860241] systemd[1]: Mounting /media/mmcboot...
[    9.878386] EXT4-fs (mmcblk1p1): mounted filesystem with ordered data mode. Opts: commit=600,errors=remount-ro
[    9.878427] ext4 filesystem being mounted at /media/mmcboot supports timestamps until 2038 (0x7fffffff)
[    9.888659] systemd[1]: Mounted /media/mmcboot.
[    9.898653] systemd[1]: Mounting /boot...
[    9.909277] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 5.08
[    9.909287] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    9.909291] usb usb6: Product: Generic Platform OHCI controller
[    9.909297] usb usb6: Manufacturer: Linux 5.8.14-rockchip64 ohci_hcd
[    9.909301] usb usb6: SerialNumber: fe3a0000.usb
[    9.910085] hub 6-0:1.0: USB hub found
[    9.910125] hub 6-0:1.0: 1 port detected
[    9.916725] systemd[1]: Mounted /boot.
[   10.180609] systemd[1]: Found device Corsair_Force_3_SSD 1.
[   10.183772] usb 2-1.4: reset SuperSpeed Gen 1 USB device number 3 using xhci-hcd
[   10.188584] systemd[1]: Found device Corsair_Force_3_SSD 1.
[   10.197291] systemd[1]: Found device Micron_5210_MTFDDAK7T6QDE 1.
[   10.199806] systemd[1]: Mounting /mnt/media...
[   10.203600] systemd[1]: Mounting /mnt/transmission...
[   10.206664] systemd[1]: Mounting /mnt/transmission-incomplete...
[   10.211862] BTRFS info (device sda1): use zstd compression, level 3
[   10.211870] BTRFS info (device sda1): disk space caching is enabled
[   10.211874] BTRFS info (device sda1): has skinny extents
[   10.212435] BTRFS info (device sdc1): use zstd compression, level 3
[   10.212440] BTRFS info (device sdc1): disk space caching is enabled
[   10.212443] BTRFS info (device sdc1): has skinny extents
[   10.215303] BTRFS info (device sdb1): use zstd compression, level 3
[   10.215312] BTRFS info (device sdb1): disk space caching is enabled
[   10.215316] BTRFS info (device sdb1): has skinny extents
[   10.221189] systemd[1]: Found device INTEL_SSDSC2BW180A3L 1.
[   10.224283] systemd[1]: Mounting /intel520-180gb...
[   10.259933] BTRFS info (device sdd1): setting incompat feature flag for COMPRESS_ZSTD (0x10)
[   10.259945] BTRFS info (device sdd1): use zstd compression, level 3
[   10.259953] BTRFS info (device sdd1): disk space caching is enabled
[   10.259960] BTRFS info (device sdd1): has skinny extents
[   10.265745] BTRFS info (device sdd1): enabling ssd optimizations
[   10.272490] systemd[1]: Mounted /intel520-180gb.
[   10.272773] BTRFS info (device sdb1): enabling ssd optimizations
[   10.274793] BTRFS info (device sdc1): enabling ssd optimizations
[   10.283082] systemd[1]: Mounted /mnt/transmission.
[   10.283732] systemd[1]: Mounted /mnt/transmission-incomplete.
[   10.317322] r8152 2-1.4:1.0 (unnamed net_device) (uninitialized): netif_napi_add() called with weight 256
[   10.319451] r8152 2-1.4:1.0 eth1: v2.13.0 (2020/04/20)
[   10.319464] r8152 2-1.4:1.0 eth1: This product is covered by one or more of the following patents:
                       US6,570,884, US6,115,776, and US6,327,625.

[   10.319640] usbcore: registered new interface driver r8152
[   10.817037] rockchip-drm display-subsystem: [drm] Cannot find any crtc or sizes
[   11.694873] random: crng init done
[   11.694883] random: 7 urandom warning(s) missed due to ratelimiting
[   11.699323] systemd[1]: Finished Load/Save Random Seed.
[   12.178193] BTRFS info (device sda1): enabling ssd optimizations
[   12.192651] systemd[1]: Mounted /mnt/media.
[   12.193208] systemd[1]: Reached target Local File Systems.
[   12.196018] systemd[1]: Starting Armbian ZRAM config...
[   12.199134] systemd[1]: Starting Set console font and keymap...
[   12.202670] systemd[1]: Starting Raise network interfaces...
[   12.206181] systemd[1]: Starting Preprocess NFS configuration...
[   12.206598] systemd[1]: Condition check resulted in Mark the need to relabel after reboot being skipped.
[   12.206781] systemd[1]: Condition check resulted in Store a System Token in an EFI Variable being skipped.
[   12.206902] systemd[1]: Condition check resulted in Commit a transient machine-id on disk being skipped.
[   12.217188] systemd[1]: Finished Set console font and keymap.
[   12.218174] systemd[1]: nfs-config.service: Succeeded.
[   12.219140] systemd[1]: Finished Preprocess NFS configuration.
[   12.222217] systemd[1]: Starting NFSv4 ID-name mapping service...
[   12.222593] systemd[1]: Condition check resulted in RPC security service for NFS client and server being skipped.
[   12.222739] systemd[1]: Reached target NFS client services.
[   12.222955] systemd[1]: Condition check resulted in RPC security service for NFS server being skipped.
[   12.238998] systemd[1]: Started NFSv4 ID-name mapping service.
[   12.314868] systemd[1]: Finished Armbian ZRAM config.
[   12.318076] systemd[1]: Starting Armbian memory supported logging...
[   12.502911] systemd[1]: Finished Raise network interfaces.
[   14.648283] systemd[1]: Finished Armbian memory supported logging.
[   14.656352] systemd[1]: Starting Journal Service...
[   14.834720] systemd[1]: Started Journal Service.
[   14.853556] systemd-journald[1063]: Received client request to flush runtime journal.

### armbian-release:

# PLEASE DO NOT EDIT THIS FILE
BOARD=helios64
BOARD_NAME="Helios64"
BOARDFAMILY=rk3399
BUILD_REPOSITORY_URL=https://github.com/armbian/build
BUILD_REPOSITORY_COMMIT=0443bb08-dirty
DISTRIBUTION_CODENAME=focal
DISTRIBUTION_STATUS=supported
VERSION=20.08.10
LINUXFAMILY=rockchip64
BRANCH=current
ARCH=arm64
IMAGE_TYPE=stable
BOARD_TYPE=wip
INITRD_ARCH=arm64
KERNEL_IMAGE_TYPE=Image

### boot environment:

verbosity=1
bootlogo=false
overlay_prefix=rockchip
rootdev=UUID=9d58c047-579d-469d-bdee-16cb4f219614
rootfstype=btrfs
usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u

### lsusb:

Bus 002 Device 003: ID 0bda:8156 Realtek Semiconductor Corp.
Bus 002 Device 002: ID 2109:0815 VIA Labs, Inc. USB3.0 Hub             
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2109:2815 VIA Labs, Inc. USB2.0 Hub             
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

### lscpu:

Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          6
On-line CPU(s) list:             0-5
Thread(s) per core:              1
Core(s) per socket:              3
Socket(s):                       2
NUMA node(s):                    1
Vendor ID:                       ARM
Model:                           4
Model name:                      Cortex-A53
Stepping:                        r0p4
CPU max MHz:                     1800.0000
CPU min MHz:                     408.0000
BogoMIPS:                        48.00
NUMA node0 CPU(s):               0-5
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Vulnerable
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid

### cpuinfo:

processor    : 0
BogoMIPS    : 48.00
Features    : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer    : 0x41
CPU architecture: 8
CPU variant    : 0x0
CPU part    : 0xd03
CPU revision    : 4

processor    : 1
BogoMIPS    : 48.00
Features    : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer    : 0x41
CPU architecture: 8
CPU variant    : 0x0
CPU part    : 0xd03
CPU revision    : 4

processor    : 2
BogoMIPS    : 48.00
Features    : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer    : 0x41
CPU architecture: 8
CPU variant    : 0x0
CPU part    : 0xd03
CPU revision    : 4

processor    : 3
BogoMIPS    : 48.00
Features    : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer    : 0x41
CPU architecture: 8
CPU variant    : 0x0
CPU part    : 0xd03
CPU revision    : 4

processor    : 4
BogoMIPS    : 48.00
Features    : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer    : 0x41
CPU architecture: 8
CPU variant    : 0x0
CPU part    : 0xd08
CPU revision    : 2

processor    : 5
BogoMIPS    : 48.00
Features    : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer    : 0x41
CPU architecture: 8
CPU variant    : 0x0
CPU part    : 0xd08
CPU revision    : 2

### meminfo:

MemTotal:        3891268 kB
MemFree:         3601388 kB
MemAvailable:    3612124 kB
Buffers:            3160 kB
Cached:           143024 kB
SwapCached:            0 kB
Active:            93808 kB
Inactive:          70768 kB
Active(anon):      19396 kB
Inactive(anon):    14212 kB
Active(file):      74412 kB
Inactive(file):    56556 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                56 kB
Writeback:             0 kB
AnonPages:         19240 kB
Mapped:            19908 kB
Shmem:             14776 kB
KReclaimable:      30096 kB
Slab:              86472 kB
SReclaimable:      30096 kB
SUnreclaim:        56376 kB
KernelStack:        4480 kB
PageTables:          984 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1945632 kB
Committed_AS:      95040 kB
VmallocTotal:   135290159040 kB
VmallocUsed:       11168 kB
VmallocChunk:          0 kB
Percpu:             2352 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:         131072 kB
CmaFree:          124416 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

### virtual memory:

vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.compact_unevictable_allowed = 1
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirtytime_expire_seconds = 43200
vm.dirty_writeback_centisecs = 500
vm.extfrag_threshold = 500
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256    256    32    0
vm.max_map_count = 65530
vm.min_free_kbytes = 45056
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 32768
vm.mmap_rnd_bits = 18
vm.mmap_rnd_compat_bits = 11
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0
vm.numa_stat = 1
vm.numa_zonelist_order = Node
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.stat_interval = 1
vm.swappiness = 100
vm.user_reserve_kbytes = 120492
vm.vfs_cache_pressure = 100
vm.watermark_boost_factor = 15000
vm.watermark_scale_factor = 10
vm.zone_reclaim_mode = 0

### ip addr:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 64:62:66:d0:04:f0 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 64:62:66:d0:04:f1 brd ff:ff:ff:ff:ff:ff
 
### partitions:

major minor  #blocks  name

 179        0   15267840 mmcblk1
 179        1      98304 mmcblk1p1
 179        2   15000464 mmcblk1p2
   8        0 7501465944 sda
   8        1 7501464903 sda1
   8       16  234431064 sdb
   8       17  234430023 sdb1
   8       32  234431064 sdc
   8       33  234430023 sdc1
   8       48  175825944 sdd
   8       49  175824920 sdd1
 251        0      43692 zram0
 251        1     324272 zram1
 251        2      43692 zram2
 251        3      43692 zram3
 251        4      43692 zram4
 251        5     324272 zram5

### df:

udev            1.8G     0  1.8G   0% /dev
tmpfs           381M  5.9M  375M   2% /run
/dev/mmcblk1p2   15G  1.3G   13G  10% /
tmpfs           1.9G  8.0K  1.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs           1.9G  4.0K  1.9G   1% /tmp
/dev/mmcblk1p1   89M   70M   13M  85% /media/mmcboot
/dev/sdd1       168G  3.4M  168G   1% /intel520-180gb
/dev/sdb1       224G   41G  183G  19% /mnt/transmission
/dev/sdc1       224G   22G  201G  10% /mnt/transmission-incomplete
/dev/sda1       7.0T  5.2T  1.9T  74% /mnt/media

### lsblk:

NAME         FSTYPE   SIZE MOUNTPOINT                   UUID
sda                     7T                              
└─sda1       btrfs      7T /mnt/media                   dd84e5f9-e112-4c70-a4a7-b4a785d69490
sdb                 223.6G                              
└─sdb1       btrfs  223.6G /mnt/transmission            fccb0287-8cf9-4852-a859-5dc57ecc8a66
sdc                 223.6G                              
└─sdc1       btrfs  223.6G /mnt/transmission-incomplete a587760c-2c3f-40a3-99b6-52e67edeebc1
sdd                 167.7G                              
└─sdd1       btrfs  167.7G /intel520-180gb              8ea91444-ba20-47a0-9932-e6a07c974e97
mmcblk1              14.6G                              
├─mmcblk1p1  ext4      96M /media/mmcboot               4f6ebe40-b7d9-4919-bcc8-4421b42bedce
└─mmcblk1p2  btrfs   14.3G /                            9d58c047-579d-469d-bdee-16cb4f219614
mmcblk1boot0            4M                              
mmcblk1boot1            4M                              

### zramctl:

NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram5 lzo-rle     316.7M   4K   73B   12K       6 [SWAP]
/dev/zram4 lzo-rle      42.7M   4K   73B   12K       6 [SWAP]
/dev/zram3 lzo-rle      42.7M   4K   73B   12K       6 [SWAP]
/dev/zram2 lzo-rle      42.7M   4K   73B   12K       6 [SWAP]
/dev/zram1 lzo-rle     316.7M   4K   73B   12K       6 [SWAP]
/dev/zram0 lzo-rle      42.7M   4K   73B   12K       6 [SWAP]

### mtab:

/dev/mmcblk1p1 /boot ext4 rw,noatime,nodiratime,errors=remount-ro,commit=600 0 0
/dev/mmcblk1p1 /media/mmcboot ext4 rw,noatime,nodiratime,errors=remount-ro,commit=600 0 0
/dev/mmcblk1p2 / btrfs rw,noatime,nodiratime,compress=lzo,ssd,space_cache,commit=600,subvolid=5,subvol=/ 0 0
/dev/mmcblk1p2 /var/log.hdd btrfs rw,noatime,nodiratime,compress=lzo,ssd,space_cache,commit=600,subvolid=5,subvol=/ 0 0
/dev/sda1 /mnt/media btrfs rw,noatime,nodiratime,compress=zstd:3,ssd,space_cache,subvolid=5,subvol=/ 0 0
/dev/sdb1 /mnt/transmission btrfs rw,noatime,nodiratime,compress=zstd:3,ssd,space_cache,subvolid=5,subvol=/ 0 0
/dev/sdc1 /mnt/transmission-incomplete btrfs rw,noatime,nodiratime,compress=zstd:3,ssd,space_cache,subvolid=5,subvol=/ 0 0
/dev/sdd1 /intel520-180gb btrfs rw,noatime,nodiratime,compress=zstd:3,ssd,space_cache,subvolid=5,subvol=/ 0 0

### mmc1:0001 info:

               fwrev: 0x0600000000000000
              uevent: DRIVER=mmcblk MMC_TYPE=MMC MMC_NAME=AJTD4R MODALIAS=mmc:block
                 cid: 150100414a54443452063c785a991700
         rel_sectors: 0x1
enhanced_rpmb_supported: 0x1
                 rca: 0x0001
        pre_eol_info: 0x01
                 csd: d02701320f5903fff6dbffef8e404000
              manfid: 0x000015
                 ocr: 0x00000080
preferred_erase_size: 4194304
enhanced_area_offset: 18446744073709551594
  raw_rpmb_size_mult: 0x20
                type: MMC
               hwrev: 0x0
                date: 01/2020
                 prv: 0x6
         ffu_capable: 1
                 dsr: 0x404
          erase_size: 524288
           life_time: 0x01 0x01
  enhanced_area_size: 4294967274
               oemid: 0x0100
              serial: 0x3c785a99
             cmdq_en: 1
                 rev: 0x8
                name: AJTD4R

### Boot system health:

Time       big.LITTLE   load %cpu %sys %usr %nice %io %irq   CPU  C.St.
19:25:34: 1200/1200MHz  2.42  41%  23%  16%   0%   0%   0% 50.0°C  0/5
19:25:35: 1200/1200MHz  2.42  86%  17%  66%   0%   0%   1% 50.6°C  0/5
19:25:35: 1200/1200MHz  2.42  86%  17%  65%   1%   0%   2% 50.0°C  0/5
19:25:36: 1200/1200MHz  2.42  81%  22%  58%   1%   0%   0% 50.6°C  0/5
19:25:36: 1200/1200MHz  2.42  64%  12%  50%   0%   0%   0% 51.7°C  0/5

It seems like when I do a lot of I/O and put a high load on the cpu, the system just freezes. There's nothing in the syslog but I haven't looked in the serial console.

I'm only running SSD's in this machine, so there's no shortage of bandwidth on the drives. If there's no load, I haven't had one freeze at all. Medium load doesn't seem to be an issue either.

Has anyone tried adding active cooling on the cpu?

I'm guessing this is the same issue that was posted about on the 27th of October in the blog, why hasn't there been an update on it?

 

And for the mods: how can I remove this forum posting limitation, like one post per week or something?

[SR-G]

@Seneca

 

I would suggest to upgrade to kernel 5.9.14 : i have freezes with both, but i had way more freezes with 5.8.14 compared to 5.9.14

 

Otherwise, on my side, it is 100% clear that in my situation the freezes are not related to disk I/O : i have nearly nothing in crontab, and i'm encountering some freezes during the night, with 0 CPU activity / 0 disk usage (and as said before, i have nearly no processes configured outside MDADM, SSH, and NETDATA).

[Seneca]

5 minutes ago, SR-G said:

@Seneca

 

I would suggest to upgrade to kernel 5.9.14 : i have freezes with both, but i had way more freezes with 5.8.14 compared to 5.9.14

 

Otherwise, on my side, it is 100% clear that in my situation the freezes are not related to disk I/O : i have nearly nothing in crontab, and i'm encountering some freezes during the night, with 0 CPU activity / 0 disk usage (and as said before, i have nearly no processes configured outside MDADM, SSH, and NETDATA).

Are you getting data corruption due to the freezes, I mean broken raids or checksum errors if it's zfs?

I can provoke a freeze by loading with a lot of IO+CPU at will, so I think the dev's have a real challenge ahead both separating the issues and troubleshooting them.

I've lowered the cpufreq min/max to 800mhz now, and max download speed to 7MB/s, hoping that will perhaps help.