Kernel panics on reboot

[meymarce]

I am having the issue that after some time running (like a day). Rebooting results in crashing during kernel loading. Than I have to hardreset the system and it boots. If I reboot right away again, no issue.

Spoiler

[62387.280373] reboot: Restarting system
DDR Version 1.24 20191016
In
soft reset
SRX
channel 0
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x10
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 1
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x10
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 0 training pass!
channel 1 training pass!
change freq to 416MHz 0,1
Channel 0: LPDDR4,416MHz
Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
Channel 1: LPDDR4,416MHz
Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
256B stride
channel 0
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x10
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 1
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x10
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 0 training pass!
channel 1 training pass!
channel 0, cs 0, advanced training done
channel 1, cs 0, advanced training done
change freq to 856MHz 1,0
ch 0 ddrconfig = 0x101, ddrsize = 0x40
ch 1 ddrconfig = 0x101, ddrsize = 0x40
pmugrf_os_reg[2] = 0x32C1F2C1, stride = 0xD
ddr_set_rate to 328MHZ
ddr_set_rate to 666MHZ
ddr_set_rate to 928MHZ
channel 0, cs 0, advanced training done
channel 1, cs 0, advanced training done
ddr_set_rate to 416MHZ, ctl_index 0
ddr_set_rate to 856MHZ, ctl_index 1
support 416 856 328 666 928 MHz, current 856MHz
OUT
Boot1: 2019-03-14, version: 1.19
CPUId = 0x0
ChipType = 0x10, 324
SdmmcInit=2 0
BootCapSize=100000
UserCapSize=14910MB
FwPartOffset=2000 , 100000
mmc0:cmd8,20
mmc0:cmd5,20
mmc0:cmd55,20
mmc0:cmd1,20
mmc0:cmd8,20
mmc0:cmd5,20
mmc0:cmd55,20
mmc0:cmd1,20
mmc0:cmd8,20
mmc0:cmd5,20
mmc0:cmd55,20
mmc0:cmd1,20
SdmmcInit=0 1
StorageInit ok = 68410
SecureMode = 0
SecureInit read PBA: 0x4
SecureInit read PBA: 0x404
SecureInit read PBA: 0x804
SecureInit read PBA: 0xc04
SecureInit read PBA: 0x1004
SecureInit read PBA: 0x1404
SecureInit read PBA: 0x1804
SecureInit read PBA: 0x1c04
SecureInit ret = 0, SecureMode = 0
atags_set_bootdev: ret:(0)
GPT 0x3380ec0 signature is wrong
recovery gpt...
GPT 0x3380ec0 signature is wrong
recovery gpt fail!
LoadTrust Addr:0x4000
No find bl30.bin
No find bl32.bin
Load uboot, ReadLba = 2000
Load OK, addr=0x200000, size=0xe5b60
RunBL31 0x40000
NOTICE:  BL31: v1.3(debug):42583b6
NOTICE:  BL31: Built : 07:55:13, Oct 15 2019
NOTICE:  BL31: Rockchip release version: v1.1
INFO:    GICv3 with legacy support detected. ARM GICV3 driver initialized in EL3
INFO:    Using opteed sec cpu_context!
INFO:    boot cpu mask: 0
INFO:    plat_rockchip_pmu_init(1190): pd status 3e
INFO:    BL31: Initializing runtime services
WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
ERROR:   Error initializing runtime service opteed_fast
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x200000
INFO:    SPSR = 0x3c9

U-Boot 2020.10-armbian (Mar 08 2021 - 14:54:58 +0000)

SoC: Rockchip rk3399
Reset cause: RST
DRAM:  3.9 GiB
PMIC:  RK808
SF: Detected w25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
MMC:   mmc@fe320000: 1, sdhci@fe330000: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Model: Helios64
Revision: 1.2 - 4GB non ECC
Net:   eth0: ethernet@fe300000
scanning bus for devices...
starting USB...
Bus usb@fe380000: USB EHCI 1.00
Bus dwc3: usb maximum-speed not found
Register 2000140 NbrPorts 2
Starting the controller
USB XHCI 1.10
scanning bus usb@fe380000 for devices... 1 USB Device(s) found
scanning bus dwc3 for devices... cannot reset port 4!?
4 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0
Card did not respond to voltage select!
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot/boot.scr
3278 bytes read in 16 ms (199.2 KiB/s)
## Executing script at 00500000
dev: vdd_log @ vdd-log
dev: vdd_center @ DCDC_REG1
Boot script loaded from mmc 0
166 bytes read in 13 ms (11.7 KiB/s)
13851967 bytes read in 1335 ms (9.9 MiB/s)
28582400 bytes read in 2730 ms (10 MiB/s)
81913 bytes read in 43 ms (1.8 MiB/s)
2698 bytes read in 30 ms (86.9 KiB/s)
Applying kernel provided DT fixup script (rockchip-fixup.scr)
## Executing script at 09000000
Moving Image from 0x2080000 to 0x2200000, end=3de0000
## Loading init Ramdisk from Legacy Image at 06000000 ...
   Image Name:   uInitrd
   Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
   Data Size:    13851903 Bytes = 13.2 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 01f00000
   Booting using the fdt blob at 0x1f00000
   Loading Ramdisk to f51b9000, end f5eeecff ... OK
   Loading Device Tree to 00000000f513c000, end 00000000f51b8fff ... OK

Starting kernel ...

[   28.598426] Internal error: Oops: 96000005 [#1] PREEMPT SMP
[   28.598935] Modules linked in: zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zcommon(POE) znvpair(POE) zavl(POE) icp(POE) spl(OE) r8152 snd_soc_hdmi_codec snd_soc_rockchip_i2s snd_soc_core leds_pwm snd_pcm_dmaengine panfrost rockchip_vdec(C) hantro_vpu(C) pwm_fan gpio_charger rockchipdrm v4l2_h264 rockchip_rga fusb302 snd_pcm dw_mipi_dsi videobuf2_dma_contig dw_hdmi v4l2_mem2mem gpu_sched tcpm analogix_dp videobuf2_dma_sg videobuf2_vmalloc typec videobuf2_memops drm_kms_helper videobuf2_v4l2 snd_timer cec snd videobuf2_common soundcore rc_core videodev mc drm drm_panel_orientation_quirks sg gpio_beeper cpufreq_dt ledtrig_netdev lm75 ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod realtek dwmac_rk stmmac_platform stmmac pcs_xpcs adc_keys
[   28.605559] CPU: 5 PID: 0 Comm: swapper/5 Tainted: P         C OE     5.10.21-rockchip64 #21.02.3
[   28.606343] Hardware name: Helios64 (DT)
[   28.606700] pstate: a0000085 (NzCv daIf -PAN -UAO -TCO BTYPE=--)
[   28.607246] pc : rcu_nmi_enter+0x34/0x150
[   28.607608] lr : rcu_nmi_enter+0x28/0x150
[   28.607967] sp : ffff800011c13f30
[   28.608264] x29: ffff800011c13f30 x28: ffff000040311d00
[   28.608744] x27: ffff000040004300 x26: ffff800011cf3de0
[   28.609222] x25: 0000000000000001 x24: 0000000000000000
[   28.609699] x23: 0000000000000001 x22: ffff80001157de38
[   28.610177] x21: ffff8000118b99c8 x20: ffff8000e6248080
[   28.610654] x19: 0000000000000080 x18: 0000000000000000
[   28.611131] x17: 00028ac100ae65dc x16: 0002e76c96f3d670
[   28.611608] x15: 00000120adc31936 x14: 00000000000000e9
[   28.612085] x13: 00000000000000e9 x12: 0000000000000001
[   28.612562] x11: 0000000000000000 x10: 0000000000000a40
[   28.613040] x9 : ffff800011cf3e80 x8 : ffff0000403127a0
[   28.613517] x7 : ffff0000f77c7d80 x6 : 0000000048439f7f
[   28.613994] x5 : 00ffffffffffffff x4 : ffff8000e6248000
[   28.614471] x3 : 0000000000000001 x2 : ffff80001156a000
[   28.614948] x1 : ffff8000112a16a8 x0 : ffff8000e6248000
[   28.615425] Call trace:
[   28.615653]  rcu_nmi_enter+0x34/0x150
[   28.615984]  rcu_irq_enter+0x10/0x70
[   28.616309]  irq_enter+0x10/0x20
[   28.616604]  __handle_domain_irq+0x70/0x108
[   28.616981]  gic_handle_irq+0xc0/0x140
[   28.617320]  el1_irq+0xc0/0x180
[   28.617606]  arch_cpu_idle+0x18/0x28
[   28.617928]  default_idle_call+0x44/0x1bc
[   28.618288]  do_idle+0x204/0x278
[   28.618580]  cpu_startup_entry+0x28/0x60
[   28.618935]  secondary_start_kernel+0x170/0x180
[   28.619346] Code: 940000d1 aa1303f4 f8605aa0 8b000294 (f9407e80)
[   28.619897] ---[ end trace 91efb583191134bd ]---
[   28.620312] Kernel panic - not syncing: Attempted to kill the idle task!
[   28.620908] SMP: stopping secondary CPUs
[   28.621266] Kernel Offset: disabled
[   28.621581] CPU features: 0x0240022,6100200c
[   28.621962] Memory Limit: none
[   28.622250] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

 

I used to have quite a few stability issues, which are now all solved with the voltage fix applied. Running stock frequencies with on-demand.

 

Anyone having any ideas?

 

I hope the Kobol team is enjoying some time off though I still see you around. Keep recharging, you really deserve it.