Regression in CB1 kernels for network drivers general instability

[ressu]

Armbianmonitor:

https://paste.next.armbian.com/okilequzav

A while back I tried the more recent kernels (non-legacy) and noticed these issues. Back then I didn't have the possibility to investigate further and I just ended up forcing the kernel back to old legacy kernel. This time around I have the time to actually troubleshoot more, so here we go.

 

When I last updated everything, the kernel got updated from 6.1.104-legacy-sunxi64 to 6.6.75-legacy-sunxi64, the immediate thing I noticed was that the WLAN did get detected. There is a thread about missing wifi with a linked PR, but either that fix hasn't landed in the 6.6.x kernel or something else is wrong.

 

The other issue is that the 6.6.x kernel seems to be very unstable. Having zram turned on will cause the system to crash within minutes from startup. Disabling zram makes the system more stable, but eventually there is a crash. The reason why I was looking at zram was due to the swapper mention in this message I got on the serial console. Note that the output is slightly corrupted due to my extra shoddy serial console that I cobbled together for emergency access.

 

[  651.303088] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000ec9
[  651.311994] Mem abort info:
[  651.314808]   ESR = 0x0000000096000006
[  651.318589]   EC = 0x25: DABT (current EL),HR
 bits
[  651.323937]   SET = 0, FnV = 0
[  651.327007]   EA = 0, S1PTW = 0
[  651.330202]   FSC = 0x06: level 2 translation fault
[  651.335153] Data abort info:
[  MLi�  ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
[  651.343601]   CM = 0, WnR = 0, Tb= 0
[  651.348705]   GCS = 0Kzٕɱ= 0, DirtyBit = 0, Xs = 0
[  651.354076] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000043760000
[  651.360576] [0000000000000ec9] pgd=0800000043c45003, p4d=080000004,SbՑ0800000043af6003, pmd=0000000000000000
[  651.371253] Inter[X
                      * Oops: 0000000096000006 [#1] SMP
[  651.376843] Modules linked in: lz4hc lz4 zram zsmalloc binfmt_misc nls_iso8859_1 rtl8xxxu mac80211 snd_soc_hdmi_codec cfg80211 rfkill libarc4 polyval_ce polyval_generic cdc_acm sunxi_cir r_ephy sunxi_ac200
[  651.417619] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G         C       $rrjbsunxi64 #1
[  651.426311] Hardware name: BigTreeTech CB1 (DT)
[  651.430848] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  651.437822] pc : n
                     K}}}0x18/0xa4
[  651.442473] lr : do_idle+0x38/0x270
[  651.445975] sp : f�0
[  651.449291] x29: ffff800081cbbe10 x28: 0000000000000000 x27: 0000000000000000
[  651.456445] x26: 0000000000000000 x25: 000000000004: 0000000000000000
[  651.463595] x23: 0000000000000000 x22: 0000000000000000 x21: 000j
[  651.470746] x20: 0000000000000001 x19: 00000000000000L8: 0000000000000000
[  651.477901] x17: 0000000000000000 x16: 0000000000000000 x15: 00000000035ffbb0
[  651.485055] x14: 000000000000028b x13: ffff8000810dde30 x12: ffff800081a29af8
[  651.492206] x11: 0000000000000001 x10: 00000000000009a0 x9 : ffff800081cbbd50C[  651.499359] x8 : ffff0000020a6c00 x7 : ffff00003fd81340 x6 : ffff000003504280
[  651.506510] x5 : 00000000410fd030 x4 : 0000000000000ec1 x3 : 0000000000000j
[  651.513638] x2 : ffff8000817f42c0 x1 : ffff8000817f42c0 x] Code: 913b0084 d000bae2 910b0042 aa0203e1 (f8607883)
[  651.546301] ---[ end trace 0000000000000000 ]---
[  651.551024] Kernel panic - not syncing: Attempted to kill the idle task!
[  651.557741] SMP: stopping secondary CPUs
[  651.561666] Kernel Offset: disabled
[  651.565150] CPU features: 0x0,00000008,00020000,0000420b
[  651.570457] Memory Limit: none
[  651.MӪ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

 

I'll keep the system in this state for a while and figure out a better serial console so that I can collect more clean data. The board is in a printer and is in a bit awkward location for more conventional troubleshooting.

[JohnTheCoolingFan]

Have you tried with current or edge kernels?

[ressu]

I haven't tried the current or edge kernels yet, I'll verify them after I have a more reliable serial bridge in place. The previous time I tried this and saw the same crashes I did try the current kernel, but I'll have a go at the other kernels to verify if I can reproduce the issues there.

[going]

1 час назад, ressu сказал:

Kernel panic - not syncing: Attempted to kill the idle task!

Please throw away this 6.6.75 core. It is no longer supported.
Or fix the regressions yourself.

[ressu]

I switched to the current kernel and the wlan is still missing. I'm still waiting for the kernel panic, but so far even with zram enabled the board has remained crash free for about 30 minutes.

 

An update armbianmonitor log is here: https://paste.next.armbian.com/aculomicah

[ressu]

40 minutes ago, going said:

Please throw away this 6.6.75 core. It is no longer supported.

The reason who I'm calling this a regression is that most CB1 users are going to be in the legacy track. The legacy track just got updated to 6.6.75, which will effectively break all current installations.

[JohnTheCoolingFan]

I'm not working on the legacy kernel at all, current and edge only, which are based on mainline. The fix for wifi should be in the upcoming armbian release (25.05). It was merged into the build repository long ago and should've been available in beta already, only now coming into the release.

To those who are still using legacy - I recommend switching to current or edge instead.

[ressu]

2 minutes ago, JohnTheCoolingFan said:

The fix for wifi should be in the upcoming armbian release (25.05).

 

Ok, I'll wait for the release to land and test again.

 

3 minutes ago, JohnTheCoolingFan said:

To those who are still using legacy - I recommend switching to current or edge instead.

 

I agree, upgrading away from legacy is good. That said, as things are now this will break any installation that is using wlan when they upgrade their packages and are on the legacy kernel. And it's even worse as the user needs to know to update to latest release and only then swap kernels due to the timing of the wlan fix.

 

To put it in different words, I'm not proposing we fix the legacy kernel. I'm saying we need to make sure that we're not breaking current users who have no way of knowing that things are going to be breaking.