[PINE64 A64] Kernel panics with headless boot

[Guest]

Hi,

I've opened with tag orangepiprime because there was no a64 tag eventhough in the Armbian site is flagged as standard support, tell me if i have to reopen it elsewhere.

Anyhow I'm constantly getting kernel panics during boot, I've tested images minimal/cli images starting from 23.5.1 down to 24.5.0 (just downloaded the image and flashed with balena etcher), attached there are all the serial logs of the boot processes, all of them are the same more or less.

I've tried:

• powering from GPIO
• changing powersupply
• changing usb cable
• 4 different uSD cards
• changing the board
• hooking it up to a display

and none of them worked.

What else can I try?

Armbian_24.5.0-trunk.223_Pine64_trixie_current_6.6.22_minimal.img.xz.log Armbian_24.2.1_Pine64_bookworm_current_6.6.16_minimal.img.xz.log Armbian_23.11.1_Pine64_bookworm_current_6.1.63.img.xz.log Armbian_23.8.1_Pine64_bookworm_current_6.1.47.img.xz.log Armbian_23.5.1_Pine64_bookworm_current_6.1.30.img.xz.log

Edited March 16 by daniele95100

[SteeMan]

Moved post to unsupported/community maintained. This is not a supported board.

 

My mistake, misread the post.  This is pine64 not pinebook a64.  That board should be supported but the forum structure isn't up to date.  I have added the correct tag for the post, but left it in the original locatation.

Edited March 16 by SteeMan
corrected

[Guest]

Thanks, maybe I've found the problem, by default the param disp_mode is set to 1920x1080p60 in /boot/armbianEnv.txt, setting this to none seems to have fixed the issue. Is it correct for an headless setup? And was actually this setting causing the kernel panics?

In my previous tests the display was 480p while testing with an actual 1080p 60fps display seemed stable.

 

EDIT: it happened again but it succesfully booted this time
 

[  443.082858] Internal error: Oops: 0000000096000004 [#1] SMP
[  443.088462] Modules linked in: lz4hc lz4 zram hci_uart btqca btrtl btintel btbcm r8723bs(C) snd_soc_hdmi_codec bluetooth polyval_ce cfg80211 sunxi_cedrus(C) snd_soc_simple_card ecdh_generic polyval_generic snd_soc_simple_card_utils rfkill sun9i_hdmi_audio libarc4 axp20x_adc ecc lima axp20x_battery axp20x_ac_power sun4i_gpadc_iio gpu_sched sun8i_a33_mbus dw_hdmi_i2s_audio dw_hdmi_cec v4l2_mem2mem drm_shmem_helper sun4i_i2s binfmt_misc videobuf2_dma_contig industrialio videobuf2_memops videobuf2_v4l2 videodev videobuf2_common display_connector mc cpufreq_dt fuse dm_mod pinctrl_axp209 realtek dwmac_sun8i mdio_mux
[  443.142637] CPU: 3 PID: 2172 Comm: modinfo Tainted: G        WC         6.6.16-current-sunxi64 #1
[  443.151520] Hardware name: Pine64+ (DT)
[  443.155366] pstate: 400000c5 (nZcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  443.162338] pc : pick_next_task_fair+0x100/0x3f4
[  443.166989] lr : pick_next_task_fair+0x130/0x3f4
[  443.171624] sp : ffff8000832bbc50
[  443.174950] x29: ffff8000832bbc50 x28: ffff000004c55400 x27: ffff80008105df50
[  443.182100] x26: ffff00003fda1380 x25: ffff80008115d390 x24: ffff00003fda1380
[  443.189251] x23: ffff00003fda1300 x22: ffff8000832bbd08 x21: ffff000004c55400
[  443.196401] x20: ffff0000029c9000 x19: 0000000000000000 x18: ffff8000832bbc58
[  443.203548] x17: 0000000000000000 x16: ffff800080f6bc10 x15: 0000b5b17cbbb478
[  443.210698] x14: 07f353bbb346defe x13: 0000000000000001 x12: 0000000000000001
[  443.217848] x11: 0000000000000001 x10: 00000000000002cc x9 : ffff0000029c9000
[  443.224999] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000002
[ 1043.720544]  0xffff8000814fbe60x15cate:R  running task     stack:0     pid:15    ppid:2      flags:0x0000000acpu=2

 

Edited March 16 by daniele95100
Wrong assumption

[Guest]

Any help on this one? I still can't figure out what's happening here.

[going]

21 час назад, daniele95100 сказал:

Any help on this one?

The only way is to do the assembly yourself.
First, it is necessary to check the correctness of applying patches to the kernel and u-boot.

[c0rnelius]

I suspect it has to do with the devfreq driver for the MBUS/DRAM controller.

https://lore.kernel.org/linux-arm-kernel/a48923b7-12f9-808e-1171-49b826bd5f1c@samsung.com/T/#ma00a3e07248dc7fb2d300b7c9c409f69ffa64c14

 

In my personal experience this introduces instability into the a64/h5. You could try blacklisting the driver and see if it solves the issue. I personally remove the entry from the DTS and create an overlay to add it back if for some reason I need it.

 

The main issue you reported `Kernel panics with headless boot` in my mind points to it being the driver.

[Guest]

Quote

The only way is to do the assembly yourself.
First, it is necessary to check the correctness of applying patches to the kernel and u-boot.

You mean to compile myself an Armbian image?

 

Quote

I suspect it has to do with the devfreq driver for the MBUS/DRAM controller.

https://lore.kernel.org/linux-arm-kernel/a48923b7-12f9-808e-1171-49b826bd5f1c@samsung.com/T/#ma00a3e07248dc7fb2d300b7c9c409f69ffa64c14

 

In my personal experience this introduces instability into the a64/h5. You could try blacklisting the driver and see if it solves the issue. I personally remove the entry from the DTS and create an overlay to add it back if for some reason I need it.

 

The main issue you reported `Kernel panics with headless boot` in my mind points to it being the driver.

Thanks! I'll give it a shot.

[going]

27 минут назад, daniele95100 сказал:

You mean to compile myself an Armbian image?

I think that in the first step it is enough to compile only the kernel and install it into an existing OS.
You have published a stack dump of the kernel panic.

But to say something definite, I need to see this particular kernel source code.

[Guest]

Quote

I think that in the first step it is enough to compile only the kernel and install it into an existing OS.
You have published a stack dump of the kernel panic.

But to say something definite, I need to see this particular kernel source code.

Mmh, ok but how can I do that if it panic before reaching the login?

[Guest]

Hi, sorry for the late reply, tried both the suggestion but the problem still remains.

What I've noticed is that the panic happen at the first boot every time, but then it happens randomly.

In details the process is:

1. first boot panic every times
2. reboot after the first panic
3. complete the setup process
4. the system seems stable until a new panic happens (last time it happened after ~30hrs)

 

[Guest]

Hi, any hint on how to debug this one? This issue makes it very difficult to use those SBCs as an headless server and I would like to use them instead of turning them to e-waste.

 

@c0rnelius maybe I've not disabled the driver correctly, can you give me an hint on how to disable it from the DTS?

Edited April 22 by daniele95100

[c0rnelius]

Try disabling it with an overlay.

 

/dts-v1/;
/plugin/;

/ {
	fragment@0 {
		target = <&mbus>;
		 __overlay__ {
			status = "disabled";
		};
	};
};

 

If that doesn't work, set the wrong compat string.

/dts-v1/;
/plugin/;

/ {
	fragment@0 {
		target = <&mbus>;
		 __overlay__ {
			compatible = "allwinner,sun8i-h3-mbus";
		};
	};
};

 

Use the `armbian-add-overlay` to compile and add the overlay.

 

NOTE: The overlay(s) may not work because of how the current DTS is setup, which is why I suggested blacklisting the driver.

The driver in question is: sun8i-a33-mbus

https://lore.kernel.org/linux-arm-kernel/a48923b7-12f9-808e-1171-49b826bd5f1c@samsung.com/T/#ma00a3e07248dc7fb2d300b7c9c409f69ffa64c14

[Guest]

Hi,

I've tried to blacklist the driver, seemed stable but crashed after 2 weeks. At this  point I don't know what else to try.

For now i set

kernel.panic=10

 in /etc/sysctl.d/96-panic.conf to see if at least reboots when a panic happens.

[Dav]

I don't know how to debug that issue, but I have your same board (Pine A64 with 1GB RAM) and I'm using Armbian 24.5.1 Jammy with Linux 6.1.92-legacy-sunxi64.
It's stable in terms of kernel panics, random freeze or so. You can change the kernel using "armbian-config" > system > other.

I have a private gist for my own, where I add the stable combo of OS version + kernel version, before updating.

I hope it helps

Btw, FYI too:
I only had an issue (from long time ago), with the ethernet port (I guess). It randomly disconnect, so I """solved""" it with mmonit (link here