How to enable kernel crash dumps to debug kernel panic ?

[mdbananapi]

Hi!

 

Can anyone help me to enable kernel crash dumps option in the menuconfig?

 

When I build  the kernel, and when the menuconfig appears, there is no option of Kernel crash dump. To be more explicit, I need the following options activated ( based on http://superuser.com/questions/280767/how-can-i-enable-kernel-crash-dumps-in-debian  ):

• CONFIG_CRASH_DUMP=y
• CONFIG_KEXEC=y
• CONFIG_DEBUG_INFO=y
• CONFIG_RELOCATABLE=y

I have searched for them with the / feature in the menu config, but there are no such options.

 

I need this in order to debug a device that is calling kernel panic at boot time.

 

Thanks!

[zador.blood.stained]

What kernel version and from what source you are trying to compile? Current 3.4.10x for Banana pi? You linked instructions related to kernel 2.2.x for PC (x64 arch).

From https://github.com/linux-sunxi/linux-sunxi/blob/sunxi-3.4/Documentation/kdump/kdump.txt

Kdump and kexec are currently supported on the x86, x86_64, ppc64, ia64,
and s390x architectures.

But I'm not sure if "supported" here means "can be enabled" or "tested and confirmed working"

config CRASH_DUMP
	bool "Build kdump crash kernel (EXPERIMENTAL)"
	depends on EXPERIMENTAL
	help
	  Generate crash dump after being started by kexec. This should
	  be normally only set in special crash dump kernels which are
	  loaded in the main kernel with kexec-tools into a specially
	  reserved region and then later executed after a crash by
	  kdump/kexec. The crash dump kernel must be compiled to a
	  memory address not used by the main kernel

	  For more details see Documentation/kdump/kdump.txt

CONFIG_CRASH_DUMP option is present and depends on CONFIG_EXPERIMENTAL, which is enabled in Igor's linux-sun7i.config, so it should be available.

[mdbananapi]

Hi!

 

Thank you for your reply. The current kernel is kernel 4.2.5-sunxi. I compiled it using the sources from https://github.com/igorpecovnik/lib

 

So as it is explained in https://github.com/linux-sunxi/linux-sunxi/blob/sunxi-3.4/Documentation/kdump/kdump.txtI need to install the kexec-tools and later to compile a new kernel with the options showed there, enabled ?

 

Wouldn't it be better if I compile a new kernel first, then install it on my sd card, and after this install the kexec-tools ?

 

Is there a way to enable the debug method without compiling and installing a new kernel?

 

Thanks!

[zador.blood.stained]

If kernel panic is caused by kernel module, you can try to recompile only that module (adding debug options or extra debug print commands) and load it with maximum log level. What device and kernel module are we talking about?

Build the system and dump-capture kernels
-----------------------------------------
There are two possible methods of using Kdump.

1) Build a separate custom dump-capture kernel for capturing the
   kernel core dump.

2) Or use the system kernel binary itself as dump-capture kernel and there is
   no need to build a separate dump-capture kernel. This is possible
   only with the architectures which support a relocatable kernel. As
   of today, i386, x86_64, ppc64, ia64 and arm architectures support relocatable
   kernel.

Looks like you'll have to build separate kernel and load it with from u-boot console with some parameters (something like this): https://help.ubuntu.com/12.04/serverguide/kernel-crash-dump.html

Looks like Arm arch is supported since kernel version 3.17, so you still need to build new kernel with right options, but you can use kexec-tools.

[mdbananapi]

I get the kernel panic only when I connect the smi2021 easycap usb device ( http://linuxtv.org/wiki/index.php/Easycap ). The module is smi2021 from https://github.com/Manouchehri/smi2021 . I have compiled it on my laptop and it works great, the device and module is working as it should. But on my banana pi it gives me the kernel panic. I have enabled all dependencies in kernel config.

 

One of the developers of the module said that this issue is caused by the module not being able to find the firmware of the device, and not the firmware itself. Here is what he said:

 

From what you write in the thread on github, your problem is not with
the firmware.
When the smi2021 driver can't find any firmware, the driver doesn't do
anything after printing these lines:

  usb 2-1: could not load any firmware for this device
  smi2021: probe of 2-1:1.0 failed with error -2

But when you add the firmware to the correct path, the driver tries to
initialize the smi2021 device over the usb bus.
It looks like your kernel panics at this stage, but you need to get a
better crash dump.

 

I have opened an issue on this in the github community for this module: https://github.com/Manouchehri/smi2021/issues/21

 

If the module would have found the firmware, and the device would have been working fine, in dmesg I should have got a message link this: https://github.com/Manouchehri/smi2021/issues/20#issuecomment-142331645

[zador.blood.stained]

If this is an USB device (that's the reason I asked about it) you may be able to use USB sniffer (usbmon + wireshark for example) to test if it tries to access the device or crashes before that.

Did you compile this module on banana itself or cross-compiled in another environment?

Edit: It also would be helpful if you connected UART adapter to banana and captured full kernel dmesg output with it.

[mdbananapi]

I have compiled the module on banana pi.

 

Regarding the usbmon, from what I have read, I think it does not record usb data at boot time, because it is not even loaded then. 

 

But I can only insert the easycap usb device only at boot time because if I insert it while the banana pi is working, it freezes. And even the lights on banana pi are stopping from working if I insert the usb in it while the banana is on. I think this issue is not related to the usb device but rather to the banana pi power supply. I will try to connect the usb device using an powered usb hub, with the red cable, responsible for power, cut, so it would not back power the banana pi.

[zador.blood.stained]

Ok, right now I think getting kernel panic messages and messages right before that via UART should be more helpful in understanging at what stage driver crashes.

[mdbananapi]

Hi!

 

My UART usb to ttl adapter came and I connected it successfully to the banana pi and it boots nicely to debian, but it does not show the kernel panic. The output of full kernel loading and output of dmesg is at http://pastebin.com/3iaMAvf5

 

I still think that I need to use kernel debugging to catch the error. I have followed your advice and installed kexec-tools but it does noting for debugging.

 

I found a really good tutorial on how to do kernel debugging with kdump and says that I need to compile 2 kernels, and when the kernel panic occurs, the kdump, via the kexec loads the second kernel and stores the error message. And after this I can read the message. But in order for this to work, in that tutorial it says I need to be able to boot a second kernel, via dual booting, which from my research, it is not possible on banana pi. I have configured my kernel and enabled kdump.

 

Here is the tutorial: http://www.dedoimedo.com/computers/kdump.html

 

Anyway, have you, or someone else managed to do kernel debugging to capture the messages of kernel panic? if so, can you please share the how to?

 

Thanks!

[zador.blood.stained]

You are missing kernel arguments, there are no messages between u-boot and login prompt.

[    0.000000] Kernel command line: console=tty1 root=/dev/mmcblk0p1 rootwait rootfstype=ext4 sunxi_ve_mem_reserve=0 sunxi_g2d_mem_reserve=0 sunxi_no_mali_mem_reserve sunxi_fb_mem_reserve=16 hdmi.audio=EDID:0 disp.screen0_output_mode=1920x1080p60 panic=10 consoleblank=0 enforcing=0 loglevel=1

You need to add (or change) and recompile boot.cmd/boot.scr, adding

console=ttyS0,115200 

to kernel command line

 

Edit: and you may want to change loglevel=1 to loglevel=7, but be warned, that you'll need to set huge scrollback buffer size or logging to file.

[splite]

Hey,

 

I'm experimenting crash on Lime2 (Kernel 4.6.2-sunxi and U-boot 2016.05). I've got no error in kern.log, message, syslog, etc.

I don't know exactly what is the best way to fellow in order to get some Kernel report...

 

Could you advise me with the best way ? Kdump or something else

 

Thanks