How to bisect the kernel

[skandalfo]

Hi. I'm not sure if this is the right place to ask.

 

I was basically running my Radxa Rock 5 ITX home server on Armbian, on the edge upstream kernel for a long time. 

 

Somewhere during the 6.19 release candidate kernel series an instability arised, where the machine would hard lock "at random", so I switched to the current kernel since, which to this day remains on unaffected 6.18.

 

But from time to time I try with edge again, and the instability is still present with the 7.1 upstream kernels. Moreover, with the load the machine has I can consistently trigger the lock up just by ssh-ing into the machine once it's been up for 10 minutes or so. I've since enabled the HW watchdog timer, which helps a bit mitigate the frustration here... 

 

I guess at some point current will move past 6.19, and I really would like to have the improved hardware support for newer kernels anyway. 

 

I would like to be able to bisect the kernel sources to find out where things broke down. I think I understand enough of the build system to generate edge kernel packages, but I don't know how to do so while bisecting the upstream kernel sources. I also couldn't find documentation about how to trigger the build system to build the kernel at a particular commit.

 

Is there any particular guidance on how to go with this? 

 

Thanks.

[Werner]

2 hours ago, skandalfo said:

6.19

Oh well, that's quite some time ago. The patchset used back then has been removed/renamed already to move on.

Look at 6df6d0d607abfd59169a0ef2fddbed5fcd5b58f9

 

The question is has this introduced upstream or with one of the patches Armbian puts on top. Quite a journey to dig through that.

 

[skandalfo]

Uhm... What repository is that commit hash for? I tried armbian/build and armbian/linux and a couple of others but it doesn't seem to belong there.

 

I don't expect this to be a single-day task. I'm OK to invest the time in finding out what happened and run a bunch of bisections, even if that needs to include both Armbian tool repos and the Linux kernel repo. I just need some pointers to what the right approach is here and what kind of scripts/setup I should be running for each iteration. I would expect having this procedure documented can help other people too, so I'd be happy to do some write up for it.

 

I can think that:

1.  Maybe there's a way to provide a kernel repo commit to check out to the build infrastructure and get kernel packages built from that. I should be able to just sync the build scripts to whatever was current when 6.19 was released.
2. Maybe there's some way to just "compile and install the kernel the old way" and ignore the packages part of it. But I'm not familiar with the uboot specific formats for images and initial ram disks and I fear the result of doing that might be missing patches or some other things and not be comparable to the package-based experience.

And, yes, 6.19 might be some time ago, but AFAIU -current is still based on 6.18... next time that gets bumped up breakage will stop being "optional" for me.

 

 

[Werner]

5 hours ago, skandalfo said:

repository [...] armbian/build

Yes.

 

commit 6df6d0d607abfd59169a0ef2fddbed5fcd5b58f9
Author: EvilOlaf <werner@armbian.com>
Date:   Thu Feb 26 18:09:38 2026 +0000

    rockchip64: 7.0: Rebase on top of main

diff --git a/patch/kernel/archive/rockchip64-6.19/0000.patching_config.yaml b/patch/kernel/archive/rockchip64-7.0/0000.patching_config.yaml
similarity index 100%
rename from patch/kernel/archive/rockchip64-6.19/0000.patching_config.yaml
rename to patch/kernel/archive/rockchip64-7.0/0000.patching_config.yaml
diff --git a/patch/kernel/archive/rockchip64-6.19/add-board-helios64.patch b/patch/kernel/archive/rockchip64-7.0/add-board-helios64.patch
similarity index 100%
rename from patch/kernel/archive/rockchip64-6.19/add-board-helios64.patch
rename to patch/kernel/archive/rockchip64-7.0/add-board-helios64.patch
: