I think my issue is actually that CONFIG_RT_GROUP_SCHED is enabled.
I was certain I had disabled it. After some sluething.. It turns out I don't know how to configure/build a kernel.
I run
./compile.sh BOARD=bananapim4zero BRANCH=edge kernel-config
I then make some changes and tell it to save:
For instance when I enter menuconfig, I see this.
I then REMOVE the check.
I say YES
OOPS. I realized I wanted to make another change.
And now it's back.
After unchecking this, and building the kernel, moving the kernel, installing, and rebooting I see the config hasn't change on board.
So, I must have a misunderstanding of how this is supposed to work.
I checked here: https://docs.armbian.com/Developer-Guide_Build-Commands/
Which says "Add or Remove" modules, but doesn't elaborate if it's copying some defaults or where they might be coming from.