seems time has healed the wounds.
I find that adding `cpufreq_dt` to blacklisted modules and running `sudo update initramfs -u` seems to have resolved the issue but at the cost of having a slow CPU.
So I think the issue is cpu frequency scaling, possible caused by some device tree changes.
[Edit] I'm currently looking at this change to see if any parts made it into mainline since I know upstream doesn't like large patches like this: https://github.com/hanwckf/linux-rk3328-box/commit/23bd4498927474aeedd7cf45dcb498583aa4cfba
