Reporting back after way too much time passed.
I played around with other distros that used an older version of U-Boot and seemingly did not suffer from the reboot issue, however the reboot issue started coming back sporadically with newer kernels.
Having read about pure UHS-I cards causing the issue I decided to buy a new SD card for my Odroid C4 and look at that, the reboot issue disappeared completely even with the 6.0 nightly kernel build.
Safe to say I'm moving back to Armbian now having abandoned it previously due to this issue.
Old card on the left (Class 10 UHS-I)
New card on the right (UHS-I, UHS class 3, A1, V30)
Old card does not reboot while new one does.
