I read your github readme, but I am not 100% that you read the instructions for armbian's first run script:
https://github.com/armbian/build/blob/main/packages/bsp/armbian_first_run.txt.template
Try following that, and maybe the SSH problems will not be an issue with Debian, even if they are an issue with Raspbian.