I have tried disabling the desktop from the SSH as well, but as soon as I close the session the network connectivity dies right away. Then I have to hook up a keyboard to access the board.
Yes, I know there are cli only images there, but sometimes I use the desktop to browse, watch videos, listen to music etc as well through vnc.
I forgot the mention before that I'm using Armbian Bionic Desktop (19.11.3) image.