I have the exact same issue on Radxa 3E 2GB model.
I installed Armbian but the 2nd port (labelled usb3) only runs my external SSD in usb2 speeds.
The same issue was faced on dietpi as well, which is why I stopped using that, but same issue here as well.