I have been able to fix this issue for myself.
It ended up being the case that the USB-C to USB-C wire that I was using would work as USB3.0 on the Zero3E in a specific orientation only (which is weird because it works every which way on other devices)
So what I did was I kept flipping the typeC port on both the Zero3E side, and the external SSD until it was detected as USB3.0, and then I marked those with sides. Now, I just look at the markings to check which sides of the cable need to be facing upwards and connect it to the Zero3E and the external SSD, and its never gone to USB2.0 mode again!
Super weird issue, and an even weirder fix I must say.
P.S. The cable works in USB3 mode in any orientation on all other devices, it's just this way with the Zero3E, and I have no clue why!