Okay so I got it to work. It seems like the network manager was interfering somehow even though I had stopped and disabled the network manager. I used the following commands to do this:
$ sudo systemctl stop NetworkManager.service
$ sudo systemctl disable NetworkManager.service
So I uninstalled the network manager completely and then tried using wpa_supplicant again as posted in my post above, connected in first go, no issues!!
I then added the following code to /etc/networking/interfaces to bring the wifi interface up automatically at boot
auto wlan0
iface wlan0 inet dhcp
pre-up wpa_supplicant -i wlan0 -c/etc/wpa_supplicant.conf -D nl80211 -B
post-down killall -q wpa_supplicant
Have restarted a few times to test, seems to be working fine. Will update here if I notice any connectivity issues (which seems like a known issue with this boards wifi)