Ok I did some more testing and it turns out that wpa_supplicant is interfering with hostapd, so I don't have to specifically worry about network manager anymore. All i did to fix the issue was define the cards in wpa_supplicant and then it worked as planned.
As for Network Manager, putting iptables over the instinctive routing path didn't change anything as I'd expected, I talked to someone that said making hard routes with the "route" command would be a better Idea and more likely to work but I've never really done that.
Overall, wpa_supplicant seems to mess around with the wireless interface AP mode and encourage anyone else with a similar problem to either disable it or write in the arguments that you need.