I tried the tinkerboard and starting an access point via USB Wifi works just fine there. I've experienced no problems with different computers and phones.
Using the same dongle for Wifi to create an access point with the rock64 isn't really as stable.. don't know what the problem might be. Connections are broken for some devices, and sometimes it's impossible to log on to the access point.
So for the tinkerboard it works just fine, not so much for the rock64. If the drivers are fine then it's probably some other software related problems.
Thanks for the update Igor!