Well just for the heck of it you could build a new image with sunxi-legacy and joystick support, test it and then grab a normal Ubuntu 20.04 for your PC which comes with kernel 5.4.x too and do a small comparison. I don't there is any difference from the kernel point of view. Though it might be possible that x86 Ubuntu Desktop ships some additional software tools with to make these toys properly work. But there is only one way to find out....