I just (again) wasted two hours to remember that any of those kernels needs a lot of code cleaning and fixing than "just" USB3. When this is done, than it would be possible to start debugging without wasting hours / days.
About year ago I was approached by Allo, when they were trying to design their own board around S500 (Sparky). They were talking about mainlining (wishes) or bringing things at least to kernel 4.4, but I guess they abandon the idea due to whatever reason. There was lots of discussion going on and after my second reminder, that they need to hire and pay for my time, if they want me involved into their activities, I tuned myself out. They didn't listen what I was trying to tell them anyway.