Nevermind, it turns out not to be a kernel problem. Leaving details about what was wrong, if anyone else has similar issues.
I have a custom USB HID device that gets accessed every 3-4 seconds by an app, which uses libusb. The libusb has 2 implementations - libusb-1.0 and hidraw. It turns out libusb-1.0 implementation was causing the host controller issues, funny it happened only with legacy kernels.
The problems went away when I switched to the hidraw implementation.
