That is probably due to the fact that your USB drive take a while to become ready... You probably need to try adding INITIAL_USB_SCAN_DELAY in u-boot config and recompile it ... https://lists.denx.de/pipermail/u-boot/2016-April/250343.html