correct me if i'm wrong, but the point of a board level patch is so that it does not apply to all devices, no?
that's the point of creating the board_x96q subdirectory instead of just adding the patches to the u-boot sunxi directory.
You can see this when building armbian.