Why not just use one cheap SBC just for this purpose? Attach one or more UART adapters to it and control devices remotely.
For my testing I'm using 3-port Wifi switch to control power to the devices. One SBC is used for USB-UART adapter, also controls one USB mux device to switch USB stick between device and this master SBC, and also controls one SD mux for same purpose as USB stick.