I don't understand why, but it seems when compression backends are mixed is when the slowness occurs. When trying this on my Rock64, RockPro64 and x86_64 VM with Armbian bullseye it seemed to be fast for a while, and then degrade... disabling the compression made them all similarly quick.