I played around with affinity, no change. It got so bad that via SSH I'd be kicked off or my commands would partially input. I did a restore from backup, and since I do use docker, back up and running perfectly again. I wish I knew what was tainting the system though.
Could it be this or similar? https://community.home-assistant.io/t/home-assistant-in-docker-causes-network-packet-loss-on-host/520067 Unknown, as I am not running anything privileged.
I am not sure *if* this is a hint as well. There has to be an explanation but as to what still at a loss. Especially since I can restore like I did, and everything is working perfectly. Perhaps I should just avoid apt-get upgrades