Add maxcpus=1 (usbcore.nousb) to kernel cmdline or e.g. echo 0 >/sys/devices/system/cpu/cpu3/online
Blacklisting modules is safe;
# cat >/etc/modprobe.d/blacklist.conf
blacklist cfg80211
blacklist ip_tables
blacklist lima
blacklist snd
blacklist snd_pcm
blacklist snd_pcm_dmaengine
blacklist snd_soc_core
blacklist snd_soc_simple_card
blacklist snd_soc_simple_card_utils
blacklist snd_timer
blacklist soundcore
blacklist sun4i_i2s
blacklist sun8i_adda_pr_regmap
blacklist sun8i_codec_analog
blacklist sunxi_cedrus
blacklist v4l2_mem2mem
^D
Note I do use headless, no usb no wifi no network-manager, you will need to adapt the blacklist according to your needs (board, used features).