after I tried to downgrade kernel to 5.31 because of these problems, I bricked the system (unrecognized MMC card device), so I had to overwrite u-boot and /boot manually on mounted SD card to bring it to life. I don't know how exactly I did it, but now I have 4.19.20 kernel without these symptoms - no more excesive CPU usage by kworker, CPU temperature drops by more than 10 degrees.
The problem is that cpufreq is not working anymore:
no or unknown cpufreq driver is active on this CPU
Maybe there is some connection to it?