Matthias Strubel

Thanks @hste . That thread helped alot. I decided to switch to the 5.10-legacy kernel on my main OrangePi5 for now. I'll continue with DTS voltage adjustments on the test opi, I guess.

I did move on with that research. It seems that the OPP readings are different, which should be the same (assumption). # uname -r 6.5.0-rc5-edge-rockchip-rk3588 # bash opp_summary.sh cpu0 ----------------- Frequency | u_watt u_amp u_volt_max u_volt_min u_volt_target ------------------------------------------------------------------- 408000000 | 0 0 950000 750000 750000 | 0 0 950000 750000 750000 600000000 | 0 0 950000 750000 750000 | 0 0 950000 750000 750000 816000000 | 0 0 950000 750000 750000 | 0 0 950000 750000 750000 1008000000 | 0 0 950000 750000 750000 | 0 0 950000 750000 750000 1200000000 | 0 0 950000 775000 775000 | 0 0 950000 775000 775000 1416000000 | 0 0 950000 825000 825000 | 0 0 950000 825000 825000 1608000000 | 0 0 950000 875000 875000 | 0 0 950000 875000 875000 1800000000 | 0 0 950000 950000 950000 | 0 0 950000 950000 950000 cpu4 ----------------- Frequency | u_watt u_amp u_volt_max u_volt_min u_volt_target ------------------------------------------------------------------- 408000000 | 0 0 1000000 600000 600000 | 0 0 1000000 675000 675000 600000000 | 0 0 1000000 600000 600000 | 0 0 1000000 675000 675000 816000000 | 0 0 1000000 600000 600000 | 0 0 1000000 675000 675000 1008000000 | 0 0 1000000 625000 625000 | 0 0 1000000 675000 675000 1200000000 | 0 0 1000000 650000 650000 | 0 0 1000000 675000 675000 1416000000 | 0 0 1000000 675000 675000 | 0 0 1000000 675000 675000 1608000000 | 0 0 1000000 700000 700000 | 0 0 1000000 700000 700000 1800000000 | 0 0 1000000 775000 775000 | 0 0 1000000 775000 775000 2016000000 | 0 0 1000000 850000 850000 | 0 0 1000000 850000 850000 2208000000 | 0 0 1000000 925000 925000 | 0 0 1000000 925000 925000 cpu6 ----------------- Frequency | u_watt u_amp u_volt_max u_volt_min u_volt_target ------------------------------------------------------------------- 408000000 | 0 0 1000000 600000 600000 | 0 0 1000000 675000 675000 600000000 | 0 0 1000000 600000 600000 | 0 0 1000000 675000 675000 816000000 | 0 0 1000000 600000 600000 | 0 0 1000000 675000 675000 1008000000 | 0 0 1000000 625000 625000 | 0 0 1000000 675000 675000 1200000000 | 0 0 1000000 650000 650000 | 0 0 1000000 675000 675000 1416000000 | 0 0 1000000 675000 675000 | 0 0 1000000 675000 675000 1608000000 | 0 0 1000000 700000 700000 | 0 0 1000000 700000 700000 1800000000 | 0 0 1000000 775000 775000 | 0 0 1000000 775000 775000 2016000000 | 0 0 1000000 850000 850000 | 0 0 1000000 850000 850000 2208000000 | 0 0 1000000 925000 925000 | 0 0 1000000 925000 925000 # uname -r 5.10.110-rockchip-rk3588 # bash ./opp_summary.sh cpu0 ----------------- Frequency | u_amp u_volt_max u_volt_min u_volt_target ------------------------------------------------------------------- 408000000 | 0 950000 675000 675000 | 0 950000 675000 675000 600000000 | 0 950000 675000 675000 | 0 950000 675000 675000 816000000 | 0 950000 675000 675000 | 0 950000 675000 675000 1008000000 | 0 950000 675000 675000 | 0 950000 675000 675000 1200000000 | 0 950000 687500 687500 | 0 950000 687500 687500 1416000000 | 0 950000 725000 725000 | 0 950000 725000 725000 1608000000 | 0 950000 812500 812500 | 0 950000 812500 812500 1800000000 | 0 950000 912500 912500 | 0 950000 912500 912500 cpu4 ----------------- Frequency | u_amp u_volt_max u_volt_min u_volt_target ------------------------------------------------------------------- 408000000 | 0 1000000 675000 675000 | 0 1000000 675000 675000 600000000 | 0 1000000 675000 675000 | 0 1000000 675000 675000 816000000 | 0 1000000 675000 675000 | 0 1000000 675000 675000 1008000000 | 0 1000000 675000 675000 | 0 1000000 675000 675000 1200000000 | 0 1000000 675000 675000 | 0 1000000 675000 675000 1416000000 | 0 1000000 700000 700000 | 0 1000000 700000 700000 1608000000 | 0 1000000 725000 725000 | 0 1000000 725000 725000 1800000000 | 0 1000000 800000 800000 | 0 1000000 800000 800000 2016000000 | 0 1000000 875000 875000 | 0 1000000 875000 875000 2208000000 | 0 1000000 962500 962500 | 0 1000000 962500 962500 2256000000 | 0 1000000 1000000 1000000 | 0 1000000 1000000 1000000 2304000000 | 0 1000000 1000000 1000000 | 0 1000000 1000000 1000000 2352000000 | 0 1000000 1000000 1000000 | 0 1000000 1000000 1000000 2400000000 | 0 1000000 1000000 1000000 | 0 1000000 1000000 1000000 cpu6 ----------------- Frequency | u_amp u_volt_max u_volt_min u_volt_target ------------------------------------------------------------------- 408000000 | 0 1000000 675000 675000 | 0 1000000 675000 675000 600000000 | 0 1000000 675000 675000 | 0 1000000 675000 675000 816000000 | 0 1000000 675000 675000 | 0 1000000 675000 675000 1008000000 | 0 1000000 675000 675000 | 0 1000000 675000 675000 1200000000 | 0 1000000 675000 675000 | 0 1000000 675000 675000 1416000000 | 0 1000000 700000 700000 | 0 1000000 700000 700000 1608000000 | 0 1000000 725000 725000 | 0 1000000 725000 725000 1800000000 | 0 1000000 800000 800000 | 0 1000000 800000 800000 2016000000 | 0 1000000 875000 875000 | 0 1000000 875000 875000 2208000000 | 0 1000000 962500 962500 | 0 1000000 962500 962500 2256000000 | 0 1000000 1000000 1000000 | 0 1000000 1000000 1000000 2304000000 | 0 1000000 1000000 1000000 | 0 1000000 1000000 1000000 2352000000 | 0 1000000 1000000 1000000 | 0 1000000 1000000 1000000 2400000000 | 0 1000000 1000000 1000000 | 0 1000000 1000000 1000000 Based off this script opp_summary.sh

Currently, I am assuming, that the reason for the different behavior is the result of a different cost calculation. (Maybe the flag "inefficient toward ondemand doesn't exists either".) My hottest candidate are the changes around this patch: Kernel PATCH thread - [PATCH v2 0/4] Energy Model power in micro-Watts and SCMI v3.1 alignment Particular commit - energy_management.h for refrence only , [PATCH 0/4] Energy Model power in micro-Watts and SCMI v3.1 alignment I am still investigating in that area and need to compare a few things. My assumption is, that there is a kind of change in OPP, energy_management and cpufreq-drivers (-dt), which needs to be adapted to rockchip-cpufreq. Something like the kernel expects unit W but the driver delivers V. edit: I discarded the assumption about the units, after reviewing the upstream code about the calculation + patches for rk3588

I did a test with a second orangePi5 using the legacy kernel. It seems this kernel identifies different frequencies as "inefficient", but the onemand governor does not take account of this information and scales across the compete frequency range. # uname -r 5.10.110-rockchip-rk3588 [ 7.956842] cpu cpu0: Looking up cpu-supply from device tree [ 7.957323] cpu cpu0: bin=0 [ 7.957521] cpu cpu0: leakage=11 [ 7.957544] cpu cpu0: Looking up cpu-supply from device tree [ 7.958986] cpu cpu0: pvtm=1459 [ 7.959055] cpu cpu0: pvtm-volt-sel=3 [ 7.959087] cpu cpu0: Looking up cpu-supply from device tree [ 7.959212] cpu cpu0: Looking up mem-supply from device tree [ 7.959376] cpu cpu4: Looking up cpu-supply from device tree [ 7.961042] cpu cpu4: bin=0 [ 7.961213] cpu cpu4: leakage=9 [ 7.961231] cpu cpu4: Looking up cpu-supply from device tree [ 7.967741] cpu cpu4: pvtm=1692 [ 7.971693] cpu cpu4: pvtm-volt-sel=4 [ 7.971725] cpu cpu4: Looking up cpu-supply from device tree [ 7.972223] cpu cpu4: Looking up mem-supply from device tree [ 7.972756] cpu cpu6: Looking up cpu-supply from device tree [ 7.974414] cpu cpu6: bin=0 [ 7.974583] cpu cpu6: leakage=9 [ 7.974601] cpu cpu6: Looking up cpu-supply from device tree [ 7.981071] cpu cpu6: pvtm=1700 [ 7.984995] cpu cpu6: pvtm-volt-sel=4 [ 7.985022] cpu cpu6: Looking up cpu-supply from device tree [ 7.985517] cpu cpu6: Looking up mem-supply from device tree [ 7.986626] cpu cpu0: avs=0 [ 7.987551] cpu cpu4: avs=0 [ 7.988494] cpu cpu6: avs=0 [ 7.988652] cpu cpu0: EM: OPP:816000 is inefficient [ 7.988655] cpu cpu0: EM: OPP:600000 is inefficient [ 7.988739] cpu cpu0: EM: created perf domain [ 7.988776] cpu cpu0: l=10000 h=85000 hyst=5000 l_limit=0 h_limit=1608000000 h_table=0 [ 7.989471] cpu cpu4: EM: OPP:2304000 is inefficient [ 7.989475] cpu cpu4: EM: OPP:816000 is inefficient [ 7.989478] cpu cpu4: EM: OPP:600000 is inefficient [ 7.989623] cpu cpu4: EM: created perf domain [ 7.989654] cpu cpu4: l=10000 h=85000 hyst=5000 l_limit=0 h_limit=2208000000 h_table=0 [ 7.998280] cpu cpu6: EM: OPP:2304000 is inefficient [ 7.998284] cpu cpu6: EM: OPP:816000 is inefficient [ 7.998287] cpu cpu6: EM: OPP:600000 is inefficient [ 7.998436] cpu cpu6: EM: created perf domain [ 7.998862] cpu cpu6: l=10000 h=85000 hyst=5000 l_limit=0 h_limit=2208000000 h_table=0 # cpufreq-info cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 2 3 CPUs which need to have their frequency coordinated by software: 0 1 2 3 maximum transition latency: 84.0 us. hardware limits: 408 MHz - 1.80 GHz available frequency steps: 408 MHz, 600 MHz, 816 MHz, 1.01 GHz, 1.20 GHz, 1.42 GHz, 1.61 GHz, 1.80 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 408 MHz and 1.80 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 408 MHz (asserted by call to hardware). cpufreq stats: 408 MHz:96.30%, 600 MHz:0.49%, 816 MHz:0.05%, 1.01 GHz:0.05%, 1.20 GHz:0.13%, 1.42 GHz:0.11%, 1.61 GHz:0.06%, 1.80 GHz:2.82% (325) analyzing CPU 1: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 2 3 CPUs which need to have their frequency coordinated by software: 0 1 2 3 maximum transition latency: 84.0 us. hardware limits: 408 MHz - 1.80 GHz available frequency steps: 408 MHz, 600 MHz, 816 MHz, 1.01 GHz, 1.20 GHz, 1.42 GHz, 1.61 GHz, 1.80 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 408 MHz and 1.80 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 408 MHz (asserted by call to hardware). cpufreq stats: 408 MHz:96.30%, 600 MHz:0.49%, 816 MHz:0.05%, 1.01 GHz:0.05%, 1.20 GHz:0.13%, 1.42 GHz:0.11%, 1.61 GHz:0.06%, 1.80 GHz:2.82% (325) analyzing CPU 2: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 2 3 CPUs which need to have their frequency coordinated by software: 0 1 2 3 maximum transition latency: 84.0 us. hardware limits: 408 MHz - 1.80 GHz available frequency steps: 408 MHz, 600 MHz, 816 MHz, 1.01 GHz, 1.20 GHz, 1.42 GHz, 1.61 GHz, 1.80 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 408 MHz and 1.80 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 408 MHz (asserted by call to hardware). cpufreq stats: 408 MHz:96.30%, 600 MHz:0.49%, 816 MHz:0.05%, 1.01 GHz:0.05%, 1.20 GHz:0.13%, 1.42 GHz:0.11%, 1.61 GHz:0.06%, 1.80 GHz:2.82% (325) analyzing CPU 3: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 2 3 CPUs which need to have their frequency coordinated by software: 0 1 2 3 maximum transition latency: 84.0 us. hardware limits: 408 MHz - 1.80 GHz available frequency steps: 408 MHz, 600 MHz, 816 MHz, 1.01 GHz, 1.20 GHz, 1.42 GHz, 1.61 GHz, 1.80 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 408 MHz and 1.80 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 408 MHz (asserted by call to hardware). cpufreq stats: 408 MHz:96.30%, 600 MHz:0.49%, 816 MHz:0.05%, 1.01 GHz:0.05%, 1.20 GHz:0.13%, 1.42 GHz:0.11%, 1.61 GHz:0.06%, 1.80 GHz:2.82% (325) analyzing CPU 4: driver: cpufreq-dt CPUs which run at the same hardware frequency: 4 5 CPUs which need to have their frequency coordinated by software: 4 5 maximum transition latency: 324 us. hardware limits: 408 MHz - 2.40 GHz available frequency steps: 408 MHz, 600 MHz, 816 MHz, 1.01 GHz, 1.20 GHz, 1.42 GHz, 1.61 GHz, 1.80 GHz, 2.02 GHz, 2.21 GHz, 2.26 GHz, 2.30 GHz, 2.35 GHz, 2.40 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 408 MHz and 2.40 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 600 MHz (asserted by call to hardware). cpufreq stats: 408 MHz:50.53%, 600 MHz:47.79%, 816 MHz:0.26%, 1.01 GHz:0.02%, 1.20 GHz:0.02%, 1.42 GHz:0.01%, 1.61 GHz:0.04%, 1.80 GHz:0.01%, 2.02 GHz:0.01%, 2.21 GHz:0.01%, 2.26 GHz:0.00%, 2.30 GHz:0.00%, 2.35 GHz:0.00%, 2.40 GHz:1.27% (148) analyzing CPU 5: driver: cpufreq-dt CPUs which run at the same hardware frequency: 4 5 CPUs which need to have their frequency coordinated by software: 4 5 maximum transition latency: 324 us. hardware limits: 408 MHz - 2.40 GHz available frequency steps: 408 MHz, 600 MHz, 816 MHz, 1.01 GHz, 1.20 GHz, 1.42 GHz, 1.61 GHz, 1.80 GHz, 2.02 GHz, 2.21 GHz, 2.26 GHz, 2.30 GHz, 2.35 GHz, 2.40 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 408 MHz and 2.40 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 600 MHz (asserted by call to hardware). cpufreq stats: 408 MHz:50.53%, 600 MHz:47.79%, 816 MHz:0.26%, 1.01 GHz:0.02%, 1.20 GHz:0.02%, 1.42 GHz:0.01%, 1.61 GHz:0.04%, 1.80 GHz:0.01%, 2.02 GHz:0.01%, 2.21 GHz:0.01%, 2.26 GHz:0.00%, 2.30 GHz:0.00%, 2.35 GHz:0.00%, 2.40 GHz:1.27% (148) analyzing CPU 6: driver: cpufreq-dt CPUs which run at the same hardware frequency: 6 7 CPUs which need to have their frequency coordinated by software: 6 7 maximum transition latency: 324 us. hardware limits: 408 MHz - 2.40 GHz available frequency steps: 408 MHz, 600 MHz, 816 MHz, 1.01 GHz, 1.20 GHz, 1.42 GHz, 1.61 GHz, 1.80 GHz, 2.02 GHz, 2.21 GHz, 2.26 GHz, 2.30 GHz, 2.35 GHz, 2.40 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 408 MHz and 2.40 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 408 MHz (asserted by call to hardware). cpufreq stats: 408 MHz:98.77%, 600 MHz:0.18%, 816 MHz:0.04%, 1.01 GHz:0.03%, 1.20 GHz:0.03%, 1.42 GHz:0.03%, 1.61 GHz:0.03%, 1.80 GHz:0.01%, 2.02 GHz:0.01%, 2.21 GHz:0.01%, 2.26 GHz:0.00%, 2.30 GHz:0.00%, 2.35 GHz:0.01%, 2.40 GHz:0.85% (156) analyzing CPU 7: driver: cpufreq-dt CPUs which run at the same hardware frequency: 6 7 CPUs which need to have their frequency coordinated by software: 6 7 maximum transition latency: 324 us. hardware limits: 408 MHz - 2.40 GHz available frequency steps: 408 MHz, 600 MHz, 816 MHz, 1.01 GHz, 1.20 GHz, 1.42 GHz, 1.61 GHz, 1.80 GHz, 2.02 GHz, 2.21 GHz, 2.26 GHz, 2.30 GHz, 2.35 GHz, 2.40 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 408 MHz and 2.40 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 408 MHz (asserted by call to hardware). cpufreq stats: 408 MHz:98.77%, 600 MHz:0.18%, 816 MHz:0.04%, 1.01 GHz:0.03%, 1.20 GHz:0.03%, 1.42 GHz:0.03%, 1.61 GHz:0.03%, 1.80 GHz:0.01%, 2.02 GHz:0.01%, 2.21 GHz:0.01%, 2.26 GHz:0.00%, 2.30 GHz:0.00%, 2.35 GHz:0.01%, 2.40 GHz:0.85% (156) edit: The scheduling seems to make use of the "low" kernels only, the edge kernel across the different kernels (which run on a higher speed on idle). edit2: more details: Little cores sys/kernel/debug/energy_model/cpu0# ls -1 ps\:*/power | xargs -I {} sh -c 'echo -n "{} : ";cat {} ' | sort -t ':' -k3 -n ps:408000/power : 18 ps:600000/power : 27 ps:816000/power : 37 ps:1008000/power : 45 ps:1200000/power : 56 ps:1416000/power : 74 ps:1608000/power : 106 ps:1800000/power : 149 # ls -1 ps\:*/cost | xargs -I {} sh -c 'echo -n "{} : ";cat {} ' | sort -t ':' -k3 -n ps:408000/cost : 79411 ps:1008000/cost : 80357 ps:600000/cost : 81000 ps:816000/cost : 81617 ps:1200000/cost : 84000 ps:1416000/cost : 94067 ps:1608000/cost : 118656 ps:1800000/cost : 149000 High cores: # ls -1 ps\:*/power | xargs -I {} sh -c 'echo -n "{} : ";cat {} ' | sort -t ':' -k3 -n ps:408000/power : 55 ps:600000/power : 82 ps:816000/power : 111 ps:1008000/power : 137 ps:1200000/power : 164 ps:1416000/power : 208 ps:1608000/power : 253 ps:1800000/power : 345 ps:2016000/power : 463 ps:2208000/power : 613 ps:2256000/power : 676 ps:2304000/power : 691 ps:2352000/power : 705 ps:2400000/power : 720 # ls -1 ps\:*/cost | xargs -I {} sh -c 'echo -n "{} : ";cat {} ' | sort -t ':' -k3 -n ps:408000/cost : 323529 ps:1008000/cost : 326190 ps:816000/cost : 326470 ps:1200000/cost : 328000 ps:600000/cost : 328000 ps:1416000/cost : 352542 ps:1608000/cost : 377611 ps:1800000/cost : 460000 ps:2016000/cost : 551190 ps:2208000/cost : 666304 ps:2256000/cost : 719148 ps:2352000/cost : 719387 ps:2304000/cost : 719791 ps:2400000/cost : 720000

No, not yet. I usually try to stay as close as upstream as possible. Next week or the week after another board will arrive, so I can do some tests with that new one. After that deep dive, I see that issue reasonable and it is ok for me. As soon as I can, I’ll do a test.

Hello everyone, I recently installed my orangePi 5 to pass workload from my x86 servers over to the orangePi. My main focus is to reduce my overall energy consumption of my homelab. TL;DR Kernel thinks, that while being idle the sweet spot for the frequencies in "ondemand" and "schedutil" is: 1024MHz on the small chips (CPU0-3) 816MHZ on the big chips (CPU4+5, CPU6+7) Reason is an internal cost calculation power vs. frequency gain, which sorts out the lower frequencies. Long story: After getting my first orangePi5 in place, I started to play around with the hardware. I am running: █ █ █ █ █ █ █ █ █ █ █ OS: Armbian (23.11.0-trunk.43) aarch64 ███████████████████████ Host: Orange Pi 5 ▄▄██ ██▄▄ Kernel: 6.5.0-rc5-edge-rockchip-rk3588 ▄▄██ ███████████ ██▄▄ Uptime: 5 days, 17 hours, 24 mins ▄▄██ ██ ██ ██▄▄ Packages: 459 (dpkg) ▄▄██ ██ ██ ██▄▄ Shell: bash 5.2.15 ▄▄██ ██ ██ ██▄▄ CPU: (8) @ 1.800GHz ▄▄██ █████████████ ██▄▄ Memory: 1439MiB / 7688MiB # cpufreq-info (cropped for low and high cores) cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: rockchip-cpufreq CPUs which run at the same hardware frequency: 0 1 2 3 CPUs which need to have their frequency coordinated by software: 0 1 2 3 maximum transition latency: 72.0 us. hardware limits: 408 MHz - 1.80 GHz available frequency steps: 408 MHz, 600 MHz, 816 MHz, 1.01 GHz, 1.20 GHz, 1.42 GHz, 1.61 GHz, 1.80 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 408 MHz and 1.80 GHz. The governor "schedutil" may decide which speed to use within this range. current CPU frequency is 1.20 GHz (asserted by call to hardware). cpufreq stats: 408 MHz:0.00%, 600 MHz:0.00%, 816 MHz:0.00%, 1.01 GHz:99.91%, 1.20 GHz:0.07%, 1.42 GHz:0.01%, 1.61 GHz:0.00%, 1.80 GHz:0.01% (38672) [..] analyzing CPU 7: driver: rockchip-cpufreq CPUs which run at the same hardware frequency: 6 7 CPUs which need to have their frequency coordinated by software: 6 7 maximum transition latency: 356 us. hardware limits: 408 MHz - 2.21 GHz available frequency steps: 408 MHz, 600 MHz, 816 MHz, 1.01 GHz, 1.20 GHz, 1.42 GHz, 1.61 GHz, 1.80 GHz, 2.02 GHz, 2.21 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 408 MHz and 2.21 GHz. The governor "schedutil" may decide which speed to use within this range. current CPU frequency is 816 MHz (asserted by call to hardware). cpufreq stats: 408 MHz:0.00%, 600 MHz:0.00%, 816 MHz:99.91%, 1.01 GHz:0.03%, 1.20 GHz:0.01%, 1.42 GHz:0.02%, 1.61 GHz:0.01%, 1.80 GHz:0.00%, 2.02 GHz:0.00%, 2.21 GHz:0.02% (41580) So, why are the lowest frequency not reached? Tests with the "powersave" showed, that the kernel is able to set these frequencies. The "ondemand" scheduler showed no difference. # uptime 15:44:20 up 5 days, 17:26, 2 users, load average: 0.00, 0.00, 0.00 Load average is pretty much like nothing. So, I started to doubt that everything is working correctly: Why does the frequency selection work like this? Why are the lowest frequencies not considered in the evaluation? Are the little/big difference between the cores accounted and used? What are those error messages on system startup? I was not able to find anything useful about any limitations, known errors or open bugs. I wasn't able to find a hint, that explained why "ondemand" and "schedutil" won't hit the lowest possible frequency. And yes! I used Google for several days. So, I started to dig into the following messages from my bootup: [ 1.790851] cpu cpu0: _parse_named_prop: Invalid number of elements in microvolt property (6) with supplies (1) [ 1.790862] cpu cpu0: _of_add_opp_table_v2: Failed to add OPP, -22 [ 1.790867] cpu cpu0: OPP table can't be empty [ 1.799837] cpu cpu0: EM: OPP:816000 is inefficient [ 1.799858] cpu cpu0: EM: OPP:600000 is inefficient [ 1.799865] cpu cpu0: EM: OPP:408000 is inefficient [ 1.800170] cpu cpu0: EM: created perf domain [ 1.800981] cpu cpu4: EM: OPP:600000 is inefficient [ 1.800992] cpu cpu4: EM: OPP:408000 is inefficient [ 1.801222] cpu cpu4: EM: created perf domain [ 1.802005] cpu cpu6: EM: OPP:600000 is inefficient [ 1.802014] cpu cpu6: EM: OPP:408000 is inefficient [ 1.802246] cpu cpu6: EM: created perf domain The first two messages are based of the OPP device drive, reading the tuples from the system(/or predefined structures?) containing power information about the given CPU cores. The message "OPP table can't be empty" seem to be emitted by cpufreq-dt, which is working side by side with OPP to organize the internal CPU information. From an earlier research I found we are using the driver "rockchip-cpufreq" on rk35xx/rk3588 for doing OPP handing, which is using the cpufreq-dt functions, too. That driver is needed, because we have to handle more then one OPP table. The next thing was "cpu cpu%d: EM: OPP:%d is inefficient" entry. That message comes from deep inside the kernels common code: "energy management". After some digging, I found out, that armbian is compiled with debugfs enabled (yeah) so we can peek into a lot of counters. The energy management is described in details here: https://docs.kernel.org/power/energy-model.html - I read the code first and found out later that documentation is very accurate. I'm glad, that armbian is build with debugfs enabled, so all counters from energy_management are available under: /sys/kernel/debug/energy_model Basically these are the counter for my orangePi5 chip and how the kernel evaluates the costs for each frequency level (I marked the frequency from above): cpu0/ps:408000/cost 230850 | cpu0/ps:600000/cost 230850 | cpu0/ps:816000/cost 230850 | cpu0/ps:1008000/cost 230850 | <--- cpu0/ps:1200000/cost 246496 cpu0/ps:1416000/cost 279327 cpu0/ps:1608000/cost 314211 cpu0/ps:1800000/cost 370386 cpu4/ps:408000/cost 330669 | cpu4/ps:600000/cost 330670 cpu4/ps:816000/cost 330669 | <--- cpu4/ps:1008000/cost 358800 cpu4/ps:1200000/cost 388078 cpu4/ps:1416000/cost 418503 cpu4/ps:1608000/cost 450077 cpu4/ps:1800000/cost 551690 cpu4/ps:2016000/cost 663635 cpu4/ps:2208000/cost 785915 This is the reason why ondemand, conservative and schedutil choose these frequency as a minimum frequency. The kernel does not seem to have an efficiency gain in terms of power vs energy consumption. The calculation is cost = max_frequency * power_for_frequency / frequency . Where does the "power_for_frequency" come from? These are also delivered and available: cpu0/ps:408000/power 52326 cpu0/ps:600000/power 76950 cpu0/ps:816000/power 104652 cpu0/ps:1008000/power 129276 cpu0/ps:1200000/power 164331 cpu0/ps:1416000/power 219738 cpu0/ps:1608000/power 280696 cpu0/ps:1800000/power 370386 cpu4/ps:408000/power 61102 cpu4/ps:600000/power 89856 cpu4/ps:816000/power 122204 cpu4/ps:1008000/power 163800 cpu4/ps:1200000/power 210912 cpu4/ps:1416000/power 268388 cpu4/ps:1608000/power 327774 cpu4/ps:1800000/power 449748 cpu4/ps:2016000/power 605928 cpu4/ps:2208000/power 785915 This cost value surprised me, that it is the same for the first frequency levels, so I did the math and it is correct. (In fact, I did that to prove my assumption, that I interpreted the cpu-flags correctly, too.) Basically, I found the reason why we are "locked" at these particular frequencies and are not saving more Watt. Unfortunately there is no official way to adjust the cost values to allow the energy_management to do a different decision. What bothered me is, that cpufreq-info shows a complete frequency list and maintains all available frequencies. It took me another hour to figure out, that the energy_management source code (em_cpufreq_update_efficiencies) reports the findings via the function cpufreq_table_set_inefficient to the frequency table inside cpufreq. On that way, the frequency table receives flags for the "inefficient" frequencies, which is taken into account by any sophisticated governor (ondemand, schedutil..). These information is not transported by any cpufreq utility to the user. Bonus question: Does the different power levels of the cores being taken account into the CPU-scheduling? 1. We can see here, that different performance domains are created: [ 1.800170] cpu cpu0: EM: created perf domain [ 1.801222] cpu cpu4: EM: created perf domain [ 1.802246] cpu cpu6: EM: created perf domain 2. Is the energy_aware scheduler active? # sysctl -a | grep sched_energy_aware kernel.sched_energy_aware = 1 It is! Writing this down shortens the path of finding everything out against the real invest into the research 😄 . The Kernel documentation is very helpful, but the debug information need to be found inside the source code (I'm fine with thaT). I recommend reading the https://docs.kernel.org/scheduler/sched-energy.html , which has a very explanation how the energy_management from above works together with the scheduler. According to the documentation, it works the best with the CPU-governor "schedutil" because it works hand in hand with task scheduling information in opposite to "ondemand" which looks at the core-cpu consumption. Further reading https://docs.kernel.org/power/energy-model.html https://docs.kernel.org/scheduler/sched-energy.html https://docs.kernel.org/scheduler/sched-capacity.html Thanks for reading, I hope this will help someone out there. Matthias edit: maybe this is more relevant to the „advanced user/development section“ 🤔