cpufreq-info: no or unknown cpufreq driver after upgrade to kernel 5.15.74-mvebu64 #22.08.6

[y52]

I was running espressobin v7 with

Welcome to Armbian 21.08.1 Bullseye with Linux 5.10.60-mvebu64
and made an upgrade in a regular way :

# apt upgrade

The following packages will be upgraded:
  apache2 apache2-bin apache2-data apache2-dev apache2-utils armbian-bsp-cli-espressobin armbian-config armbian-firmware
  armbian-zsh asterisk asterisk-config asterisk-modules asterisk-voicemail avahi-autoipd base-files bash curl
  deb.torproject.org-keyring dirmngr distro-info-data dpkg dpkg-dev ffmpeg gnupg gnupg-l10n gnupg-utils gnupg2 gpg gpg-agent
  gpg-wks-client gpg-wks-server gpgconf gpgsm gpgv gzip hostapd ifenslave inetutils-inetd inetutils-telnet
  libapache2-mod-php7.4 libavahi-client3 libavahi-common-data libavahi-common3 libavcodec58 libavdevice58 libavfilter7
  libavformat58 libavresample4 libavutil56 libc-bin libc-dev-bin libc-l10n libc6 libc6-dev libcups2 libcurl3-gnutls libcurl4
  libdpkg-perl libfreetype6 libfribidi0 libgnutls30 libgssapi-krb5-2 libk5crypto3 libkrb5-3 libkrb5support0 libldap-2.4-2
  libldap2-dev libldb2 liblzma5 libnm0 libnss-myhostname libntfs-3g883 libpam-systemd libpcre2-16-0 libpcre2-32-0 libpcre2-8-0
  libpcre2-dev libpcre2-posix2 libpostproc55 libpq5 libsdl2-2.0-0 libsmbclient libsnmp-base libsnmp40 libssl-dev libssl1.1
  libswresample3 libswscale5 libsystemd0 libtiff5 libtirpc-common libtirpc-dev libtirpc3 libudev1 libwbclient0 libxml2
  libxslt1.1 linux-dtb-current-mvebu64 linux-image-current-mvebu64 linux-libc-dev linux-u-boot-espressobin-current locales
  logrotate nano network-manager ntfs-3g openssh-client openssh-server openssh-sftp-server openssl php7.4 php7.4-cli
  php7.4-common php7.4-curl php7.4-json php7.4-opcache php7.4-readline python3-ldb python3-samba rsyslog samba samba-common
  samba-common-bin samba-libs smbclient systemd systemd-sysv tcpdump tor tor-geoipdb tzdata udev unzip wireless-regdb xz-utils
  zlib1g

 

...

update-initramfs: Converting to u-boot format
update-initramfs: Converting to u-boot FIT image

FIT description: EspressoBIN 3720 FIT Image
Created:         Sun Oct 23 13:08:21 2022
 Image 0 (kernel)
  Description:  Vanilla Linux kernel
  Created:      Sun Oct 23 13:08:21 2022
  Type:         Kernel Image
  Compression:  lzma compressed
  Data Size:    6882687 Bytes = 6721.37 KiB = 6.56 MiB
  Architecture: AArch64
  OS:           Linux
  Load Address: 0x07000000
  Entry Point:  0x07000000
  Hash algo:    sha1
  Hash value:   c8d92b9c244fdcc9fb56f640d05e6fc4d3f7ffb6
 Image 1 (ramdisk)
  Description:  Boot ramdisk
  Created:      Sun Oct 23 13:08:21 2022
  Type:         RAMDisk Image
  Compression:  Unknown Compression
  Data Size:    8909801 Bytes = 8700.98 KiB = 8.50 MiB
  Architecture: AArch64
  OS:           Linux
  Load Address: unavailable
  Entry Point:  unavailable
  Hash algo:    sha1
  Hash value:   3be954cf886386101311f7673115fa0be0c95366
 Image 2 (fdtv5)
  Description:  Flattened Device Tree ebinv5
  Created:      Sun Oct 23 13:08:21 2022
  Type:         Flat Device Tree
  Compression:  uncompressed
  Data Size:    11618 Bytes = 11.35 KiB = 0.01 MiB
  Architecture: AArch64
  Load Address: 0x06f00000
  Hash algo:    sha1
  Hash value:   dcfa1e753a26493b713b8d6eee19e0895db09cc3
 Image 3 (fdtv7)
  Description:  Flattened Device Tree ebinv7
  Created:      Sun Oct 23 13:08:21 2022
  Type:         Flat Device Tree
  Compression:  uncompressed
  Data Size:    11646 Bytes = 11.37 KiB = 0.01 MiB
  Architecture: AArch64
  Load Address: 0x06f00000
  Hash algo:    sha1
  Hash value:   0230f9e11a6f364dc66c0aca9a72620b19a0f1e0
 Default Configuration: 'v5'
 Configuration 0 (v5)
  Description:  Standard Boot ebinv5
  Kernel:       kernel
  Init Ramdisk: ramdisk
  FDT:          fdtv5
  Hash algo:    sha1
  Hash value:   unavailable
 Configuration 1 (v7)
  Description:  Standard Boot ebinv7
  Kernel:       kernel
  Init Ramdisk: ramdisk
  FDT:          fdtv7
  Hash algo:    sha1
  Hash value:   unavailable

 

After upgrade and reboot the kernel was upgraded to :

Welcome to Armbian 22.08.6 Bullseye with Linux 5.15.74-mvebu64
 

The side effect is that cpu-freq regulation seems to be broken: 

root@tiger:~# cpufreq-info 
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  no or unknown cpufreq driver is active on this CPU
  maximum transition latency: 4294.55 ms.
analyzing CPU 1:
  no or unknown cpufreq driver is active on this CPU
  maximum transition latency: 4294.55 ms.

 

The

# htop    is unable to determine it either : Cpu Freq:    0 MHz

 

The previous setup before upgrade "Armbian 21.08.1 Bullseye with Linux 5.10.60-mvebu64" ran frequency switch flawlessly:

# cpufreq-info 
  current CPU frequency is 200 MHz (asserted by call to hardware).
  cpufreq stats: 200 MHz:93.04%, 300 MHz:0.48%, 600 MHz:0.18%, 1.20 GHz:6.30%  (215)

 

 

Due to inability of CPU frequency regulation, I found that the board temperature has been increased by about 5 deg. C.

 

What could be done to fix this issue?

[y52]

By some unknown reason cpufreq device has become unavailable  after upgrade:

cat: /sys/devices/system/cpu/cpufreq/policy0/scaling_available_governors: No such file or directory

 

The old working system had :

# cat /sys/devices/system/cpu/cpufreq/policy0/scaling_available_governors
conservative ondemand userspace powersave performance schedutil 

[y52]

What could also be relevant to fix CPU scaling:

# ls /sys/devices/system/cpu/cpu0
cpu_capacity  crash_notes  crash_notes_size  hotplug  node0  of_node  online  power  regs  subsystem  topology  uevent

(no cpufreq)

 

CPU governors are not available.

root@tiger:~# dmesg -l emerg,alert,crit,err
[    1.917740] debugfs: Directory 'd0060900.xor' with parent 'dmaengine' already present!
[    2.420702] Unsupported CPU frequency 1200 MHz
[    3.937067] advk-pcie d0070000.pcie: link never came up

 

 

I use the mainline u-boot with CPU frequency 1200 MHz, which proved itself stable before upgrade:

Marvell>> reset
resetting ...
TIM-1.0
mv_ddr-devel-g7753d4b DDR4 16b 1GB 1CS
WTMI-devel-18.12.1-4392eaf
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.225V
Setting clocks: CPU 1200 MHz, DDR 750 MHz
CZ.NIC's Armada 3720 Secure Firmware v2021.09.07-27-g489262b (Feb  9 2022 18:50:42)
Running on ESPRESSObin
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.6(release):v2.6-365-ge0a6a512b
NOTICE:  BL1: Built : 18:52:58, Feb  9 2022
NOTICE:  BL1: Booting BL2
NOTICE:  BL2: v2.6(release):v2.6-365-ge0a6a512b
NOTICE:  BL2: Built : 18:53:02, Feb  9 2022
NOTICE:  BL1: Booting BL31
NOTICE:  BL31: v2.6(release):v2.6-365-ge0a6a512b
NOTICE:  BL31: Built : 18:53:12, Feb  9 2022

U-Boot 2022.04-rc1-00087-gb5c5b9a0be (Feb 09 2022 - 18:44:33 +0100)
 

  

[y52]

I used Igor's advice for another issue :

 

 

Workaround:
armbian-config -> system  -> Other ->  Switch to other kernels , pick previous, freeze, wait
             
                                | |linux-image-current-mvebu64=22.08.2       5.15.69-mvebu64 | |
                                | |linux-image-current-mvebu64=22.05.3       5.15.48-mvebu64 | |
                                | |linux-image-current-mvebu64=22.05.1       5.15.35-mvebu64 | |
                                | |linux-image-current-mvebu64=21.08.1       5.10.60-mvebu64 | |    <<-- CPU scaling is not broken
 

I progressively downgraded to kernels 5.15.69, than to 5.15.48 and 5.15.35. None of them allowed to fix CPU scaling.

This is rather strange, as I run another espressobin, which doesn't loose  CPU scaling with kernel 5.15.48-mvebu64.

This second espressobin is version 5 and u-boot running at 1000 MHz.

This could give a clue to fix broken CPU scaling in kernel 5.15.xx.

Just to remind, that

root@ebinv7# dmesg -l emerg,alert,crit,err
[    1.917740] debugfs: Directory 'd0060900.xor' with parent 'dmaengine' already present!
[    2.420702] Unsupported CPU frequency 1200 MHz    <<-- vs -->> ebinv5 @1000 MHz

 

After 5.15.x disenchantment I've downgraded to 5.10.60-mvebu64, which fixed CPU scaling :

root@ebinv7:/boot# dmesg -l emerg,alert,crit,err
[    2.319871] debugfs: Directory 'd0060900.xor' with parent 'dmaengine' already present!
[    5.739975] advk-pcie d0070000.pcie: link never came up
root@ebinv7:/boot# cpufreq-info 
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1
  CPUs which need to have their frequency coordinated by software: 0 1
  maximum transition latency: 0.97 ms.
  hardware limits: 200 MHz - 1.20 GHz
  available frequency steps: 200 MHz, 300 MHz, 600 MHz, 1.20 GHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil
  current policy: frequency should be within 200 MHz and 1.20 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 1.20 GHz (asserted by call to hardware).
  cpufreq stats: 200 MHz:22.65%, 300 MHz:8.94%, 600 MHz:0.09%, 1.20 GHz:68.32%  (1637)
analyzing CPU 1:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1
  CPUs which need to have their frequency coordinated by software: 0 1
  maximum transition latency: 0.97 ms.
  hardware limits: 200 MHz - 1.20 GHz
  available frequency steps: 200 MHz, 300 MHz, 600 MHz, 1.20 GHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil
  current policy: frequency should be within 200 MHz and 1.20 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 1.20 GHz (asserted by call to hardware).
  cpufreq stats: 200 MHz:22.65%, 300 MHz:8.94%, 600 MHz:0.09%, 1.20 GHz:68.32%  (1637)

 

root@ebinv7:/boot# lscpu
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       ARM
Model:                           4
Model name:                      Cortex-A53
Stepping:                        r0p4
CPU max MHz:                     1200.0000
CPU min MHz:                     200.0000
BogoMIPS:                        25.00
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid

 

 

Has this bug been reported and scheduled for fixing?

 

On the other side, why do I still receive 2 upgrades available, while 22.08 is already setup?

Welcome to Armbian 22.08.6 Bullseye with Linux 5.10.60-mvebu64

root@ebinv7:/boot# apt list --upgradable
Listing... Done
linux-dtb-current-mvebu64/bullseye 22.08.6 arm64 [upgradable from: 21.08.1]
linux-image-current-mvebu64/bullseye 22.08.6 arm64 [upgradable from: 21.08.1]

[Pali]

On 10/23/2022 at 9:52 PM, y52 said:

[    2.420702] Unsupported CPU frequency 1200 MHz

 

Here is the issue. 1.2 GHz variant of A3720 CPU is not supported by Linux. Trying to do frequency scaling on that variant cause CPU crashes. Hence it was disabled.

 

On 10/25/2022 at 10:48 PM, y52 said:

Has this bug been reported?

 

Yes, on many places and there is just a silence:

https://lore.kernel.org/linux-pm/20210809040224.j2rvopmmqda3utc5@vireshk-i7/

https://github.com/MarvellEmbeddedProcessors/linux-marvell/issues/20

 

On 10/23/2022 at 8:55 PM, y52 said:

What could be done to fix this issue?

 

Contacting your support departement or directly Marvell FAE. Opening RMA or returning broken product back to seller.

[y52]

Hello Pali,

Thank you again for pointing out this flaw. I recall your alert earlier this year :

https://forum.armbian.com/topic/19564-making-espressobin-v7-work-in-2022/page/2/#

Beware, 1.2 GHz CPU is broken and Marvell refused to fix it. Hence kernel maintainers decided to delete/deactive 1.2 GHz mode.  https://lore.kernel.org/linux-pm/20210809040224.j2rvopmmqda3utc5@vireshk-i7/

So rather stick with 1 GHz mode.

 

Now kernel upgrade reminded this bug again. It remains yet unclear to me, why the 5.10.60-mvebu64  kernel works on 1.2 GHz variant of A3720 SOC and apparently does frequency scaling, while the next 5.15 tree does not. If init sequence related to CPU voltage configuration satisfy the 5.10 tree, why is it no more valid for 5.15?

 

I have another ancillary issue with uboot implementation. When I followed your u-boot build guidance in February 2022, I didn't implement U-Boot distroboot feature by that time as Armbian distribution was not yet ready. Thus I haven't set default environment in u-boot so far. Armbian is currently passed from 
boot_targets=usb sata mmc1 mmc0
bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done

instead if distroboot implementation :

bootcmd=run distro_bootcmd
 

What are implications of not setting default environment (=> env default -a) in u-boot with "distroboot Armbian"? 

 

[Pali]

1 minute ago, y52 said:

Now kernel upgrade reminded this bug again. It remains yet unclear to me, why the 5.10.60-mvebu64  kernel works on 1.2 GHz variant of A3720 SOC and apparently does frequency scaling, while the next 5.15 tree does not. If init sequence related to CPU voltage configuration satisfy the 5.10 tree, why is it no more valid for 5.15?

 

The whole thing is broken. For 1.2 GHz variant is officially documented CPU errata related to CPU voltage together with workaround. But source of the issue is not documented and is Marvell secret. That workaround is implemented in kernel for a long time (including 5.15). But for some unknown reasons since some kernel version 1.2 GHz variant of CPU crashes even with workaround. And because only Marvell knows source of one documented crahes, we can just guess... if this that documented issue, or some other issue... Only HW engineer of that A3720 CPU can answer to those questions.

 

Anyway, if somebody is interested in, wants to debug this issue and is skilled in compiling, patching and debugging kernel, I can forward some new test info which I have (write me an email).

[Pali]

On 11/1/2022 at 10:18 PM, Pali said:

Anyway, if somebody is interested in, wants to debug this issue and is skilled in compiling, patching and debugging kernel, I can forward some new test info which I have (write me an email).

 

Just reminding, nobody?

[y52]

Sadly, there is nobody.  1.2 GHz variant of A3720 will remain flawed.

[ManoftheSea]

@Pali

Hi!

I've run into this issue.  I've got an ebin-v5 with cpu frequency stepping, and an ebin-v7 without, both on 5.15.86.  Based on the CPU markings, the v7 is using platform firmware set for 1200 MHz. 

 

I'd be interested in seeing what ysou know, but claim no special skill.  Likely, I'll just drop the u-boot config back to 1000 MHz even on the v7, if that solves it.

[y52]

kernel 5.10.60 is the last version I've managed running  v7 @ 1200 MHz with cpu frequency stepping.

Thus, if you want the latest kernel, I fear you would need to downgrade your u-boot to 1000 MHz.

I don't believe there were recent development debugging this issue, although there is a need for.

[ManoftheSea]

I switched to NixOS back in December, and so far have not been able to complete compiling u-boot on that system with any toolchain.  The TF-A code needs to get into marvell git repos instead of just gathering binaries, it's not very clean.  As such, I don't have any update on the issue either.

[Pali]

Just in case if somebody want to test or debug this issue please let me know...