Jump to content

Regression: Increased Power Consumption with Kernel 5.10.160

ct100

By ct100
in Rockchip

 Share
Go to solution Solved by ct100,

Recommended Posts

ct100

ct100

Posted
Posted (edited)

I did a lot of power consumption tests with my NanoPi R6C and noticed something strange when switching from kernel 5.10.110 to kernel 5.10.160.

 

Hardware

NanoPi R6C 4GB

32GB Sandisk High Endurance microSD Card

Ugreen 18W USB Power Supply

 

Software

Armbian 23.5.1 Nanopi-r6s bookworm

Default settings (only changing fdtfile=rockchip/rk3588s-nanopi-r6s.dtb to fdtfile=rockchip/rk3588s-nanopi-r6c.dtb in /boot/armbianEnv.txt)

Switching between legacy 5.10.160 and legacy 5.10.110 kernels with armbian-config

 

With just 1Gbit Ethernet and the power supply connected in idle (measured at wall):

 

kernel 5.10.110: 0,92W

kernel 5.10.160: 1,21W

 

I also tested different M.2 SSD, with ASPM L1 enabled/disabled, HDMI and USB devices connected/not connected. I even tried different usb power supply. In every case the power consumption with kernel 5.10.160 is higher with no apparent benefit. You could argue that it is not a big difference but when running the system from a battery it is!

 

What is causing this increased power consumption? I tried to run a lot of commands to find the difference between the 2 kernels but could not find a significant one.

 

WinDiff between the 2 kernels:

 

  Reveal hidden contents

 

 

Edited by ct100
0
Dantes

Dantes

Posted
Posted

Running from a battery, very interesting.. I'll be following this thread ;)

 

I'm still on the edge to buy an M.2 because a lot of them are power hogs.

So what brand/type M.2 did you use ?

 

0
ct100

ct100

Posted
  • Author
Posted

M.2 NVMe flash drives are only power hogs when you buy a drive with a controller manufactured in an old process node, lots of dram cache, many inefficient flash chips with low layer count and then run them without power saving modes enabled.

 

A modern M.2 NVMe flash drive with power saving modes active (APST, ASPM L1 + L1 substates) uses less than 0,05W in idle.

 

NanoPi R6C 4GB + 32GB SanDisk mSD + Lexar NM790 2TB + Ugreen CD122 70273 18W = 0,85W idle

(1G connection via WAN-port, no HDMI/USB, SSD APST active, PCIe ASPM L1 active)

 

For 4TB drives Lexar NM790 is prob most efficient drive too and it is single sided so it can fit inside the NanoPi.

 

 

Btw. i am still wondering where the increased power consumption in kernel 5.10.160 comes from... doesnt seem to be cpu. Could it be RAM or a device that isnt used but still is active?!

 

0
Dantes

Dantes

Posted
Posted

I think it might be related to the USB3.0 port , which does not work in 5.10.160. A fix has been submitted for that and has been accepted, so its waiting on the 170.

 

That Lexar is mighty interesting, but still a bit pricey for the 4TB. Does you 2TB fit properly or did you leave the bottom out ?

 

 

 

 

0
ct100

ct100

Posted
  • Author
Posted
  On 8/8/2023 at 6:27 PM, Dantes said:

I think it might be related to the USB3.0 port , which does not work in 5.10.160. A fix has been submitted for that and has been accepted, so its waiting on the 170.

Expand  

 

I don't think its the USB port. I did more tests today and what i noticed was...

 

Fresh install of "Armbian_23.5.1_Nanopi-r6s_bookworm_legacy_5.10.160_minimal" with just .dtb change started cold: 0,92W idle (same as Kernel 5.10.110)

Restarting the system instead of cold booting: 1,21W idle

 

But if i install all the updates with apt even a cold boot cannot get the system back to 0,92W idle, it stays permanently on 1,21W idle.

 

Is it possible that this is caused by the bootloader? I wonder why there is a difference between restart and cold boot with a fresh install.

 

 

  On 8/8/2023 at 6:27 PM, Dantes said:

That Lexar is mighty interesting, but still a bit pricey for the 4TB. Does you 2TB fit properly or did you leave the bottom out ?

Expand  

 

All single-sided drives without a heat sink fit inside the NanoPi and you can close the bottom. The Lexar fits. Only double-sided drives are a problem. Easy to see if you take a look at this Picture.

0
Dantes

Dantes

Posted
Posted
  On 8/9/2023 at 7:50 AM, ct100 said:

But if i install all the updates with apt even a cold boot cannot get the system back to 0,92W idle, it stays permanently on 1,21W idle.

Expand  

Did you check running services ? There might be some extra services installed after the updates.

 

  On 8/9/2023 at 7:50 AM, ct100 said:

Is it possible that this is caused by the bootloader? I wonder why there is a difference between restart and cold boot with a fresh install.

Expand  

 

You could use the u-boot bootloader that comes with OpenWRT (unofficial as of yet) to verify, but that requires some tinkering. U-Boot SPL 2023.04-OpenWrt-r0-1f3cc70 (May 21 2023) that can be found here:

 

https://forum.openwrt.org/t/nanopi-r6s-linux-6-3-arm-soc-updates/153072/15

0
ct100

ct100

Posted
  • Author
Posted
  On 8/9/2023 at 2:32 PM, Dantes said:

Did you check running services ? There might be some extra services installed after the updates.

Expand  

 

I did check that. I also spend some time analyzing CPU frequencies and CPU core idle times with cpufrequtils and powertop. There is no difference that would explain such higher idle power consumption.

 

Whats interesting is that i managed to reproduce the same problem on kernel 5.10.110 by moving a system from micro-sd to emmc with armbian-config.

The system running from micro-sd: 0,92W idle

Same system copied and started from eMMC: 1,21W idle

(and this effect is not because the eMMC uses more power)

 

Is there a different boot loader or a different boot loader configuration for booting from emmc? Is it possible that the boot loader permanently activates an unused device?

 

Does armbian-config directly copy u-boot from micro-sd to emmc or is that downloaded from a repo?

 

Is there a difference between a cold start and a reboot with u-boot and how it configures the system?

 

 

  On 8/9/2023 at 2:32 PM, Dantes said:

You could use the u-boot bootloader that comes with OpenWRT (unofficial as of yet) to verify, but that requires some tinkering. U-Boot SPL 2023.04-OpenWrt-r0-1f3cc70 (May 21 2023) that can be found here:

 

https://forum.openwrt.org/t/nanopi-r6s-linux-6-3-arm-soc-updates/153072/15

Expand  

 

I will try that after i read more about u-boot. It would be nice to have a newer u-boot directly from armbian.

 

 

Btw. does any1 know if its possible to read out the negotiated USB PD modes with the NanoPi R6x like some Rock 5 Models can? I want to rule out any USB PD quirks.

0
ct100

ct100

Posted
  • Author
Posted

It seems the armbian images i was testing with had both kernel (armbian/linux-rockchip) and u-boot (radxa/u-boot) from april or may 2023.

 

I made a completely new Armbian 23.08.0-trunk image with the armbian build script that included kernel 5.10.160 and u-boot 2017.09 with the latest commits (2023-08-08).

 

With the new image booted from sd card i can reboot the system as many times as i want and install all the updates without permanently increasing the idle power consumption. It always returns to 0,92W. So my NanoPi R6C 4GB without emmc are fine now with the new image :)

 

But sadly the problem is still there when booting from emmc for the NanoPi R6C 8GB model.

 

After copying the system from sd to emmc with nand-sata-install and booting from emmc the idle power consumption is back at 1,21W. And this is not due to increased power usage from emmc vs sd. When the same NanoPi with the same armbian system is booted from sd with both sd and emmc active the idle power consumption returns to 0,92W.

0
Dantes

Dantes

Posted
Posted (edited)
device emmc-boot sdcard-boot idle updates reboot
NanoPi r6c 4GB no-emmc n/a yes 0.92 0.92 0.92
NanoPi r6c 8GB emmc 32GB no yes 0.92 0.92 0.92
  yes no 1.21 1.21 1.21
           
           

 

Is the above correct ?

 

The difference afaict is that the OS is running from the *emmc* but  you could test if it has something to do with the boot sequence by:
 

1. Backup and checksum EMMC boot image 

# dd if=/dev/mmcblk2 bs=16M count=1 > /tmp/emmc-boot.img
# sha256sum /tmp/emmc-boot.img | tee emmc-boot.img.sha256sum
# strings /tmp/emmc-boot.img | grep 'U-Boot SPL'

 

2. Backup and checksum sdcard boot image 

# dd if=/dev/mmcblk0 bs=16M count=1 > /tmp/sdcard-boot.img
# sha256sum /tmp/sdcard-boot.img | tee sdcard-boot.img.sha256sum
# strings /tmp/sdcard-boot.img | grep 'U-Boot SPL'

 

If the output differs you could transfer the sdcard boot image to the emmc by:

 

1. Backup EMMC partition table 

# sgdisk --backup=/tmp/partition-table.original /dev/mmcblk2

 

2. Copy all created files to the sdcard from /tmp

 

then:

 

3. Write sdcard-boot.img to EMMC 

# dd if=/tmp/sdcard-boot.img bs=16M count=1 of=/dev/mmcblk2
1+0 records in
1+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0,0489076 s, 343 MB/s

 

4. Restore EMMC partition table 

# sgdisk --load-backup=/tmp/partition-table.original /dev/mmcblk2
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.

 

5. Reboot and test with the power meter :lol:

 

 

 

If it fails, or if you want to revert, you can boot from sdcard and then write emmc-boot.img  back to the emmc: 

# dd if=emmc-boot.img bs=16M count=1 >/dev/mmcblk2

 

 

Good luck,

 

Dantes

Edited by Dantes
0
ct100

ct100

Posted
  • Author
Posted (edited)

I read a lot about u-boot and checked the source code that writes the bootloader to virtual sd (in build scripts) and to the mmc (with platform_install.sh).

 

They are identical files and written like: 

dd if=$1/idbloader.img of=$2 seek=64 conv=notrunc
dd if=$1/u-boot.itb of=$2 seek=16384 conv=notrunc

(If you don't use any seek= to skip blocks you are reading and overwriting the partition table too.)

 

What i didn't understand at the start was that its not just the kernel. This is a system of u-boot + dtb files + kernel.

 

I now also realize that im dealing with 2 bugs here.

 

1. Bug with different kernels / dtb files between armbian-versions that increases idle power consumption.

2. Bug with booting from mmc instead of sd-card that increases idle power consumption.

 

 

For the first bug that is is fixed in 28.08.0-trunk i noticed the difference in dtb files. Inside the dtb-file compared to older file there is:

 

USB3 fix: dr_mode = "otg" -> dr_mode "host"

Enable CEC: cec-enable = true

Cursor fix that adds "disable-win-move", "cursor-win-id" stuff and "rockchip, plane" stuff.

 

I think one of these fixed the bug (or a change in the kernel?!).

 

From the datasheet of RK3588s:

rk3588s_usb3_dp1.4_combo.png.7254234215a7899634069be20edc5bd7.png

 

Is it possible that this bug is resulting from the DP1.4 lanes that are combined with USB3? And that using USB3 in OTG mode somehow activates them?  Some RK3588s boards have these "USB 3.0 OTG Type-C port with DisplayPort Alt mode" combo ports, while the NanoPi R6C just has a normal USB3 A port. Why does the USB3 port not work in OTG mode at all?

 

Activating the HDMI port on the NanoPi R6C results in ~0,30-0,35W more power consumption and that bug is suspiciously in the same power range.

 

 

For the second bug i have no idea how it is possible to explain the difference when using identical bootloader, dtb-files and kernel. I can see some difference in the bootloader code when booted with mmc as boot drive, like adjusting mmc "drive strength", but i don't think that can explain 0,3W more power consumption.

 

Edited by ct100
0
Dantes

Dantes

Posted
Posted (edited)
  On 8/15/2023 at 6:17 AM, ct100 said:

They are identical files and written like: 

dd if=$1/idbloader.img of=$2 seek=64 conv=notrunc
dd if=$1/u-boot.itb of=$2 seek=16384 conv=notrunc

(If you don't use any seek= to skip blocks you are reading and overwriting the partition table too.)

 

Expand  

Yes, that's how it is done , but "conv=notrunc" only make sense if the output is a file , not  if it is a block device.

 

  On 8/15/2023 at 6:17 AM, ct100 said:

I now also realize that im dealing with 2 bugs here.

Expand  

I made the matrix so I could get a clearer picture of what you where asking, but this adds and extra dimension for sure.

 

Did you try disabling CEC, I reckon you are not using a TV with a remote control ? That might be as you said the cause (HDMI-CEC).

Edited by Dantes
0
  • Solution
ct100

ct100

Posted
  • Author
  • Solution
Posted (edited)

Ok i found the cause of the first bug. It was the USB3 in OTG-Mode!

 

I took a kernel image where the bug is present (linux-image-legacy-rockchip-rk3588/now 23.05.1--5.10.160-S03c9-Db165-P0000-C5e28Hfe66-HK01ba-Vc222-B9c18 arm64) and added an user overlay:

  

/dts-v1/;
/plugin/;

/ {
        fragment@0 {
                target = <&usbdrd_dwc3_0>;

                __overlay__ {
                        dr_mode = "host";
                };
        };
};

 

That fixed the increased idle power consumption.

 

The kernel images on the nightly branch beta.armbian.com (linux-image-legacy-rk35xx_23.8.0-trunk*) all have the fix and work fine.

 

 

I'll mark this post as the solution and make a new thread for the 2nd bug.

 

Edited by ct100
0

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept