Increased idle power consumption when booting from eMMC vs microSD

[ct100]

I have Armbian running on a NanoPi R6C 8GB with 32GB eMMC.

 

I made a new Armbian image with the build framework that uses newest 

 

linux-u-boot-nanopi-r6s-legacy 23.8.0-trunk

linux-dtb-legacy-rk35xx 23.8.0-trunk

linux-image-legacy-rk35xx 23.8.0-trunk

 

Booted from microSD this system has an idle power consumption of 0,92w. I can access both the eMMC and microSD storage.

 

When i transfer this system from microSD to eMMC with "nand-sata-install" to boot & OS on eMMC and cold boot without the microSD, the idle power consumption increases to 1,21w.

 

After a shutdown and booting from the microSD again the idle power consumption is back at 0,92w.

 

 

So far i have not been able to find a reason for this. The only things i noticed was:

 

(microSD)
Device          Start      End  Sectors  Size Type
/dev/mmcblk1p1  32768   557055   524288  256M Linux extended boot
/dev/mmcblk1p2 557056 61702144 61145089 29.2G Linux filesystem

(eMMC)
Device          Start      End  Sectors  Size Type
/dev/mmcblk0p1  32768 59998207 59965440 28.6G Linux filesystem

 

While microSD uses a vfat partition for /boot and an EXT4 partition for the OS the eMMC has a single EXT4 partition for everything. Does that change how the system is booting?

 

 

https://github.com/radxa/u-boot/commit/bf47e8171f44804c7f2004f53d3f05801f8dbbb8

Different drive strength settings in u-boot for when eMMC is boot drive or not? Can that really increase the power usage of the eMMC by 0,3w?

 

Edited August 21, 2023 by ct100

[Dantes]

There is a thread on OpenWRT, where it show it can be installed on sdcard or emmc. You could try installing those and measuring the power usage between sdcard and emmc. If the problem is the same, I'll bet its somehow related to the emmc driver.

 

link: https://forum.openwrt.org/t/nanopi-r6s-linux-6-3-arm-soc-updates/153072/34

link to images for nanopi: https://github.com/mj22226/openwrt/releases/tag/linux-6.4

 

 

[ct100]

I found the reason this happens. Its not the eMMC itself, the boot from eMMC or booting from FAT vs EXT4. It's the much faster system boot speed from eMMC and the network configuration that leads to this bug.

 

The device that consumes the additional idle power is the Realtek RTL8125 2.5GbE NIC when left in an bugged unmanaged/unconfigured/off state.

 

When using stock Armbian with Network-Manager it seems to depend on the image, software distribution and the system boot speed with how often the Realtek NIC gets into this bugged state. Properly configuring the NIC in Network-Manager prevents this from happening.

 

 

When using Armbian Bullseye with OpenMediaVault installed with the OMVExtras InstallScript this can be reproduced 100%. The InstallScript (without using the option -n to skip network configuration) removes Network-Manager and lets systemd-networkd and OpenMediaVault/Netplan manage the network.

 

Since the MAC of the Realtek RTL8125 2.5GbE NIC is not fixed and changes every boot systemd-networkd with the policy "MACAddressPolicy=persistent" (default in /usr/lib/systemd/network/99-default.link) automatically adds a persistent MAC address to the Adapter. At the same time OpenMediaVault creates netplan files that assign network configuration based on this added MAC address.

 

This leads to a race condition in systemd-networkd. If systemd-networkd adds the persistent MAC to the adapter before it tries to assign the configuration matched to this persistent MAC everything works as expected. When the system boots faster systemd-networkd tries to match the configuration to a MAC that is not added yet to the adapter and that leaves the adapter off, unmanaged and bugged.

 

(In these code parts eth0 = GMAC & Realtek RTL8211F 1GbE and Realtek RTL8125 2.5GbE NIC = enP3p49s0)

 

Random MAC adresses:

Aug 28 09:53:57 nanopi-r6s kernel: r8125 0003:31:00.0 (unnamed net_device) (uninitialized): Invalid ether addr 00:00:00:00:00:00
Aug 28 09:53:57 nanopi-r6s kernel: r8125 0003:31:00.0 (unnamed net_device) (uninitialized): Random ether addr 4e:54:94:b5:ad:c7

 

OMV Netplan file (matching with mac!):

root@nanopi-r6s:~# cat /etc/netplan/20-openmediavault-enP3p49s0.yaml
network:
  ethernets:
    enP3p49s0:
      match:
        macaddress: de:d1:aa:55:c9:bf
      accept-ra: true
      dhcp4: yes
      dhcp4-overrides:
        use-dns: true
        use-domains: true
      dhcp6: yes
      dhcp6-overrides:
        use-dns: true
        use-domains: true

 

Netplan auto generated systemd-networkd .network:

root@nanopi-r6s:~# cat /run/systemd/network/10-netplan-enP3p49s0.network
[Match]
MACAddress=de:d1:aa:55:c9:bf

[Network]
DHCP=yes
LinkLocalAddressing=ipv6
IPv6AcceptRA=yes

[DHCP]
RouteMetric=100
UseMTU=true
UseDomains=true

 

When booting from slow microSD status of enP3p49s0 is ok:

root@nanopi-r6s:~# networkctl
IDX LINK      TYPE     OPERATIONAL SETUP
  1 lo        loopback carrier     unmanaged
  2 eth0      ether    routable    configured
  3 enP3p49s0 ether    no-carrier  configuring

3 links listed.
root@nanopi-r6s:~# networkctl status enP3p49s0
● 3: enP3p49s0
                     Link File: /usr/lib/systemd/network/99-default.link
                  Network File: /run/systemd/network/10-netplan-enP3p49s0.network
                          Type: ether
                         State: no-carrier (configuring)
                          Path: platform-fe180000.pcie-pci-0003:31:00.0
                        Driver: r8125
                        Vendor: Realtek Semiconductor Co., Ltd.
                         Model: RTL8125 2.5GbE Controller
                    HW Address: de:d1:aa:55:c9:bf
          HW Permanent Address: 4e:54:94:b5:ad:c7
                           MTU: 1500 (min: 68, max: 9194)
                         QDisc: mq
  IPv6 Address Generation Mode: eui64
          Queue Length (Tx/Rx): 4/4
              Auto negotiation: yes
                         Speed: n/a
                          Port: tp
             DHCP6 Client DUID: DUID-EN/Vendor:0000ab1166a0671791d79f890000

 

When booting from much faster eMMC enP3p49s0 is left in a bugged state:

root@nanopi-r6s:~# networkctl
IDX LINK      TYPE     OPERATIONAL SETUP
  1 lo        loopback carrier     unmanaged
  2 eth0      ether    routable    configured
  3 enP3p49s0 ether    off         unmanaged

3 links listed.
root@nanopi-r6s:~# networkctl status enP3p49s0
● 3: enP3p49s0
                     Link File: /usr/lib/systemd/network/99-default.link
                  Network File: n/a
                          Type: ether
                         State: off (unmanaged)
                          Path: platform-fe180000.pcie-pci-0003:31:00.0
                        Driver: r8125
                        Vendor: Realtek Semiconductor Co., Ltd.
                         Model: RTL8125 2.5GbE Controller
                    HW Address: de:d1:aa:55:c9:bf
          HW Permanent Address: 92:f6:18:f3:59:af
                           MTU: 1500 (min: 68, max: 9194)
                         QDisc: noop
  IPv6 Address Generation Mode: eui64
          Queue Length (Tx/Rx): 4/4
              Auto negotiation: yes
                         Speed: n/a
                          Port: tp

 

The fix is to install OpenMediaVault without re configuring the network with InstallScript option -n. That keeps Network-Manger and doesn't add the ethernet adapters to the OMV database.

 

Another option is to remove Network-Manager and also remove the matching by MAC address from OMV netplan files by editing "/srv/salt/omv/deploy/systemd-networkd/netplan/files/ethernet.j2" and deleting:

      match:
        macaddress: {{ mac_address }}

After saving and applying the new network configuration in OMV webgui the MAC address matching is gone and adapters get only matched by name.

 

Edited August 28, 2023 by ct100

[ct100]

Update: I reported this bug to OpenMediaVault and it was fixed in OpenMediaVault Version 6.8.0.

 

https://github.com/openmediavault/openmediavault/commit/82c74b07fb7f57a6c55832f6e3aa17ba6b67b55d

(Issue #1589: Fix a problem in the netplan.io configuration for ethernet network interfaces with non-permanent MAC addresses.)

 

 

For stock Armbian with Network-Manager you still have to make sure that the Realtek RTL8125 2.5GbE NIC is correctly configured.

[q-bert]

@ct100 How can I make sure my NIC is configured correctly?

 

cat /etc/armbian-image-release
# PLEASE DO NOT EDIT THIS FILE
BOARD=rock-5b
BOARD_NAME="Rock 5B"
BOARDFAMILY=rockchip-rk3588
BUILD_REPOSITORY_URL=https://github.com/armbian/build
BUILD_REPOSITORY_COMMIT=1e6fdcb13
LINUXFAMILY=rk35xx
ARCH=arm64
IMAGE_TYPE=stable
BOARD_TYPE=csc
INITRD_ARCH=arm64
KERNEL_IMAGE_TYPE=Image
FORCE_BOOTSCRIPT_UPDATE=
FORCE_UBOOT_UPDATE=
VENDOR="Armbian"
VENDORDOCS="https://docs.armbian.com"
VENDORURL="https://www.armbian.com/"
VENDORSUPPORT="https://forum.armbian.com"
VENDORBUGS="https://www.armbian.com/bugs"
BOOTSCRIPT_FORCE_UPDATE="no"
BOOTSCRIPT_DST="boot.cmd"
VERSION=23.2.1
REVISION=23.2.1
IMAGE_UUID=7c90ea8a-9f70-45ed-af73-86d1927c2fbc

uname -a
Linux rock5b 5.10.160-legacy-rk35xx

 

[ct100]

@q-bert Use the command "nmcli dev status" (for Network-Manager) or "networkctl" (for systemd-networkd) and make sure the NIC is not off / unmanaged.

[q-bert]

@ct100 Thank you! "nmcli dev status" did the job (I wasn't aware of it). I tried ""networkctl" before and all interfaces were unmanaged. Looks good now:

 

nmcli dev status
DEVICE           TYPE      STATE                   CONNECTION
enP4p65s0        ethernet  connected               Armbian ethernet
docker0          bridge    connected (externally)  docker0

 

Edited 3 hours ago by q-bert