Slow WiFi speeds on Banana Pi CM4

[vimal]

I have a Banana Pi CM4 + CM4 IO Board - literally the same one pictured in https://www.armbian.com/bananapicm4io/

 

I downloaded and installed Ubuntu 24.04 (Noble) Server Image - "Armbian_24.8.1_Bananapicm4io_noble_current_6.6.47.img"

For convenience, I installed the image to an SD card.

 

Armbianmonitor link: https://paste.armbian.com/igahozatow

Everything boots and I can use armbian-config to connect to my local WiFi network, but the speeds are really really slow:

iperf3 gives me ~40mbps pretty consistently:

--------------------

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  51.1 MBytes  42.9 Mbits/sec    2             sender
[  5]   0.00-10.20  sec  50.1 MBytes  41.2 Mbits/sec                  receiver

 

I'm connected to a 5GHz network, good connection strength/quality (and I do have antennas plugged into the CM4)

--------------------

wlan0     IEEE 802.11  ESSID:"unimportant"
          Mode:Managed  Frequency:5.22 GHz  Access Point: 70:A7:41:69:BA:14
          Bit Rate=650 Mb/s   Tx-Power=20 dBm
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Encryption key:off
          Power Management:off
          Link Quality=68/70  Signal level=-42 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:2   Missed beacon:0

Other devices on my network consistently push 300-400mbps iperf3. Also if I install and configure a mini-PCIe network card on the CM4IO board, that also gives me 300-400Mbps. 

I suspect the SDIO connection itself might be the bottleneck, though I'm new at this and might be barking up the wrong (device) tree here.

The RTL8822CS module is connected to mmc2 per dmesg output:
-----------------------

[    7.539802] rtw_8822cs mmc2:0001:1: WOW Firmware version 9.9.4, H2C version 15

It appears mmc2 is running at only 25MHz 4-bit mode: cat /sys/kernel/debug/mmc2/ios

-----------------------

clock:          25000000 Hz
actual clock:   25000000 Hz
vdd:            21 (3.3 ~ 3.4 V)
bus mode:       2 (push-pull)
chip select:    0 (don't care)
power mode:     2 (on)
bus width:      2 (4 bits)
timing spec:    0 (legacy)
signal voltage: 0 (3.30 V)
driver type:    0 (driver type B)

Of note - I came across this review/comparison (https://bret.dk/banana-pi-cm4-review/) of the BPi CM4 using the same CM4IO board as I'm using, except those iperf3 results make even less sense - 81 Mbps down / 158 Mbps up "bare", 12 Mbps down / 58 Mbps up using antennas. The author suspects that performance is better with "onboard PCB antennas", except I....don't see anything that looks like it would be an onboard PCB antenna - either on the module itself or in the schematic (https://drive.google.com/file/d/1IXXok1P2OLiW3p8tavkbfEPTGTrM3b-R/view?usp=sharing)

They were using an older kernel / image (unclear where they got it from) so I wasn't able to replicate their findings, but perhaps there was some regression in the mmc controller setup?

I'm not sure if this will help or not, but here are relevant excerpts from device tree on my image for the three mmc controllers: dtc -I fs /sys/firmware/devicetree/base
-------------------------------

mmc@ffe05000 {
        pinctrl-names = "default\0clk-gate";
        pinctrl-0 = <0x30>;
        clock-names = "core\0clkin0\0clkin1";
        cap-sd-highspeed;
        vqmmc-supply = <0x32>;
        bus-width = <0x04>;
        resets = <0x05 0x2d>;
        interrupts = <0x00 0xbe 0x04>;
        clocks = <0x02 0x22 0x02 0x3d 0x02 0x02>;
        vmmc-supply = <0x2f>;
        compatible = "amlogic,meson-axg-mmc";
        pinctrl-1 = <0x31>;
        status = "okay";
        disable-wp;
        reg = <0x00 0xffe05000 0x00 0x800>;
        phandle = <0x130>;
        max-frequency = <0x2faf080>;
        cd-gpios = <0x07 0x2f 0x01>;
};




mmc@ffe07000 {
        pinctrl-names = "default\0clk-gate";
        pinctrl-0 = <0x33 0x34 0x35>;
        clock-names = "core\0clkin0\0clkin1";
        vqmmc-supply = <0x1f>;
        mmc-hs200-1_8v;
        bus-width = <0x08>;
        resets = <0x05 0x2e>;
        interrupts = <0x00 0xbf 0x04>;
        clocks = <0x02 0x23 0x02 0x3e 0x02 0x02>;
        vmmc-supply = <0x2f>;
        mmc-ddr-1_8v;
        compatible = "amlogic,meson-axg-mmc";
        pinctrl-1 = <0x36>;
        status = "okay";
        disable-wp;
        mmc-pwrseq = <0x37>;
        reg = <0x00 0xffe07000 0x00 0x800>;
        phandle = <0x131>;
        max-frequency = <0xbebc200>;
        cap-mmc-highspeed;
};




mmc@ffe03000 {
        pinctrl-names = "default\0clk-gate";
        #address-cells = <0x01>;
        amlogic,dram-access-quirk;
        pinctrl-0 = <0x2c>;
        clock-names = "core\0clkin0\0clkin1";
        vqmmc-supply = <0x2f>;
        bus-width = <0x04>;
        non-removable;
        resets = <0x05 0x2c>;
        interrupts = <0x00 0xbd 0x04>;
        clocks = <0x02 0x21 0x02 0x3c 0x02 0x02>;
        #size-cells = <0x00>;
        vmmc-supply = <0x2f>;
        keep-power-in-suspend;
        compatible = "amlogic,meson-axg-mmc";
        pinctrl-1 = <0x2d>;
        status = "okay";
        disable-wp;
        mmc-pwrseq = <0x2e>;
        reg = <0x00 0xffe03000 0x00 0x800>;
        phandle = <0x12e>;
        sd-uhs-sdr104;
        max-frequency = <0x2faf080>;

        wifi@1 {
                reg = <0x01>;
                phandle = <0x12f>;
        };

};

The RTL8822CS module itself should support SDIO 1.1 / 2.0 / 3.0 up to clock frequency of 208MHz, but it seems like for some reason the controller just isn't running at that speed and I'm not sure what exactly to change to fix that - anyone have any suggestions on what I'm doing wrong or what I should check next to debug this? 


 

 

[vimal]

Continuing down the route of debugging:

I found the geerlingguy review of the CM4 from 2023: https://github.com/geerlingguy/sbc-reviews/issues/11
This referenced a specific image (debian 10 buster) linked from an old wiki page: https://wiki.banana-pi.org/index.php?title=Banana_Pi_BPI-CM4&oldid=13245#Linux

 

I installed that image and was greeted with an ancient 4.9 kernel:
-------------
 

Linux bananapi 4.9.241-BPI-M2S #1 SMP PREEMPT Fri Jun 17 12:23:43 HKT 2022 aarch64 GNU/Linux

 

I enabled overlays="wifi_bt_rtl8822cs" in /boot/firmware/env.txt, restarted, modprobe 88x2cs, configured my wifi info in /etc/network/interfaces.d/wlan0, ifup wlan0 and wifi successfully connected!

I then installed and repeated my iperf3 test and this time got the expected 300-400Mbps - yay!

Of note on above image kernel 4.9.241 and wifi_bt_rtl8822cs overlay, I get the following results for: cat /sys/kernel/debug/sdio/ios
--------------

clock:          200000000 Hz
actual clock:   199999997 Hz
vdd:            21 (3.3 ~ 3.4 V)
bus mode:       2 (push-pull)
chip select:    0 (don't care)
power mode:     2 (on)
bus width:      2 (4 bits)
timing spec:    6 (sd uhs SDR104)
signal voltage: 1 (1.80 V)
driver type:    0 (driver type B)

This looks a lot more appropriate - 200MHz, 100MB/sec bus speed gives plenty of bandwidth for the WiFi chipset to do its thing.

 

And turns out WiFi works a lot better when you add antennas.... (that really needs to go into an onboarding guide)

 

So now I know the hardware is fine and whatever is wrong is down to different kernel, different device tree, or different driver. 
Next step (and would love some help / suggestions here) is to figure out which change caused the problems. 

 

 

[c0rnelius]

1 hour ago, vimal said:

And turns out WiFi works a lot better when you add antennas....

This is the important part here. Use an antenna.

 

What that ancient kernel is running is an old github driver.

[vimal]

I mean....I have always been using the antennas - it's other reviewers who were getting slow speeds with the older kernel who were not using antennas.

Even with antennas, newer kernel + rtw_88 driver is only initializing to 25MHz SDIO speed (100mbit bus speed), while older kernel +  88x2cs initialized to 200MHz SDIO speed (800mbit bus speed)  

I still think the issue is the device tree and not the driver - I guess next step is comparing the wifi overlay for the older kernel to the device tree of the newer kernel 

[c0rnelius]

The build comes with both rtw88 and the github variant. rtw88 gets loaded first "it has priority", so blacklist rtw88 and see if you get better performance.

 

I seriously doubt it's the DTS. But ur more than welcome to go that route. If you find edits to the DTS are beneficial in this department please let us know.

 

EDIT: I just remembered something. During the bring up process, I found the github variant didn't work on this unit. This was the main motivation for backporting rtw88 into Armbian when it was still on 6.1.y.

[vimal]

Okay - I tried blacklisting rtw88 and indeed 88x2cs was loaded, but you're right - it didn't actually work. 

wlan0 shows up in iwconfig but not in ifconfig, and none of the wifi operations (scanning, associating with APs, etc) work correctly.

Out of curiosity if you happen to have a CM4 handy, what kind of iperf3 speeds do you get with rtw88? 

I also tried patching dts to increase the frequency - I set the limit to 200MHz, but the actual bus speed ended up only 50MHz and nothing worked at all (SDIO errors right from the get-go). Next will try poking around to see if I can force I/O voltage to 1.8V and maybe that will nudge this into UHS104 speeds? 

I should also shop around for other boards that have rtl8822cs sdio modules and see if there are any reviews that benchmark wifi performance using rtw88
 

 

[c0rnelius]

Sorry. Both of my CM4's have a heatsink+fan combo on them, making it pretty tough to attach the antennas without removing the setup. I use them both wired.

 

I do have other units with the same SDIO and yes, they all get the same speeds using rtw88. It has been a complaint since day one as far as I know. This was the last discussion we had about the driver: https://github.com/armbian/build/pull/6703#issuecomment-2154619094 which goes into other problems.

 

One option would be; figure out why the github driver doesn't work on the unit and do a PR on the repo. https://github.com/jethome-ru/rtl88x2cs

 

 

 

[vimal]

No problem, and thank you for the pointers - good to know this is a well-known issue and other folks have looked into it before

 

There appears to be some recent-ish (past year) commits on the github driver to support newer kernels - I'll give it a shot and see where I end up.
 

[vimal]

Compiling latest github driver didn't help - same behavior as the one included in the image.

 

I did dig up an older armbian image based on Ubuntu Jammy with 6.0.14 kernel and 88x2cs module installed by default - that worked great.

https://wiki.banana-pi.org/index.php?title=Banana_Pi_BPI-CM4&oldid=14504

"Armbian_23.02.0-trunk_Bananapicm4_jammy_current_6.0.14.img"

 

6.0.x kernel is...new enough and Armbian 23 is a new-enough version of my preferred distro (thank you for the effort maintaining it), so I guess my immediate problem is solved. But I do still want to figure out what broken when.

 

One thing I noticed - 6.0.x kernel dts is configured to support 200MHz, 1.8V I/O, cap-sd-highspeed vs. later DTS that drop down to 50MHz and 3.3V I/O cap-sd-highspeed. From meson-g12b-a311d-bananapi-cm4 in the image with the 6.0 kernel:

sd@ffe03000 {
        compatible = "amlogic,meson-axg-mmc";
        reg = <0x00 0xffe03000 0x00 0x800>;
        interrupts = <0x00 0xbd 0x01>;
        status = "okay";
        clocks = <0x02 0x21 0x02 0x3c 0x02 0x02>;
        clock-names = "core\0clkin0\0clkin1";
        resets = <0x05 0x2c>;
        pinctrl-0 = <0x2d>;
        pinctrl-1 = <0x2e>;
        pinctrl-names = "default\0clk-gate";
        #address-cells = <0x01>;
        #size-cells = <0x00>;
        bus-width = <0x04>;
        cap-sd-highspeed;
        max-frequency = <0xbebc200>;
        sd-uhs-sdr12;
        sd-uhs-sdr25;
        sd-uhs-sdr50;
        sd-uhs-sdr104;
        non-removable;
        disable-wp;
        keep-power-in-suspend;
        mmc-pwrseq = <0x2f>;
        vmmc-supply = <0x30>;
        vqmmc-supply = <0x20>;
        phandle = <0x130>;

        wifi@1 {
                reg = <0x01>;
                phandle = <0x131>;
        };
};

 

 

Next thing I want to check - can I build/install rtw88 on 6.0 kernel and does it work (and does it work better or the same)

Also I wonder what would happen using above DTB with newer kernel - this is all new territory for me so I have no idea if there's any reasonable expectation of compatibility or if that's just a dumb thing to try

 

[vimal]

 Well this is an interesting development:

 

Using 23.02.0-trunk Armbian image with 6.0.14-meson64 kernel and building+installing latest rtw88, I get almost the same good iperf3 speeds:
- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   323 MBytes   271 Mbits/sec  555             sender
[  5]   0.00-10.02  sec   320 MBytes   268 Mbits/sec                  receiver

 

Sure technically 88x2cs was giving me 400-500Mbps while rtw88 is giving me 200-300Mbps, but that's still much better than the 40Mbps I was getting before. 

Furthermore, with both 88x2cs AND rtw88, the SDIO bus is correctly initialized to UHS104 200MHz speeds: cat /sys/kernel/debug/mmc2/ios
 

clock:          200000000 Hz
actual clock:   199999997 Hz
vdd:            21 (3.3 ~ 3.4 V)
bus mode:       2 (push-pull)
chip select:    0 (don't care)
power mode:     2 (on)
bus width:      2 (4 bits)
timing spec:    6 (sd uhs SDR104)
signal voltage: 1 (1.80 V)
driver type:    0 (driver type B)

 

 

[vimal]

And one more interesting development:

 

I start with the latest Armbian 24.8.1 Noble image with Linux 6.6.47-current-meson64 kernel

I copy over the dtb from Armbian_23.02.0-trunk_Bananapicm4_jammy_current_6.0.14.img and configure /boot/armbianEnv.txt to use it instead

 

Restart and viola - wifi is much faster with no other changes - iperf3 results:
 

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   329 MBytes   276 Mbits/sec  952             sender
[  5]   0.00-10.03  sec   327 MBytes   273 Mbits/sec                  receiver

 

But wait - there's more - if I blacklist rtw88_8822cs and restart, the 88x2cs module loads just fine and I get even faster iperf3 results:
 

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   484 MBytes   406 Mbits/sec  496             sender
[  5]   0.00-10.02  sec   481 MBytes   403 Mbits/sec                  receiver

 

 

So this is progress - I'm now quite convinced the issue is definitely in newer device tree since latest kernel, rtw88, and 88x2cs drivers work well with older device tree

 

There are a lot of changes between the the device trees, and when I try the obvious thing of bringing over just the mmc config properties for some reason the device is not detected and initialized correctly. Also when using the older dtb on new image, I get some new errors in dmesg (eg: bluetooth hci0)

Armbian Monitor link for my current setup: https://paste.armbian.com/jubupayiqo

DTS and DTB extracted from older image attached here.

 

@c0rnelius - I'm not sure if this is a helpful finding or not - for now this is good enough for me since I get to use the latest kernel + image and still get good wifi speeds. Any ideas on what motivated changes to the mmc configuration between the 6.0 image and newer images? 
 

meson-g12b-a311d-bananapi-cm4.dts meson-g12b-a311d-bananapi-cm4.dtb

[c0rnelius]

@vimal

 

20 minutes ago, vimal said:

I'm not sure if this is a helpful finding or not - for now this is good enough for me since I get to use the latest kernel + image and still get good wifi speeds. Any ideas on what motivated changes to the mmc configuration between the 6.0 image and newer images? 

 

I can't say for sure; but my guess is that the dts being used was based on this https://github.com/armbian/build/blob/v23.02/patch/kernel/archive/meson64-6.1/add-board-bananapi-m2s-initial-support.patch

 

There was no official CM4 DTS in mainline on 6.0.y and the two boards have a lot in common. I've messed around with &sd_emmc_a node, but I'm personally not seeing any improvements.

0001-sdio-fixups.patch

[vimal]

Well that patch at least got me started looking in the right place - it didn't work by itself, but once I added back the post-power-on-delay-ms that was in the old dtb but not in the current one I did get wifi to successfully come up with 200MHz UHS104 speed.

 

My tweaked version of your patch attached below.

 

Despite the bus speed issue now being fixed, with the latest device tree and this patch, I only get ~100Mbps iperf3 on rtw88 and 88x2cs still refuses to load correctly. So this patch is a substantial improvement for wifi speeds but still not as good as it should be.

 

Next stop - switching from mmc node to sd node - that's the only other significant difference I can see to anything sdio/mmc-related between old and new dts

 

 

 

0001-sdio-fixups-for-wifi.patch

[c0rnelius]

@vimal

 

try this patch with the github driver.

0001-sdio-wifi-fixups.patch

[vimal]

@c0rnelius - I was literally about to send you the same patch!

 

Disabling dram-access-quirk was required to go from 100Mbps on rtw88 + broken 88x2cs to 300Mbps on rtw88 and 500Mbps on 88x2cs

Looks like this was added here: https://patchwork.kernel.org/project/linux-mmc/patch/20190527124307.32075-3-narmstrong@baylibre.com/

 

Some related discussion is here - suggests that the bug impacts both g12a and g12b, but only mentions brcmfmac: https://forum.khadas.com/t/wifi-ap6398s-mainline-kernel/4929/49

I'm going to keep my board running with this change for a few days and see if I run into any stability issues - if it turns out that rtl8822cs with the rtw88 and 88x2cs drivers are not impacted by the SDIO chip bug, then it's probably safe to disable this quirk for this board.

 

 

 

[c0rnelius]

Sweet. Let me know in a few days and if it's a go I'll do a PR.

[c0rnelius]

This patch seems to be more stable when using rtw88. I removed a few bits that shouldn't be required and dropped the freq down a bit.

 

0001-BananaPi-CM4-improve-SDIO-WiFi-speeds.patch

[vimal]

What were you doing to trigger instability in rtw88? I'd like to try and replicate that. 

 

I'm currently 10+hr into iperf3 at a steady 250Mbps (intentionally limited for testing purposes) on 88x2cs and haven't seen so much as a hiccup with the prior patch.

Next up I'm going to try and set this up as some sort of bridge so that I can put a regular day's worth of my laptop traffic through it. 

 

I noticed in your newer patch that you also dropped the post-power-on-delay-ms - are you certain that is unnecessary?

Agreed that the intermediate uhs speeds are unnecessary

One of the reasons I'm interested in pushing the speeds here is that the BPi CM4 right now is the only available 2T2R WiFi 5 equipped module in the CM4 form factor - that means it's the only built-in wifi solution capable of (theoretically) hitting 867Mbps, while all the other CM4-equipped boards have 1T1R WiFi 5 and are capped to (theoretic) 433Mbps. There's a bunch of use cases where this extra speed comes in handy (eg: wireless bridging)

I literally learned what a device tree was like 72 hours ago, so I'm not sure if I'm even asking for the right thing here, but could we have say conservative defaults (100MHz) and then maybe an overlay that enables full speed wifi and a recommendation to use that with 88x2cs and/or a warning of potential stability issues with rtw88?  
 

[c0rnelius]

8 minutes ago, vimal said:

I noticed in your newer patch that you also dropped the post-power-on-delay-ms - are you certain that is unnecessary?

 

 

Adding it doesn't seem to bring anything to the table.

 

After messing around with it further the key changes / additions appear to be:

 

max-frequency = <100000000>;

vqmmc-supply = <&vddao_1v8>;

/delete-property/ amlogic,dram-access-quirk;

 

20 minutes ago, vimal said:

What were you doing to trigger instability in rtw88? I'd like to try and replicate that. 

 

Booting the board. I suspect it's the freq being at 200000000, as I haven't seen it happen since I lowered it.

 

As for adding an overlay to change the max freq, sure I can do that when i submit the final patch.

 

[vimal]

I wonder if there's just some hardware variation? At 200MHz with 200ms post-power-on delay, I didn't see any issues wither either cold boots (physically unplugging and replugging power), resets via the button on the CM4io carrier, or restarts initiated by the OS

 

At 200MHz with no post-power-on delay, I absolutely could not get the SDIO bus to initialize correctly - this was the exact problem I encountered with your very first patch which then prompted me to look for what other attributes I should bring over. 

May I trouble you to try 200MHz with 200ms post-power-on-delay and see if that helps your instability issue? 

I'm going to spend the rest of the day just stressing the current setup - I don't want to power cycle, I want to see if issues creep up while running continuously. 

But after that, I'll try the 100MHz with no delay patch. 

 

[c0rnelius]

32 minutes ago, vimal said:

May I trouble you to try 200MHz with 200ms post-power-on-delay and see if that helps your instability issue? 

I have tried it "post-power-on-delay". With and without it, I didn't see any difference. So I opted to just remove it. If for sure its needed on ur end, I can add it back in. My current theory is that it was caused by the increase in the freq.

 

As for diff revs of the unit? I'm not sure.

 

The current unit I am using to run the wifi tests on is attached to a Waveshare CM4-IO-BASE-B. As that's the only one I could easily attach an antenna to. I wouldn't think the baseboard would create any wifi hiccups though?

 

The one interesting thing I did notice; is wifi was faster when running from SDCARD. One of my units run via NVMe and the other from SATA. Maybe the extra power draw when running from NVMe is effecting the wifi?

 

[vimal]

Right now I'm testing on the Banana Pi CM4IO board, however I have a Waveshare CM4-IO-BASE-B kicking around somewhere and a spare NVMe drive to toss into it - if I can dig it up in the next day or so I can swap parts and test it there. 

 

Do you think it matters if the NVMe drive is hosting the root partition, or is it fine if it's just there and has some sort of I/O activity? 

[c0rnelius]

I would think it just being there is good enough.

[vimal]

Okay now we're into totally insane hardware debugging territory.

 

So I found my CM4-IO-BASE-B as well as a 256GB SKhynix M.2 2230 NVMe drive.

Boot off SD, Original DTB (with 25MHz SDIO) - works just fine with and without NVMe drive

Boot off SD, DTB with 200MHz SDIO + 200ms power-on delay - works jsut fine without NVMe drive

Boot off SD, 256GB NVMe present, DTB with 200MHz SDIO + 200ms power-on delay, USB-C power supply cable plugged in one direction - multiple restarts / attempts to boot but eventually does boot and work as expected (on like the 5th to 10th try judging from the LEDs)

Boot off SD, 256GB NVMe present, DTB with 200MHz SDIO + 200ms power-on delay, USB-C power supply cable flipped - starts right up, first time around, works as expected

There's definitely something funny going on with the power supply situation on the Waveshare CM4-IO-Base-B 🙃

 

The good news is once booted, everything seemed to work - I could run iperf3 and hdparm -t against the nvme simultaneously and there didn't appear to be any impact on each other. 

I think if you offer an overlay that's configured for full 200MHz speed + 200ms power-on delay (I do consistently need that otherwise at 200MHz my module is not initialized), and then have the defaults be the conservative 100MHz that is consistently working for you, that might be a good compromise - certainly better than where we started. 

 

 

[c0rnelius]

https://github.com/armbian/build/pull/7193

[vimal]

Thank you very much - I look forward to the new images once they're available!