vimal

Thank you very much - I look forward to the new images once they're available!

Okay now we're into totally insane hardware debugging territory. So I found my CM4-IO-BASE-B as well as a 256GB SKhynix M.2 2230 NVMe drive. Boot off SD, Original DTB (with 25MHz SDIO) - works just fine with and without NVMe drive Boot off SD, DTB with 200MHz SDIO + 200ms power-on delay - works jsut fine without NVMe drive Boot off SD, 256GB NVMe present, DTB with 200MHz SDIO + 200ms power-on delay, USB-C power supply cable plugged in one direction - multiple restarts / attempts to boot but eventually does boot and work as expected (on like the 5th to 10th try judging from the LEDs) Boot off SD, 256GB NVMe present, DTB with 200MHz SDIO + 200ms power-on delay, USB-C power supply cable flipped - starts right up, first time around, works as expected There's definitely something funny going on with the power supply situation on the Waveshare CM4-IO-Base-B 🙃 The good news is once booted, everything seemed to work - I could run iperf3 and hdparm -t against the nvme simultaneously and there didn't appear to be any impact on each other. I think if you offer an overlay that's configured for full 200MHz speed + 200ms power-on delay (I do consistently need that otherwise at 200MHz my module is not initialized), and then have the defaults be the conservative 100MHz that is consistently working for you, that might be a good compromise - certainly better than where we started.

Right now I'm testing on the Banana Pi CM4IO board, however I have a Waveshare CM4-IO-BASE-B kicking around somewhere and a spare NVMe drive to toss into it - if I can dig it up in the next day or so I can swap parts and test it there. Do you think it matters if the NVMe drive is hosting the root partition, or is it fine if it's just there and has some sort of I/O activity?

I wonder if there's just some hardware variation? At 200MHz with 200ms post-power-on delay, I didn't see any issues wither either cold boots (physically unplugging and replugging power), resets via the button on the CM4io carrier, or restarts initiated by the OS At 200MHz with no post-power-on delay, I absolutely could not get the SDIO bus to initialize correctly - this was the exact problem I encountered with your very first patch which then prompted me to look for what other attributes I should bring over. May I trouble you to try 200MHz with 200ms post-power-on-delay and see if that helps your instability issue? I'm going to spend the rest of the day just stressing the current setup - I don't want to power cycle, I want to see if issues creep up while running continuously. But after that, I'll try the 100MHz with no delay patch.

What were you doing to trigger instability in rtw88? I'd like to try and replicate that. I'm currently 10+hr into iperf3 at a steady 250Mbps (intentionally limited for testing purposes) on 88x2cs and haven't seen so much as a hiccup with the prior patch. Next up I'm going to try and set this up as some sort of bridge so that I can put a regular day's worth of my laptop traffic through it. I noticed in your newer patch that you also dropped the post-power-on-delay-ms - are you certain that is unnecessary? Agreed that the intermediate uhs speeds are unnecessary One of the reasons I'm interested in pushing the speeds here is that the BPi CM4 right now is the only available 2T2R WiFi 5 equipped module in the CM4 form factor - that means it's the only built-in wifi solution capable of (theoretically) hitting 867Mbps, while all the other CM4-equipped boards have 1T1R WiFi 5 and are capped to (theoretic) 433Mbps. There's a bunch of use cases where this extra speed comes in handy (eg: wireless bridging) I literally learned what a device tree was like 72 hours ago, so I'm not sure if I'm even asking for the right thing here, but could we have say conservative defaults (100MHz) and then maybe an overlay that enables full speed wifi and a recommendation to use that with 88x2cs and/or a warning of potential stability issues with rtw88?

@c0rnelius - I was literally about to send you the same patch! Disabling dram-access-quirk was required to go from 100Mbps on rtw88 + broken 88x2cs to 300Mbps on rtw88 and 500Mbps on 88x2cs Looks like this was added here: https://patchwork.kernel.org/project/linux-mmc/patch/20190527124307.32075-3-narmstrong@baylibre.com/ Some related discussion is here - suggests that the bug impacts both g12a and g12b, but only mentions brcmfmac: https://forum.khadas.com/t/wifi-ap6398s-mainline-kernel/4929/49 I'm going to keep my board running with this change for a few days and see if I run into any stability issues - if it turns out that rtl8822cs with the rtw88 and 88x2cs drivers are not impacted by the SDIO chip bug, then it's probably safe to disable this quirk for this board.

Well that patch at least got me started looking in the right place - it didn't work by itself, but once I added back the post-power-on-delay-ms that was in the old dtb but not in the current one I did get wifi to successfully come up with 200MHz UHS104 speed. My tweaked version of your patch attached below. Despite the bus speed issue now being fixed, with the latest device tree and this patch, I only get ~100Mbps iperf3 on rtw88 and 88x2cs still refuses to load correctly. So this patch is a substantial improvement for wifi speeds but still not as good as it should be. Next stop - switching from mmc node to sd node - that's the only other significant difference I can see to anything sdio/mmc-related between old and new dts 0001-sdio-fixups-for-wifi.patch

And one more interesting development: I start with the latest Armbian 24.8.1 Noble image with Linux 6.6.47-current-meson64 kernel I copy over the dtb from Armbian_23.02.0-trunk_Bananapicm4_jammy_current_6.0.14.img and configure /boot/armbianEnv.txt to use it instead Restart and viola - wifi is much faster with no other changes - iperf3 results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 329 MBytes 276 Mbits/sec 952 sender [ 5] 0.00-10.03 sec 327 MBytes 273 Mbits/sec receiver But wait - there's more - if I blacklist rtw88_8822cs and restart, the 88x2cs module loads just fine and I get even faster iperf3 results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 484 MBytes 406 Mbits/sec 496 sender [ 5] 0.00-10.02 sec 481 MBytes 403 Mbits/sec receiver So this is progress - I'm now quite convinced the issue is definitely in newer device tree since latest kernel, rtw88, and 88x2cs drivers work well with older device tree There are a lot of changes between the the device trees, and when I try the obvious thing of bringing over just the mmc config properties for some reason the device is not detected and initialized correctly. Also when using the older dtb on new image, I get some new errors in dmesg (eg: bluetooth hci0) Armbian Monitor link for my current setup: https://paste.armbian.com/jubupayiqo DTS and DTB extracted from older image attached here. @c0rnelius - I'm not sure if this is a helpful finding or not - for now this is good enough for me since I get to use the latest kernel + image and still get good wifi speeds. Any ideas on what motivated changes to the mmc configuration between the 6.0 image and newer images? meson-g12b-a311d-bananapi-cm4.dts meson-g12b-a311d-bananapi-cm4.dtb

Well this is an interesting development: Using 23.02.0-trunk Armbian image with 6.0.14-meson64 kernel and building+installing latest rtw88, I get almost the same good iperf3 speeds: - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 323 MBytes 271 Mbits/sec 555 sender [ 5] 0.00-10.02 sec 320 MBytes 268 Mbits/sec receiver Sure technically 88x2cs was giving me 400-500Mbps while rtw88 is giving me 200-300Mbps, but that's still much better than the 40Mbps I was getting before. Furthermore, with both 88x2cs AND rtw88, the SDIO bus is correctly initialized to UHS104 200MHz speeds: cat /sys/kernel/debug/mmc2/ios clock: 200000000 Hz actual clock: 199999997 Hz vdd: 21 (3.3 ~ 3.4 V) bus mode: 2 (push-pull) chip select: 0 (don't care) power mode: 2 (on) bus width: 2 (4 bits) timing spec: 6 (sd uhs SDR104) signal voltage: 1 (1.80 V) driver type: 0 (driver type B)

Compiling latest github driver didn't help - same behavior as the one included in the image. I did dig up an older armbian image based on Ubuntu Jammy with 6.0.14 kernel and 88x2cs module installed by default - that worked great. https://wiki.banana-pi.org/index.php?title=Banana_Pi_BPI-CM4&oldid=14504 "Armbian_23.02.0-trunk_Bananapicm4_jammy_current_6.0.14.img" 6.0.x kernel is...new enough and Armbian 23 is a new-enough version of my preferred distro (thank you for the effort maintaining it), so I guess my immediate problem is solved. But I do still want to figure out what broken when. One thing I noticed - 6.0.x kernel dts is configured to support 200MHz, 1.8V I/O, cap-sd-highspeed vs. later DTS that drop down to 50MHz and 3.3V I/O cap-sd-highspeed. From meson-g12b-a311d-bananapi-cm4 in the image with the 6.0 kernel: sd@ffe03000 { compatible = "amlogic,meson-axg-mmc"; reg = <0x00 0xffe03000 0x00 0x800>; interrupts = <0x00 0xbd 0x01>; status = "okay"; clocks = <0x02 0x21 0x02 0x3c 0x02 0x02>; clock-names = "core\0clkin0\0clkin1"; resets = <0x05 0x2c>; pinctrl-0 = <0x2d>; pinctrl-1 = <0x2e>; pinctrl-names = "default\0clk-gate"; #address-cells = <0x01>; #size-cells = <0x00>; bus-width = <0x04>; cap-sd-highspeed; max-frequency = <0xbebc200>; sd-uhs-sdr12; sd-uhs-sdr25; sd-uhs-sdr50; sd-uhs-sdr104; non-removable; disable-wp; keep-power-in-suspend; mmc-pwrseq = <0x2f>; vmmc-supply = <0x30>; vqmmc-supply = <0x20>; phandle = <0x130>; wifi@1 { reg = <0x01>; phandle = <0x131>; }; }; Next thing I want to check - can I build/install rtw88 on 6.0 kernel and does it work (and does it work better or the same) Also I wonder what would happen using above DTB with newer kernel - this is all new territory for me so I have no idea if there's any reasonable expectation of compatibility or if that's just a dumb thing to try

No problem, and thank you for the pointers - good to know this is a well-known issue and other folks have looked into it before There appears to be some recent-ish (past year) commits on the github driver to support newer kernels - I'll give it a shot and see where I end up.

Okay - I tried blacklisting rtw88 and indeed 88x2cs was loaded, but you're right - it didn't actually work. wlan0 shows up in iwconfig but not in ifconfig, and none of the wifi operations (scanning, associating with APs, etc) work correctly. Out of curiosity if you happen to have a CM4 handy, what kind of iperf3 speeds do you get with rtw88? I also tried patching dts to increase the frequency - I set the limit to 200MHz, but the actual bus speed ended up only 50MHz and nothing worked at all (SDIO errors right from the get-go). Next will try poking around to see if I can force I/O voltage to 1.8V and maybe that will nudge this into UHS104 speeds? I should also shop around for other boards that have rtl8822cs sdio modules and see if there are any reviews that benchmark wifi performance using rtw88

I mean....I have always been using the antennas - it's other reviewers who were getting slow speeds with the older kernel who were not using antennas. Even with antennas, newer kernel + rtw_88 driver is only initializing to 25MHz SDIO speed (100mbit bus speed), while older kernel + 88x2cs initialized to 200MHz SDIO speed (800mbit bus speed) I still think the issue is the device tree and not the driver - I guess next step is comparing the wifi overlay for the older kernel to the device tree of the newer kernel

Continuing down the route of debugging: I found the geerlingguy review of the CM4 from 2023: https://github.com/geerlingguy/sbc-reviews/issues/11 This referenced a specific image (debian 10 buster) linked from an old wiki page: https://wiki.banana-pi.org/index.php?title=Banana_Pi_BPI-CM4&oldid=13245#Linux I installed that image and was greeted with an ancient 4.9 kernel: ------------- Linux bananapi 4.9.241-BPI-M2S #1 SMP PREEMPT Fri Jun 17 12:23:43 HKT 2022 aarch64 GNU/Linux I enabled overlays="wifi_bt_rtl8822cs" in /boot/firmware/env.txt, restarted, modprobe 88x2cs, configured my wifi info in /etc/network/interfaces.d/wlan0, ifup wlan0 and wifi successfully connected! I then installed and repeated my iperf3 test and this time got the expected 300-400Mbps - yay! Of note on above image kernel 4.9.241 and wifi_bt_rtl8822cs overlay, I get the following results for: cat /sys/kernel/debug/sdio/ios -------------- clock: 200000000 Hz actual clock: 199999997 Hz vdd: 21 (3.3 ~ 3.4 V) bus mode: 2 (push-pull) chip select: 0 (don't care) power mode: 2 (on) bus width: 2 (4 bits) timing spec: 6 (sd uhs SDR104) signal voltage: 1 (1.80 V) driver type: 0 (driver type B) This looks a lot more appropriate - 200MHz, 100MB/sec bus speed gives plenty of bandwidth for the WiFi chipset to do its thing. And turns out WiFi works a lot better when you add antennas.... (that really needs to go into an onboarding guide) So now I know the hardware is fine and whatever is wrong is down to different kernel, different device tree, or different driver. Next step (and would love some help / suggestions here) is to figure out which change caused the problems.

I have a Banana Pi CM4 + CM4 IO Board - literally the same one pictured in https://www.armbian.com/bananapicm4io/ I downloaded and installed Ubuntu 24.04 (Noble) Server Image - "Armbian_24.8.1_Bananapicm4io_noble_current_6.6.47.img" For convenience, I installed the image to an SD card. Armbianmonitor link: https://paste.armbian.com/igahozatow Everything boots and I can use armbian-config to connect to my local WiFi network, but the speeds are really really slow: iperf3 gives me ~40mbps pretty consistently: -------------------- [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 51.1 MBytes 42.9 Mbits/sec 2 sender [ 5] 0.00-10.20 sec 50.1 MBytes 41.2 Mbits/sec receiver I'm connected to a 5GHz network, good connection strength/quality (and I do have antennas plugged into the CM4) -------------------- wlan0 IEEE 802.11 ESSID:"unimportant" Mode:Managed Frequency:5.22 GHz Access Point: 70:A7:41:69:BA:14 Bit Rate=650 Mb/s Tx-Power=20 dBm Retry short limit:7 RTS thr:off Fragment thr:off Encryption key:off Power Management:off Link Quality=68/70 Signal level=-42 dBm Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0 Tx excessive retries:0 Invalid misc:2 Missed beacon:0 Other devices on my network consistently push 300-400mbps iperf3. Also if I install and configure a mini-PCIe network card on the CM4IO board, that also gives me 300-400Mbps. I suspect the SDIO connection itself might be the bottleneck, though I'm new at this and might be barking up the wrong (device) tree here. The RTL8822CS module is connected to mmc2 per dmesg output: ----------------------- [ 7.539802] rtw_8822cs mmc2:0001:1: WOW Firmware version 9.9.4, H2C version 15 It appears mmc2 is running at only 25MHz 4-bit mode: cat /sys/kernel/debug/mmc2/ios ----------------------- clock: 25000000 Hz actual clock: 25000000 Hz vdd: 21 (3.3 ~ 3.4 V) bus mode: 2 (push-pull) chip select: 0 (don't care) power mode: 2 (on) bus width: 2 (4 bits) timing spec: 0 (legacy) signal voltage: 0 (3.30 V) driver type: 0 (driver type B) Of note - I came across this review/comparison (https://bret.dk/banana-pi-cm4-review/) of the BPi CM4 using the same CM4IO board as I'm using, except those iperf3 results make even less sense - 81 Mbps down / 158 Mbps up "bare", 12 Mbps down / 58 Mbps up using antennas. The author suspects that performance is better with "onboard PCB antennas", except I....don't see anything that looks like it would be an onboard PCB antenna - either on the module itself or in the schematic (https://drive.google.com/file/d/1IXXok1P2OLiW3p8tavkbfEPTGTrM3b-R/view?usp=sharing) They were using an older kernel / image (unclear where they got it from) so I wasn't able to replicate their findings, but perhaps there was some regression in the mmc controller setup? I'm not sure if this will help or not, but here are relevant excerpts from device tree on my image for the three mmc controllers: dtc -I fs /sys/firmware/devicetree/base ------------------------------- mmc@ffe05000 { pinctrl-names = "default\0clk-gate"; pinctrl-0 = <0x30>; clock-names = "core\0clkin0\0clkin1"; cap-sd-highspeed; vqmmc-supply = <0x32>; bus-width = <0x04>; resets = <0x05 0x2d>; interrupts = <0x00 0xbe 0x04>; clocks = <0x02 0x22 0x02 0x3d 0x02 0x02>; vmmc-supply = <0x2f>; compatible = "amlogic,meson-axg-mmc"; pinctrl-1 = <0x31>; status = "okay"; disable-wp; reg = <0x00 0xffe05000 0x00 0x800>; phandle = <0x130>; max-frequency = <0x2faf080>; cd-gpios = <0x07 0x2f 0x01>; }; mmc@ffe07000 { pinctrl-names = "default\0clk-gate"; pinctrl-0 = <0x33 0x34 0x35>; clock-names = "core\0clkin0\0clkin1"; vqmmc-supply = <0x1f>; mmc-hs200-1_8v; bus-width = <0x08>; resets = <0x05 0x2e>; interrupts = <0x00 0xbf 0x04>; clocks = <0x02 0x23 0x02 0x3e 0x02 0x02>; vmmc-supply = <0x2f>; mmc-ddr-1_8v; compatible = "amlogic,meson-axg-mmc"; pinctrl-1 = <0x36>; status = "okay"; disable-wp; mmc-pwrseq = <0x37>; reg = <0x00 0xffe07000 0x00 0x800>; phandle = <0x131>; max-frequency = <0xbebc200>; cap-mmc-highspeed; }; mmc@ffe03000 { pinctrl-names = "default\0clk-gate"; #address-cells = <0x01>; amlogic,dram-access-quirk; pinctrl-0 = <0x2c>; clock-names = "core\0clkin0\0clkin1"; vqmmc-supply = <0x2f>; bus-width = <0x04>; non-removable; resets = <0x05 0x2c>; interrupts = <0x00 0xbd 0x04>; clocks = <0x02 0x21 0x02 0x3c 0x02 0x02>; #size-cells = <0x00>; vmmc-supply = <0x2f>; keep-power-in-suspend; compatible = "amlogic,meson-axg-mmc"; pinctrl-1 = <0x2d>; status = "okay"; disable-wp; mmc-pwrseq = <0x2e>; reg = <0x00 0xffe03000 0x00 0x800>; phandle = <0x12e>; sd-uhs-sdr104; max-frequency = <0x2faf080>; wifi@1 { reg = <0x01>; phandle = <0x12f>; }; }; The RTL8822CS module itself should support SDIO 1.1 / 2.0 / 3.0 up to clock frequency of 208MHz, but it seems like for some reason the controller just isn't running at that speed and I'm not sure what exactly to change to fix that - anyone have any suggestions on what I'm doing wrong or what I should check next to debug this?