5kft

Unfortunately I'm not an expert in RCU stalls (https://www.kernel.org/doc/Documentation/RCU/stallwarn.txt). Do you have other processes running on the device that could be impacting the system? E.g., what does "top" show while you try to use networking?

It might be worth trying to reinstall the original posted -rc3 version again (esp. the dtb and image debs), just to see if that helps? (Just force an install via dpkg -i *.deb)

This problem sounds like this issue: https://github.com/armbian/build/issues/2315

@going - are you asking why the kernel packages don't include the specific kernel version in them, like as is the case with a native Debian kernel build? I'm not sure why this isn't the case with Armbian, but I suppose that support could be added - although I'm not sure of the dependencies elsewhere in the build tools (or if there are any).

Wonderful - this is great to hear! It is a new patch, so keep an eye on the pull requests for the kernels regarding this change, I'm sure it will get pulled into v5.10 soon (and hopefully backported to v5.9 as well).

I'm not an expert in this area whatsoever, but this problem looks like an upstream kernel issue, and perhaps is related to this fallback deadlock fix? https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git/commit/?id=a2715fbdc6fc387e85211df917a4778761ec693d The posted fix patch is only applicable to kernel v5.10; just in case you want to give it a try, I've uploaded a ToT test build of v5.10-rc3 with this kernel patch applied: https://drive.google.com/file/d/1Eg_cn3bEYsRFZ3vrP9Mc0fxJjPREczcW/view. Again, I'm not sure if this will fix the problem, but there are a number of changes in cryptodev here compared to kernel v5.9, so it might be worth a try.

I think that more research is needed here, as the problem is non-obvious. Unfortunately I can't reproduce any issues on my OrangePi R1 with either 5.8.5 or 5.9.5. @Igor this doesn't appear to be DVFS related; the DTs are correct and the board is correctly maxed at 1GHz (but it can support 1.3GHz via the overclocking overlay). I can successfully run all of "openssl speed -multi 4", throttling is fine, etc. Using iperf3 testing, I can't reproduce any issues on my board with the Ethernet interfaces either; both eth0 and enxc0742bfffa39 work properly for me... I'm seeing 94Mbps on both eth0 and enxc0742bfffa39 (for testing, I purged network-manager and statically set the IPs in /etc/network/interfaces.d/) - here is the iperf3 server output for the two runs from the R1: Of course this is not to say that there isn't a problem, but with this simple/quick iperf3 testing I can't reproduce anything on my board. @Ford_Prefect I think it would be interesting to see what your serial console shows. @guidol would it be possible to get serial console output for your situation as well?

A lot has changed in the thermal drivers since 4.19, it's possible that the "better temperatures" you are seeing in the older kernel are actually incorrect. E.g., see https://github.com/megous/linux/commits/orange-pi-5.3/drivers/thermal/sun8i_thermal.c (in particular https://github.com/megous/linux/commit/eac37f0ec6f33e8f3edb8dbdb9d11b4cb8e218e5#diff-e5ac7e093b36d92344ed52a2747602d5e46d366ca34b4abb46cc005f92ee679f). There are numerous posts in the forums about this and there have been a number of changes in the upstream drivers in this area over the past couple of years... New (post-4.19) thermal driver history here: https://lore.kernel.org/linux-arm-kernel/20191226202607.GA9524@Red/T/#mfffc26980cb13605429156c3833464fe3cbdb661

I actually tried both - Debian works fine (as you noted), but Focal crashes just like I was seeing yesterday:

Glad it is working for you, but very curious indeed! Edit: I tried on another NEO2, and indeed 4.9.5 works fine . Perhaps my first test was odd because I just upgraded the kernel on a much older platform rootfs. But I can confirm the platform-provided v4.9.5 is working under v5.10-rc2 on my "cleaner" NEO2. I'm also glad to see that the latest v4.13.2 also works!

Hi @guidol, I'm actually seeing similar issues with Debian (!). The default Debian Buster v4.9.5 Samba server works fine with kernel v5.9.5, but for me with kernel v5.10-rc2 it consistently crashes with "INTERNAL ERROR" faults when a client would access it. Samba has had issues like this in the past where a particular version would start crashing on newer kernels, so I installed the latest version of Samba (v4.13.2) and it works fine with kernel v5.10 - no crashes, and everything works well. It looks like there have been a number of changes made for Samba in the new kernel (e.g., see https://wiki.samba.org/index.php/LinuxCIFSKernel#Changes_by_release), so perhaps the older server versions provided by the distributions are simply too old and incompatible for the new v5.10 kernel. Here are some quick screen captures showing Samba working on my NEO2; I first make a directory "testing", and then copy some files to it, and then I map and access the NEO2 share via Windows: I copied a number of files across the share as well and it worked fine. I don't use Ubuntu, so I'm not sure what version of Samba is provided with Focal, but perhaps you're encountering the same issue? You might try installing a newer Samba version and see how it works for you. Note that I did this all as a test and simply built Samba v4.13.2 from source on the NEO2 (I used https://download.samba.org/pub/samba/samba-latest.tar.gz to download the source). I hope this helps!

Yes, added appropriate pre-merge tags for both -current and -dev. (Interestingly, at least at present, the delta between 5.9.x and 5.10.x is tiny.)

Agreed...I'll go ahead and make the changes.

@Igor, kernel 5.8 has gone EOL with 5.8.18...should we look at moving 5.9 to -current? It's easy enough to move sunxi-dev to 5.10 (and it works quite well). Interested in your thoughts.

I'm seeing the same problem, exactly as @guidol described. I cleared the cache directory and then cleared the output directory, then building 5.9 for nanopi-r1 I ended up with (correct) .debs for sunxi, but (incorrect?) .debs for sunxi64 as well Note: I'm using docker to build; I performed a full build again after removing all "linux-*" files from "/var/lib/docker/volumes/armbian-cache/_data/sources/linux-mainline/" first, and then the output .debs were correct. So this is consistent with the workarounds above - just need to clear the right "cache" directory.

If you're feeling adventurous and like living life on the edge, I have provided a buildable 5.10 tree here for sunxi: git clone -b v5.10-dev-test https://github.com/5kft/build Build as normal using "./compile.sh" using "BRANCH=dev" from this git branch and it'll build 5.10.0-rc1 targets. Both sunxi and sunxi64 builds are enabled. Note that I only tested on a few H3 and H5 boards (both Nano Pi and Orange Pi), but it seems to work pretty well. But no guarantees, support, etc. - this is just a side project atm

Understood. A lot has changed since 4.19...it could be any number of things. I checked the passive polling rate and 5.x actually polls more often than 4.x, so that's interesting... A couple of questions: In terms of your older 4.19 kernel, what was the maximum clock rate of the CPU? If you don't overclock to 1.3GHz with 5.x, does it work okay (i.e., doesn't overheat)?

It is if we want to switch -current to 5.9 (i.e., deprecate 5.8)

With just a few local test hacks, I was able to bring up kernel 5.10-rc1 on sunxi this morning (based on the new megous branch): root@air's password: _ _ ____ _ _ _ | \ | | _ \(_) / \ (_)_ __ | \| | |_) | | / _ \ | | '__| | |\ | __/| | / ___ \| | | |_| \_|_| |_| /_/ \_\_|_| Welcome to Armbian 20.08.14 Buster with Linux 5.10.0-rc1-sunxi System load: 2% Up time: 11 min Memory usage: 14% of 491M IP: 172.24.18.151 CPU temp: 34°C Usage of /: 24% of 7.2G Last login: Tue Oct 27 07:20:02 2020 from 172.24.18.20 root@air:~# cat /proc/version Linux version 5.10.0-rc1-sunxi (root@355045fc2473) (arm-none-linux-gnueabihf-gcc (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 9.2.1 20191025, GNU ld (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 2.33.1.20191209) #trunk SMP Tue Oct 27 14:13:23 UTC 2020 root@air:~# root@air:~# cpufreq-info -c 0 cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 2 3 CPUs which need to have their frequency coordinated by software: 0 1 2 3 maximum transition latency: 5.44 ms. hardware limits: 480 MHz - 1.20 GHz available frequency steps: 480 MHz, 648 MHz, 816 MHz, 960 MHz, 1.01 GHz, 1.10 GHz, 1.20 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 480 MHz and 1.20 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 816 MHz (asserted by call to hardware). cpufreq stats: 480 MHz:97.72%, 648 MHz:0.22%, 816 MHz:0.20%, 960 MHz:0.23%, 1.01 GHz:0.15%, 1.10 GHz:0.19%, 1.20 GHz:1.29% (679) root@air:~# I tested this on a spare NanoPi NEO Air board I had; I haven't tried an arm64 build yet. dmesg is clean; cpufreq works, overclocking works, wireless works, etc. Our 5.9 kernel patchset applied almost completely without error (there are a few other changes needed such as the builddeb patches and fbcon reversion patch we did). In any case I just wanted to let people know. Given that 5.10 is confirmed to be the new LTS and 5.9 is the new primary -stable, I'm wondering if 5.8's days are numbered (like what happened to 5.7). I'm happy to put some work into bringing 5.10 into build if there is interest.

Thanks @linuxjosef, these videos are very helpful! From what I can see, it is working just fine in the default configuration, and overheating in this instance when overclocking to the maximum speed possible. Clearly you have insufficient cooling present on this board, especially when overclocking (e.g., too small a heatsink, no fan). I've had the same problem on my Orange Pi Zero Plus2 H5s, because the heatsink is so small - unfortunately the board isn't designed well for supporting a large heatsink. However, on my Nano Pis for example I can overclock to the maximum 1.3GHz without an issue, because the heatsink options are significantly larger, e.g., see https://www.friendlyarm.com/image/catalog/description/neo-heat-sink_en_03.jpg and https://www.friendlyarm.com/image/catalog/description/NEO2newcase_en_02.jpg - with these heatsink configurations the CPUs barely get hot. To avoid this problem I would suggest that you use a larger heatsink regardless, and/or add a cooling fan. When using a larger heatsink, if overclocking, try overclocking to 1.2GHz instead of 1.3GHz, or simply don't overclock at all (since nothing is guaranteed when overclocking anyway). I took a look at tweaking the maps to downclock more aggressively at the higher temperatures (testing on an OPi Zero Plus2), but it became quite obvious that there is only so much that can be done with purely passive cooling and a small heatsink. Even clocking down significantly at over 90C would still spike with the "openssl" test and hit critical shutdown. Then, as a test I simply sat a bigger heatsink on top of the existing small heatsink, and then I couldn't get it to overheat - the openssl test successfully ran to completion when overclocked to 1.3GHz: I hope this helps!

Yes, I agree - especially if this patch from @Icenowy makes it into the mainline: https://groups.google.com/g/linux-sunxi/c/QzBM7nciPLo. We should wait for the dust to settle then can remove our revert patch and adapt the other DTs appropriately. Thanks!

@Igor I'd like to get your thoughts regarding this...: The Realtek kernel change that caused this issue is indeed correct and good (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/?id=bbc4d71d63549bcd003a430de18a72a742d8c91e), as it fixes the earlier invalid implementation in the kernel. The failure problems with the RTL8812E occur with this change because "phy-mode" in the DT now matters and must be correct, whereas previously it did not. By changing the "phy-mode" to "rgmii-id" correctly for sunxi boards that have the TXDLY and RXDLY pull-ups on the PHY, the RTL8812E is properly configured and works. However this means that all of the relevant DTs need to be updated to correctly reflect their actual hardware configuration. I started making these changes in the patches in sunxi-dev, confirming the existence of the delay configuration pull-ups with the board schematics to determine the proper phy-mode to set, and removing my reversion patch. However any boards that are missed will have Ethernet failures until the DTs are properly patched. Should we just leave the reversion patch in instead (long-term)?

If the kernel is trapping/crashing for some reason then this would likely be shown on the console, which could help provide a clue as to what might be going on...

For what it's worth, I'm not seeing any issues on a 256MB OrangePi R1 running 5.8.14-current (I only have a 512MB Pi Zero). It works without issue - e.g., root@orangepi-r1:~# uname -a Linux orangepi-r1 5.8.14-sunxi #trunk SMP Sun Oct 11 14:48:02 UTC 2020 armv7l GNU/Linux root@orangepi-r1:~# cpufreq-info -c 1 cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 1: driver: cpufreq-dt CPUs which run at the same hardware frequency: 0 1 2 3 CPUs which need to have their frequency coordinated by software: 0 1 2 3 maximum transition latency: 5.44 ms. hardware limits: 480 MHz - 1.01 GHz available frequency steps: 480 MHz, 648 MHz, 816 MHz, 960 MHz, 1.01 GHz available cpufreq governors: conservative, userspace, powersave, ondemand, performance, schedutil current policy: frequency should be within 480 MHz and 1.01 GHz. The governor "ondemand" may decide which speed to use within this range. current CPU frequency is 480 MHz (asserted by call to hardware). cpufreq stats: 480 MHz:86.78%, 648 MHz:0.30%, 816 MHz:0.18%, 960 MHz:0.06%, 1.01 GHz:12.69% (193) root@orangepi-r1:~# free total used free shared buff/cache available Mem: 243992 78052 79360 1936 86580 155792 Swap: 121992 0 121992 root@orangepi-r1:~# echo 0 > /sys/devices/system/cpu/cpu2/online root@orangepi-r1:~# apt-get update Hit:1 http://deb.debian.org/debian buster InRelease Hit:2 http://deb.debian.org/debian buster-updates InRelease Hit:3 http://deb.debian.org/debian buster-backports InRelease Hit:4 http://security.debian.org buster/updates InRelease Hit:5 https://armbian.systemonachip.net/apt buster InRelease Reading package lists... Done root@orangepi-r1:~# wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.8.14.tar.xz --2020-10-11 15:52:46-- https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.8.14.tar.xz Resolving cdn.kernel.org (cdn.kernel.org)... 2a04:4e42:600::432, 2a04:4e42::432, 2a04:4e42:200::432, ... Connecting to cdn.kernel.org (cdn.kernel.org)|2a04:4e42:600::432|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 114506716 (109M) [application/x-xz] Saving to: ‘linux-5.8.14.tar.xz’ linux-5.8.14.tar.xz 100%[===================>] 109.20M 3.19MB/s in 30s 2020-10-11 15:53:17 (3.68 MB/s) - ‘linux-5.8.14.tar.xz’ saved [114506716/114506716] root@orangepi-r1:~# md5sum linux-5.8.14.tar.xz b4ed25c9a13f2ac90405b74da26e31d6 linux-5.8.14.tar.xz root@orangepi-r1:~# cat /sys/class/thermal/thermal_zone0/temp 60297 root@orangepi-r1:~# @maracuja Might it be possible to hook a serial console to the board, and then capture the output when your board freezes/crashes? (It might also be interesting to try using Wi-Fi instead of Ethernet to eliminate that vector as a possibility?)

Yes, do a pull and try again - I just removed the offending patch as it was merged upstream in -rc6 (See https://github.com/armbian/build/commit/9637d3d0f94203a8ea18495d61090dbe202aeeaf)