Nitrax

There are quite some linux kernel issues with Odroid N2 and USB, there are even issues with home assistant OS after kernel upgrades. Im not technical enough to find the solution so i just hope "someone" does somewhere on the interwebs, in the meantime i run older kernels 😏

Tested out a few kernels and 5.15.93 seems to be one of the later that's still working. No USB 3 ports work with my USB stick whith recent kernels (it's not just the OTG port that is broken).

I changed this in the dt and got more than double the speeds from my Samsung U3 SD card: However I have no idea what the process of getting this into the kernel source looks like. --- rk3399-rock-pi-4b.dts 2023-10-04 15:36:05.031221270 +0200 +++ rk3399-rock-pi-4b-mod.dts 2023-10-04 00:34:15.773888431 +0200 @@ -342,6 +342,7 @@ disable-wp; pinctrl-names = "default"; pinctrl-0 = <0x26 0x27 0x28 0x29>; + sd-uhs-sdr104; phandle = <0xce>; };

I also got problems with my HC4 when running some of the latest kernels: 1) LVM would not start correctly on my cached devices (however if i dont mount them at boot i could get them to work, can have something to do with root partition on LVM) 2) I could not get cgroups to work in my k3s cluster 3) reboot command causes a shutdown (however this is not verified it is the kernels fault, could be the SD card according to this forum, but it has worked before) I don't have much more info as i quickly downgraded to 22.08.1 / Kernel 5.19.5 again to get my cluster running, and cant really remember what kernels I tried. But two of the "more later" kernels had these issues. To be fair it was not my meaning to upgrade the kernel as the HC4 have had some kernel issues historically, I had frozen kernel updates but must have accidentally un-frozen them. This is just a F.Y.I my fileserver node is running fine now and I'm not in a need to update to debian 12.

No problems here and I reboot pretty frequently, now running 6.1.11. Using a 2Tb 3.5 inch HDD and a 256mb SSD. Could it be unrelated to system upgrades and just be dodgy hardware that is likely to break at boot when there is a power surge? Anyway good to know about the power regulator. Edit: Now found the other threads that makes it seem like a kernel issue. Anyway I'm using pretty old drives on my HC4, and it is also a quite old HC4 that I bought just after it was announced.

Ok thanks! But Ill also try to set a static one, for some reason i often get stuff that looks chinese even though Im in sweden.

Very often when i do apt update or apt dist-upgrade some of the armbian mirrors acts up and pretty much denies connections (it isnt the generic debian mirrors as other distros are working fine). I assume they are just overloaded and theres not much to do about that. But it has caused one of my system to be out of date for a few weeks I assume because the unattended upgrade script doesnt retry. How do I get this to work? for example some times i have to retry up to 10 times before I can get apt to work (when I try manually). The apt configuration in etc is a bit of magic to me so Id appreciate if someone can guide me to get unattended upgrades to retry connections.

Yes this is only for testing distros and nuking the data is assumed. I should have been more clear about this!

Ok I saw the debug output now. It seems you have some processes dying because they go OOM., possibly from memory leaks, most likely causing issues with the system in general.

I just have to share my method to test distros headless with my OdroidN2. The N2 boots eMMC before SD card, so what I did was to flash the SD card with an armbian distro and get it up and running. the SD card always shows up on mmcblk0 and the eMMC on mmcblk1. So it boots from SD when it can't find a bootloader on the eMMC. I can flash any image easily with dd if=image of=/dev/mmcblk1 and just do a reboot, and it will boot the eMMC. Whenever I want to test another image, i can just do dd if=/dev/zero of=/dev/mmcblk1 bs=1M count=1 to make the eMMC unbootable, causing it to start on the SD image on the next reboot. A very nice way to test stuff out remotely. I only know it works on the N2, but the concept should be appliable to other SBCs (maybe USB/SD boot..?). I'm sure it's nothing new but the "aha" moment for me was the blanking of the bootloader (doing an unbounded dd if=/dev/zero of=/dev/mmcblk1 caused my device to hang not being able to reboot).

Sorry for late reply. I had a hiatus with my cluster(s). I really need the RAM, so I can't increase the size, and I'm using an eMMC chip for storage so there's no issues with performance. But I'm sure you are correct that most users benefit from log2ram. I will also check that I disable log2ram the correct way. Thank you. Regarding the hardware: I'm practically running k3s (from rancher) on everything I have (with 2+ GB RAM) because it makes life so much simpler. Main ARM cluster is 3 rpi4's and the Odroid N2+ as master. An Odroid HC4 with SSD disks is providing NFS storage for all clusters (through nfs-client-provisioner). The HC4 is also running a single node k3s where I run storage related services such as samba, webdav, docker registry etc. Then I have a Tinkerboard that runs k3s with a few services mostly to test 32-bit arm stuff. I also have a x86 cluster with 40gb RAM, for all the stuff that can't run on ARM or needs too much memory.

The only thing I've had issues with on armbian, and that I often have issues with, is the default log2ram. As soon as you start doing anything that actually logs stuff, like access logs, or container logs and so on, the log space is not enough, even if allocating 500 MB RAM. And when /var/log fills up there are going to be crashes... in all kinds of ways. I have an ansible script that disables this now, but sometimes when doing dist-upgrade, log2ram settings come back (and also zram swap, which is also bad because kubernetes does not like swap, but thats not a disaster at least). So I simply propose to not make log2ram default, unless it can have some way to "spill over" to disk. But what about wear and tear? Well if you are logging enough to wear out the SD card, you are logging more than can fit in log2ram anyway (?). (Fact is, log2ram took down my entire kubernetes cluster, because when /var/log was full on one node, containers started to crash, causing more logs to be written on the other nodes (by a factor of 10-100) causing these to also crash within a few hours timeframe.)

Is there any distro out there that has 2D hardware accelerated graphics so you can get a decent desktop experience out of the N2? I've tried armbian and the distro that comes from Odroid, but all of them are sluggish. Maybe I have missed to select some options? I've "just" installed it and tried som different.. window managers? It's a bit confusing. I'm only going to use it for python and golang development in VS Code, maybe some arduino too, but it's unusable with non accelerated graphics. Chromium or Firefox needs to be usable too. I have a feeling there is some kind of 1 2 3 recipe I've missed like install armbian, select this window manager, install this patch.. but I cant figure it out. Im not trying to criticize the armbian project! I'm using armbian very successfully for server operations on differnt SBCs. But I figured if anyone knows they'll be hanging out here...

And after trying around 20 times: # apt update Hit:1 http://deb.debian.org/debian stable InRelease Hit:2 http://deb.debian.org/debian stable-updates InRelease Hit:3 http://ports.ubuntu.com focal InRelease Hit:4 http://deb.debian.org/debian-security stable/updates InRelease Get:5 http://ports.ubuntu.com focal-security InRelease [109 kB] Hit:6 https://download.docker.com/linux/debian buster InRelease Hit:7 https://repos.influxdata.com/debian focal InRelease Get:8 http://ports.ubuntu.com focal-updates InRelease [114 kB] Get:9 http://ports.ubuntu.com focal-backports InRelease [101 kB] Hit:11 https://packages.gitlab.com/runner/gitlab-runner/ubuntu focal InRelease Get:10 https://us.mirrors.fossho.st/armbian/apt focal InRelease [18.2 kB] Get:12 https://us.mirrors.fossho.st/armbian/apt focal/main armhf Packages [646 kB] Fetched 988 kB in 4s (243 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done 25 packages can be upgraded. Run 'apt list --upgradable' to see them. And now its working every time after something like 2 days of failing.

# sudo apt-get -o Acquire::ForceIPv4=true -o Acquire::CompressionTypes::Order::=gz -o Acquire::http::No-Cache=true -o Acquire::BrokenProxy=true -o Acquire::http::Pipeline-Depth=0 update Hit:1 http://deb.debian.org/debian stable InRelease Hit:2 http://deb.debian.org/debian stable-updates InRelease Hit:3 http://ports.ubuntu.com focal InRelease Hit:4 http://deb.debian.org/debian-security stable/updates InRelease Hit:5 http://ports.ubuntu.com focal-security InRelease Hit:6 https://download.docker.com/linux/debian buster InRelease Hit:7 https://repos.influxdata.com/debian focal InRelease Hit:8 http://ports.ubuntu.com focal-updates InRelease Hit:9 http://ports.ubuntu.com focal-backports InRelease Get:10 https://armbian.tnahosting.net/apt focal InRelease [18.2 kB] Hit:11 https://packages.gitlab.com/runner/gitlab-runner/ubuntu focal InRelease Get:12 https://armbian.tnahosting.net/apt focal/main armhf Packages [645 kB] Ign:12 https://armbian.tnahosting.net/apt focal/main armhf Packages Get:12 https://armbian.systemonachip.net/apt focal/main armhf Packages [645 kB] Err:12 https://armbian.systemonachip.net/apt focal/main armhf Packages File has unexpected size (646150 != 645411). Mirror sync in progress? [IP: 51.75.2.52 443] Hashes of expected file: - Filesize:645411 [weak] - SHA512:b1c4ae525ec721387a734673069af21268482d7324822871727d7b54f49a807cac2ce64b739b23cedbe309d3cd203af9ba297dee6042e4a9071cae5f986071c8 - SHA256:0ba674c754337dffa93851eb62e0f4841a4a29841ea708dd6bfde44ca2eedf3f - SHA1:703b61507243cdddd410855b86e45d07955c53c9 [weak] - MD5Sum:e0745d84180e2a4deaa33c5ad469c24a [weak] Release file created at: Sat, 12 Dec 2020 15:43:37 +0000 Reading package lists... Done E: Failed to fetch https://armbian.systemonachip.net/apt/dists/focal/main/binary-armhf/Packages.gz File has unexpected size (646150 != 645411). Mirror sync in progress? [IP: 51.75.2.52 443] Hashes of expected file: - Filesize:645411 [weak] - SHA512:b1c4ae525ec721387a734673069af21268482d7324822871727d7b54f49a807cac2ce64b739b23cedbe309d3cd203af9ba297dee6042e4a9071cae5f986071c8 - SHA256:0ba674c754337dffa93851eb62e0f4841a4a29841ea708dd6bfde44ca2eedf3f - SHA1:703b61507243cdddd410855b86e45d07955c53c9 [weak] - MD5Sum:e0745d84180e2a4deaa33c5ad469c24a [weak] Release file created at: Sat, 12 Dec 2020 15:43:37 +0000 E: Some index files failed to download. They have been ignored, or old ones used instead. Hit:1 http://deb.debian.org/debian stable InRelease Hit:2 http://deb.debian.org/debian stable-updates InRelease Hit:3 http://deb.debian.org/debian-security stable/updates InRelease Hit:4 http://ports.ubuntu.com focal InRelease Hit:5 http://ports.ubuntu.com focal-security InRelease Hit:6 https://download.docker.com/linux/debian buster InRelease Hit:7 http://ports.ubuntu.com focal-updates InRelease Hit:8 https://repos.influxdata.com/debian focal InRelease Hit:9 http://ports.ubuntu.com focal-backports InRelease Get:10 https://armbian.hosthatch.com/apt focal InRelease [18.2 kB] Hit:11 https://packages.gitlab.com/runner/gitlab-runner/ubuntu focal InRelease Get:12 https://armbian.hosthatch.com/apt focal/main armhf Packages [645 kB] Ign:12 https://armbian.hosthatch.com/apt focal/main armhf Packages Get:12 https://mirrors.dotsrc.org/armbian-apt focal/main armhf Packages [645 kB] Err:12 https://mirrors.dotsrc.org/armbian-apt focal/main armhf Packages File has unexpected size (646150 != 645411). Mirror sync in progress? [IP: 130.225.254.116 443] Hashes of expected file: - Filesize:645411 [weak] - SHA512:b1c4ae525ec721387a734673069af21268482d7324822871727d7b54f49a807cac2ce64b739b23cedbe309d3cd203af9ba297dee6042e4a9071cae5f986071c8 - SHA256:0ba674c754337dffa93851eb62e0f4841a4a29841ea708dd6bfde44ca2eedf3f - SHA1:703b61507243cdddd410855b86e45d07955c53c9 [weak] - MD5Sum:e0745d84180e2a4deaa33c5ad469c24a [weak] Release file created at: Sat, 12 Dec 2020 15:43:37 +0000 Fetched 18.2 kB in 3s (6,143 B/s) Reading package lists... Done E: Failed to fetch https://mirrors.dotsrc.org/armbian-apt/dists/focal/main/binary-armhf/Packages.gz File has unexpected size (646150 != 645411). Mirror sync in progress? [IP: 130.225.254.116 443] Hashes of expected file: - Filesize:645411 [weak] - SHA512:b1c4ae525ec721387a734673069af21268482d7324822871727d7b54f49a807cac2ce64b739b23cedbe309d3cd203af9ba297dee6042e4a9071cae5f986071c8 - SHA256:0ba674c754337dffa93851eb62e0f4841a4a29841ea708dd6bfde44ca2eedf3f - SHA1:703b61507243cdddd410855b86e45d07955c53c9 [weak] - MD5Sum:e0745d84180e2a4deaa33c5ad469c24a [weak] Release file created at: Sat, 12 Dec 2020 15:43:37 +0000 E: Some index files failed to download. They have been ignored, or old ones used instead.

I am getting this problem every time now on apt update on my armbian tinkerboard: Hit:1 http://deb.debian.org/debian stable InRelease Hit:2 http://deb.debian.org/debian stable-updates InRelease Hit:3 http://deb.debian.org/debian-security stable/updates InRelease Hit:4 http://ports.ubuntu.com focal InRelease Hit:5 http://ports.ubuntu.com focal-security InRelease Hit:6 https://download.docker.com/linux/debian buster InRelease Hit:7 https://repos.influxdata.com/debian focal InRelease Hit:8 http://ports.ubuntu.com focal-updates InRelease Hit:9 http://ports.ubuntu.com focal-backports InRelease Get:10 https://minio.k-space.ee/armbian/apt focal InRelease [18.2 kB] Hit:11 https://packages.gitlab.com/runner/gitlab-runner/ubuntu focal InRelease Get:12 https://minio.k-space.ee/armbian/apt focal/main armhf Packages [645 kB] Ign:12 https://minio.k-space.ee/armbian/apt focal/main armhf Packages Get:12 https://us.mirrors.fossho.st/armbian/apt focal/main armhf Packages [645 k B] Err:12 https://us.mirrors.fossho.st/armbian/apt focal/main armhf Packages File has unexpected size (646150 != 645411). Mirror sync in progress? [IP: 23. 237.182.68 443] Hashes of expected file: - Filesize:645411 [weak] - SHA512:b1c4ae525ec721387a734673069af21268482d7324822871727d7b54f49a807cac2c e64b739b23cedbe309d3cd203af9ba297dee6042e4a9071cae5f986071c8 - SHA256:0ba674c754337dffa93851eb62e0f4841a4a29841ea708dd6bfde44ca2eedf3f - SHA1:703b61507243cdddd410855b86e45d07955c53c9 [weak] - MD5Sum:e0745d84180e2a4deaa33c5ad469c24a [weak] Release file created at: Sat, 12 Dec 2020 15:43:37 +0000 Fetched 18.2 kB in 4s (4,942 B/s) Reading package lists... Done E: Failed to fetch https://us.mirrors.fossho.st/armbian/apt/dists/focal/main/binary-armhf/Packages.gz File has unexpected size (646150 != 645411). Mirror sync in progress? [IP: 23.237.182.68 443] Hashes of expected file: - Filesize:645411 [weak] - SHA512:b1c4ae525ec721387a734673069af21268482d7324822871727d7b54f49a807cac2ce64b739b23cedbe309d3cd203af9ba297dee6042e4a9071cae5f986071c8 - SHA256:0ba674c754337dffa93851eb62e0f4841a4a29841ea708dd6bfde44ca2eedf3f - SHA1:703b61507243cdddd410855b86e45d07955c53c9 [weak] - MD5Sum:e0745d84180e2a4deaa33c5ad469c24a [weak] Release file created at: Sat, 12 Dec 2020 15:43:37 +0000 E: Some index files failed to download. They have been ignored, or old ones used instead.

Just reporting that there is some tx-offload bug in the ASUS Tinker Board Rockchip. I think I read about this issue regarding other rockchip/platforms when checksum offload bugs on large packets. When I run ethtool -K eth0 tx off I stop having issues with NFS over TCP The last part of diagnosis is when I forced a NFS crash by enabling tx offloading.