Matthias

Hallo, I've been using Armbian on the NanoPi M4 V2 for a while and now I'm thinking about switchen from the legacy kernel to the current kernel. Unfortunately that's not so easy to do when using armbian-config. You could think you can just select the current-kernel in the list but you won't find it there. The reason is simple: the legacy kernel belongs to the rk3399-family and the current kernel is rockchip64. Which is basically the same, but from an organisational point of view it is not (please forgive me if I do not show an adequate technical precision here, the details and the plan to clean that up are explained here). What I would be interested in is now: Is there a proper way to switch the kernel manually? As I said, I have been using the system for a while now, so setting it up from scratch with the right kernel is something I would like to avoid. I had a look into the source of armbian-config so I know that manipulating the LINUXFAMILY in /etc/armbian-release will display the current-kernel in the list. But I don't want to imagine what kind of side-effects will hit me when starting the installation process that way. Cheers, Matthias

I played around with setting the governor to "conservative" or "ondemand". Result: "conservative" results in a stable system that allows copying of several GBs of data, "ondemand" results in an unstable system that fails after copying some GBs of data and corrupts its filesystem I also did a scripted load test that copied 100GB (fast: data source was tmpfs) of data to the hard drives with governor set to "conservative". Result: No reported file system errors and a MD5-sum-check of every file copied to the disks passed successfully. If anybody is interested and likes to repeat that test on his configuration I attached the script. It's currently made for BTRFS. If you are using a different file system, just remove the part with the scrubbing. load_test.sh

And repeated the copying with kernel 5.4.11. The CPU frequencies were limited in the sysfs (govenor "conservative", scaling_max_freq) to 1.42/1.8GHz. I copied 30GB of data to the hard drive (this time BTRFS) and got the first errors (file system repors CRC-errors) while calculating the MD5-sums of the copied files. My conclusion: No indicators that the CPU-frequency spoils the data, but something seems to be odd with kernel 5.4.x and NanoPi M4V2. Or do you think the frequency needs to be reduced the hard way in the kernel? @piter75 Just noticed I only used the governor "conservative" for the A72, the A53 was set to "on demand". Frequencies were correct. So I need to recall my conclusion. I repeated the test, this time all CPUs were set to "on demand". And voilà: All data was written correctly. Strange. I don't have the time right now to go back and repeat the test again with the "on demand" governor to see that the problems appear again, but so far my conclusion is: The NanoPi M4V2 only runs reliable if the CPU frequencies are limited to 1.42/1.8GHz AND the governor is NOT set to "on demand" ("conservate" seems to work).

Just did some copying of files using kernel 4.4.210: I copied around about 15GB of random data and about the same amount of other data (videos) on a ZFS-volume. Also copied about 5GB of data from SD to the ZFS-volume. MD5-checks indicate that there has not been any data corrupted. Cpufreq shows the maximum allowed frequency of the CPUs is 1.42/1.8GHz. So the combination of kernel 4.4.x and "low" CPU frequency seems to do the trick. I'll try again with a 5.4.x-kernel and throttled CPUs.

Don't know exactly, I think it was 4.4.192.

I didn't observe any crashes because of high load but what I encountered when compiling code were messages on the console indicating (indirectly) that there is something odd with the CPU (sorry, I don't remember the text). I tried to fix that using more powerful power supply, but that did not help. Not even the 12V supply via the SATA-hat. What did help was setting the governor for the CPU frequency to "conservative". Since I did that, I never got any strange messages anymore. So my conclusion was, that the board might have problems handling spikes or large changes in power consumption. But of course, that's not a prove against crashes...

Don't worry. I remember darkly that when I tested it one month ago the legacy kernel was good at receiving and bad at sending while the current kernel was cood at sending and bad at receiving. See here:

Interesting, I also had problems copying files using the 5.4 kernel on my NanoPi M4V2. I compiled it myself and copies files from the sd-card to and external hard drive connected via the SATA-hat. I used file systems on the hard drive that were able to detect corrupted data on the drive (ZFS and BTRFS) and every time I did a scrub after copying the files, failed CRC-checks within the data were reported, some of them even at the same position on two redundant drives (mirror). I'll try again with the legacy kernel (4.x). Would be greate if it works in this configuration. @piter75: Did you apply your fixes for ethernet also to the legacy kernel? Then it would be much easier to get the data onto the device in the first place.

That would explain some things. For example: During scrubbing a mirrored BTRFS reported broken CRCs of the same block of data on both disks. I observed that more than once within 40gb of data. Chances of this happening by accident are of course very low. But if the data gets corrupted at the source (RAM) this could easily happen. Configuring the timing of the RAM is way beyond my knowledge, so I'll stop here and wait for any improvements. As always: If there is something you would like to get tested: Don't hesitate to contact me.

I need to recall my thesis about ZFS interfering with Samba and causing trouble. I cannot reproduce it anymore. For testing with NFS I installed the NFS client for Windows 10 wich seemed to interfere a lot with SMB. I don't know why, but after uninstalling it, everything was fine and I was able to download and upload files in large amounts. Of course having the offloading disabled. This is true for transferring data from and to the SD card. When using a hard drive connected to the SATA hat I get stopping network connections, broken file systems (ZFS and BTRFS two-disk-mirrors that show uncorrectable errors without any forced reboot) and kernel panics. But that seems to be a different story (broken hardware? temperature?) and does not have a direct connection to Samba or the network connection. Long story short: I consider the Samba problem to be solved and need to continue examining my hardware setup.

Ok @piter75, looks like things are a little more complicated: First of all I must apologise: My "fresh install" had at least one customisation I did not mention: The kernel had support for ZFS (via zfs-dkms). When using this installation Piter's use of ethtool does not bring any improvement. Then I remembered something: ZFS brings has support for SMB and NFS. Don't know why they mixed that but seems to be a relict from Solaris. So just to make sure I reinstalled the system without ZFS-support. The following observations were made using the "ZFS-less" system connected to a Windows 10 PC via gigabit ethernet: Copying a lot of data via SMB to and from the share hosted on the NanoPi without "sudo ethtool -K eth0 tx off rx off": After some gigabytes the transfer speed drops to zero and the NanoPi gets unresponsible. It looked like the green LED changed it's flashing-pattern so it was flashing faster, but I might be wrong. Doing the same with "sudo ethtool -K eth0 tx off rx off": I successfully copied about 20GB to and from the NanoPi. No problems detected. So I think this change brings improvement. And combining Samba and ZFS does not seem to be a good idea, at least not for the versions I used (kernel 5.4.10, Samba 4.9.5, zfs-dkms 8.2.3). For whatever reason. Shall we add the call of ethtool to the build process of Armbian images for the NanoPi M4V2?

That's good news, thank you for that hint, Piter. And good to see that my issue is reproducible on more than one board. I also read about that a couple of weeks ago. But as that was the time where we all struggled with the stability of the networking, it didn't bring any effect and I dropped that idea. I can give it another try this evening on the current state of the networking. Hopefully it has the effect I'm looking for.

Checked NFS: Works fine, I was not able to observe any drops of transfer speed like mentioned above.

I can check NFS, that's a good idea. Then I get a better impressing how "strongly" related the issue is to Samba. If Samba works on one board family and not on another, the only difference relevant to Samba I can think of is the kernel. Or did I miss something?

Edit: Issue is solved, see here. It's an application problem but according to an extensive Google research this does not seem to be a common problem of Samba, so it might be an integration problem with the environment it is running in. The setup: Armbian 19.11.6 (Debian Buster) running the current kernel on a NanoPi M4V2, offering network shares via Samba. Clients are Windows 10 and a BananaPi running Armbian (Debian Buster 4.19.y kernel) as well. The observation: When downloading (large) files from the Samba share to Windows 10, the download speed very often suddenly drops down to zero and stays on that rate (forever). This can be best observed on files of several GB size as it usually takes some seconds until the problem appears. There seems to be a correlation with the download speed: Using ethernet it happens more often than when using slower WiFi. I also increased the log level of Samba but I could not detect anything special the moment the rate drops down (In the Samba logs. There are no significant lines in /var/log/messages or /var/log/syslog). The transfer just stops. The same behavior can be observed when mounting the share on the BananaPi and then copy large files. My first guess was a generally unstable network connection as there have been committed some fixes in this area to Armbian recently. But when stressing the network via iperf3 in both directions I could not detect any variations in the transfer speed. In both directions. When writing data to the Samba shares I do not observe any problems. I was able to upload about 30GB without any significant drop in speed. I also took the working smb.conf from my BananaPi, which serves files flawlessly via Samba, adapted it and ran tests. But I observed the same behavior. So I assume there is a problem the way Samba accesses the files or uses the network stack in the Armbian environment. But as this is only speculation and I did not find any further indicators for the source of the problem I now hope that somebody else stumbled over the same thing and maybe already has a solution for it. If somebody likes to reproduce it: Take a fresh install, install Samba add a share to a directly on the SD card in smb.conf (no further changes in the config file). Then copy a file of about 3GB to the share. Then try to download the file again. You will be impressed by the transfer speed (as long as the file is cached in RAM on the NanoPi M4V2) of more than 100MB/s but then the transfer suddenly stops.

@piter75 I just tested the values for rx_delay you provided with kernel 5.4.6-rockchip64. 0x16 seems to be the best value for me, 0x14 offers a similar quality. You can find the details in the table below. rx_delay Observation 0x0E No connection to device via SSH possible. Problems receiving data? 0x14 Receives data at ~ 930mbit/s, but on very rare occasions drops down to ~200mbit/s but then normalizes again 0x0C No connection to device via SSH possible. Problems receiving data? 0x16 Receives data at ~ 930mbit/s, seems to be more stable than 0x14

Just one thing I would like to mention here for the sake of documentation: The modification of the dependencies of the kernel headers should get into existing installations sooner or later by regular updates of packages (as soon as PR 1697 is merged). But the modification of /etc/environment won't, as far as I understand the build process of Armbian. That modification needs to be made manually on existing installations in order to get DKMS working. Sorry about all those tiny increments of information day by day, but I was a complete newbee to the build system of Armbian when I joined this quest and I'm still learning a lot on every step I take...

Sure, I can do that. Thanks for the manual, looks quite straight forward. I was afraid I need to much "heavier" steps of the build process to check the different options. Just give me a day or two.

I can confirm your patched legacy-kernel can receive and transmit at about 930mbps. Good job, @piter75! Unfortunately I cannot confirm high rates for receiving data using kernel 5.4.6 on my hardware. I used this image to test the current kernel: Armbian_19.11.4_Nanopim4v2_buster_current_5.4.6.img (downloaded from server, no local compilation)

I finally complete and tested my fixes. Concerning the dependencies of the linux headers I added the following for kernels <=4.14: make, gcc, libc6-dev, libssl-dev. And the following for newer kernels: make, gcc, libc6-dev, bison, flex, libssl-dev The results of my tests on a NanoPi M4V2: Legacy kernel (4.14-rockchip): Works out perfectly. After installing the kernel headers, zfs-dkms can be installed flawlessly. The modules are compiled. Current kernel (mainline 5.4.6): ZFS (0.7.12) cannot be installed. I assume this is caused by the problem mentioned above (https://github.com/vpsfreecz/zfs/commit/36c110f9943f3abe2ac59ffa7e76b48e8dbfc1b6): This version of ZFS is not compatible with kernels >=5.2. I also tried the backport of ZFS (0.8.2) which should be compatible with the kernel but then next problem pops up (https://github.com/zfsonlinux/zfs/issues/8545). So I decided ZFS is a bad reference for confirming the fix of DKMS as there seem to be too many issues with ZFS itself. But I was able to install other DKMS-packages successfully: dm-writeboost-dkms lime-forensics-dkms My conclusion: My changes seem to fix DKMS in Armbian but it seems to be too optimistic to expect DKMS-packages compile successfully against the most current kernels. The following branch contains my changes: https://github.com/mmriech/armbian-build/tree/fix_dkms. I'm going to create a pull request for that.

Update on the current state of my activities. All changes mentioned below are not yet on Armbian's master: I modified /etc/environment and removed the export of the variable ARCH. This fixes the problem about that variable mentioned at the begin of the thread. I added dependencies to the linux headers compiled during build of Armbian images. That way packages like "make" and "flex" are automatically installed when installing the header package. I was not able to test all supported kernels (and adjust the dependecies to all of them) yet, so this work is incomplete. So this part is still in progress. Also: If the headers are installed during the build of the image the part "Compiling headers - please wait ..." still fails as make cannot be found in the chrooted environment. Update: The previous sentence was a misobservation. Thanks to QEMU the installation can proceed within the chrooted environment. Is just very slow if you choose to install the headers during the build of the image. I also found out that the version of ZFS for Linux that is part of Debian Buster is not compatible with kernels >5.2. (see https://github.com/vpsfreecz/zfs/commit/36c110f9943f3abe2ac59ffa7e76b48e8dbfc1b6). So it is possible that I won't get it running with current kernels after all. But it makes me believe that DKMS is not generally broken and I can get it running with the modifications 1 and 2. I'll keep you posted...

Some further observations of mine in case it helps identifying the problems using ethernet: After noticing that the throughput using the native ethernet connection (eth0) of the M4V2 is low, I checked it using iperf3. I tested the legacy kernel, the current kernel and the dev kernel. The results are quite interesting: Legacy kernel: Can receive data via TCP from anoter host with high speed (~940MBit/s) but the other way around is quite slow (<1MBit/s). Current/dev kernel: Can send data via TCP to another host with high speed (~950MBit/s) but the other way around is quite slow (~3MBit/s). Further information: If I set the link speed to 10MBit half duplex there seems to be almost no packet loss that slows down the connection. Using a USB3-ethernet adapter speed is good in both directions. I did not care about WiFi. It's not connected. There is a SATA hat attached to my NanoPi M4V2. Only power supply is used, no hdds are connected. Of course I only have one device at hand, so I cannot guarantee that the behavior is representative. Generally: If there is somebody working on further bringup of NanoPi M4V2 and needs somebody for testing, I'm happy to assist.

Ok, now I think I understand the background why dkms-packages cannot be installed. The example from my previous post is still valid, but now I can give further explanations: apt update # Make sure all packages required to build zfs-dpkg are installed. # Easiest way to do it, is by installing the package itself. # As a side effect, this also provides all tools required to successfully compile the headers of the kernel. # Afterwards the dkms-packages can (and need to) be deinstalled again. apt install zfs-dkms apt remove spl-dkms zfs-dkms # The installed packages dkms has a problem: In the postinst script the local variable ARCH is in conflict with the environment variable ARCH. Two solutions are avaiable: # 1) Manually patch file /usr/lib/dkms/common.postinst ($ARCH -> $ARCH_) # 2) Install patched deb-package available as attachment (dpkg -i dkms) # Install linux-headers. This is a (soft) requirement of the package dkms-zfs. (not enforced as dependency) # The step "Compiling headers" takes quite some time, indicating that work is actually done. apt install linux-headers-rockchip64 # Now the package zfs-dkms can be installed without problems apt install zfs-dkms So I can nail down the problem to two individual problems: The Debian package "dkms" is incompatible with Armbian (at least with version Armbian_5.98_Rockpi-4b_Debian_buster_default_4.4.192_minimal) The packages linux-headers is patched by Armbian to have no dependencies and run some scripts (make) after installation. The success of the execution of the scripts is not checked, so problems are hidden from the user. Due to the lack of dependencies the linux-headers cannot be installed successfully on the minimal Armbian without manually installing required packages (even "make" is not installed by default). So far for the observations. The interpretation of those is more complex and beyond my knowledge of Armbian and Debian. Those are my personal thoughts: I expect fixing the package "dkms" is outside the scope of Armbian. So this would need to be clarified with the Debian project. I can think of two theoretical solutions how that could be handled within Armbian: Create patched version of official debian package "dpkg" by the builder of Armbian and provide it for download via apt. Is this even intended within Armbian? Make Armbian work without the environment variable "ARCH". Potentially goes extremely deep into the system. Add dependencies to the linux-headers so the postinst can run successfully. Standard debian defines dependencies to packages like gcc and kbuild. Why doesn't Armbian? Another example: When running "make scripts" after the end of the sequence above it still fails because "libssl-dev" is not installed. Igor, what's your point of view on that? dkms_2.6.1-4_all.deb

I have some news. After playing around a bit randomly to understand (a tiny part of) the internals of dpkg and dkms I did the following, knowing that we have some problems with the environment variable ARCH (see comments above): I checked the file /usr/lib/dkms/common.postinst. I replaced all occurences of ARCH by ARCH_. The idea behind that: In general ARCH is a global variable, in the script is used like a local variable: It it set to the architecture the script receives as parameter $4. Then I ran "apt install zfs-dkms". It succeeded. (just to make it clear: I ran it before the edit and it failed. Of course I deinstalled zfs-dkms and spl-dkms afterwards) Could that count as a fix or do you think it's just a workaround for some problem hidden much deeper in the system? I installed the same dkms-modules some while ago on a Debian or Ubuntu x86 system and it worked like a charm right from the beginning. So I'm a bit it doublt that we are facing a general problem of DKMS here... Edit 1: Just realised that this is more or less what Petr's script (above) does. Edit 2: The patch mentioned above only(?) works when running the following sequence of commands on a fresh installation of Armbian (important part: install linux headers AFTER the first attempt to build zfs-dkms and try again): apt update apt install zfs-dkms # yes, do this without any kernel headers installed. # patch /usr/lib/dkms/common.postinst ($ARCH -> $ARCH_) apt remove spl-dkms zfs-dkms # allow zfs-dkms to be installed again apt install linux-headers-rockchip64 apt install zfs-dkms

Thanks for the quick answer. I've absolutely no clue about the builder of Armbian and have never built Armbian by myself. But it sounds like an interesting challenge and the patches on your branch look quite understandable (as understandable as patches are...). Together with the information in this thread at least I have an idea of what needs to be done. So let's hope I get some enlightment to figure out the "how". Don't expect quick results.