Advise on forking armbian/build for PREEMPT_RT support (Rock64 as test board)

[awesomebytes]

Hello community,

 

I have just been successful building my own image of Ubuntu focal + preempt_rt (kernel 5.15.76-rt53) for my Rock64 board for the first time (a bit of additional background at the end of my post). Exciting! I had a few hiccups along the way, and I need to figure out a few more things. So far I'm happy and impressed with the project.

 

However, given I want to share my work (and the resulting image(s)) I would like to ask for some pointers on what is the best way to go in regards of forking armbian/build and the changes I found myself doing.

 

• I have added my kernel compilation flags to config/kernel/linux-rockchip64-current.config but I feel like maybe I should create a new file, something like 'linux-rockchip64-current-rt.config' but I do not know if the compile.sh will find it (or how to make it find it) if I do so. What would be the best way to go about it?
• I added the PREEMPT_RT patches to patch/kernel/rockchip64-current , but again, there should probably be a folder for RT maybe? (I wonder how to make compile.sh find it).
• I created userpatches/lib.config with the content 'KERNELBRANCH='tag:v5.15.76'' to use the latest PREEMPT_RT compatible kernel, ideally this could be scripted to use the latest closest to 'current', which also would avoid needing to modify the patch/kernel/BOARD_NAME folder.
• I also needed to remove the box64 package cause it would stop building my image (GPG key outdated?) is this package necessary for a minimal installation? I don't think I found I needed it so far. I left a comment about it in this github issue: https://github.com/armbian/build/issues/3968#issuecomment-1310390875
• I needed to add a couple of additional cmdline flags, I edited by hand (after booting the board) the resulting /boot/armbianEnv.txt with 'extraargs=nohz_full=3 isolcpus=3' and that worked, but I'd like to create the image with that already done. What would be the best place to do that? userpatches/customize-image.sh ?
• If I place anything in userpatches/customize-image.sh am I able to run the build just from there?

 

Thank you very much for any pointers you can give me about best practices to go about it, anything I did wrong, or templates to follow.

 

My first goal is to provide a reproducible build of ubuntu focal + preempt_rt for the rock64 board (which I'm very close!), but I'd be happy to extend the work to be able to apply the PREEMPT_RT patches to any board saving people a bit of the guess work I needed to do (at least for as long the patches are not part of the kernel by default!).

 

Additional background: I need to use Ubuntu as my plan is to use ROS 2 on it, and the officially supported distro is Ubuntu (and focal happens to be the most useful for my use-case). I am aiming, in conjunction with some members of the ROS real-time working group, to ease the entry barrier of learning to make real-time apps for robotics. I am using a Rock64 board because it's similar to a Raspberry Pi 4, as I am unable to source a Raspi4 and I believe it is the same for everyone.

 

Bonus: a quick test (running 10min) of cyclictest with stress on the system provided the following plot on latency (74us max, 15us min, about 20ish us average):

 

 

Edited November 11, 2022 by awesomebytes

[rodw]

I know its been a couple of months since your post but I am interested in getting PREEMPT_RT running with Debian Bookworm on an Odroid N2.

My interest is because PREEMPT_RT is a prerequisite for linuxcnc to run cnc machines. This requires strict timing to ensure the linuxcnc servo-thread fires on time every millisecond.

Linuxcnc has recently been added to the Debian repositories and is available in Bookworm and Sid.

 

I have recently built a Debian linux-image-rt .deb for the 6.1 PREEMPT_RT kernel which solved a few latency problems we were facing.

What has become a bigger issue is network latency on the debian kernels from kernels 5.10-6.1 (bulleye and above). This is impacting on our RT ethernet devices as the thread timing is violated casusing the device and possibly a big machine to lock up!

 

Today I think I was able to build the 6.2 RT kernel for the N2+  but until my N2+ hardware arrives of somebody tests my image stored on Google Drive, I won't know.

The approach I took was to copy the RT patch as follows:

wget https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/6.2/patch-6.2-rc3-rt1.patch.gz
gunzip patch-6.2-rc3-rt1.patch.gz

then copy the unzipped patch to  build/userpatches/kernel/meson64-edge

This seemed to work as if I told the build system I wanted to change the kernel flags, it asked me to choose the premptible kernal method.  

So I decided to see if I could automate this more.

so I copied build/config/kernel/linux-meson64-edge.config to userpatches and edited it to add

 

CONFIG_PREEMPT_RT_FULL=y

This file could be edited further to change any other flags 

Time will tell If I have got it right!

 

I must say while it was a hard slog to work this out, I am impressed with the Armbian build system.

I think all of this could be scripted.

PS If you know anybody in the Debian RT team, please tell them their RT patch is broken!

[Igor]

1 hour ago, rodw said:

I am impressed with the Armbian build system.

Thank you! BTW. We are just about to switch to a completely reworked version of it:

https://www.armbian.com/newsflash/code-freeze-and-moving-to-new-framework/

 

This should also help dealing with RT patches. If we had someone around to deal with RT patch discrepancy, we could also generate RT kernels. 

[going]

28 минут назад, Igor сказал:

If we had someone around to deal with RT patch discrepancy, we could also generate RT kernels. 

Oh my God! Igor, I have been in the Armbian team for a long time.

[Igor]

Just now, going said:

Oh my God! Igor, I have been in the Armbian team for a long time.

Oh, you want to take care of this? 

[going]

I have been building this project for a long time.

And at the moment, I don't see an alternative for the ARM architecture for real-time core operation with short response times.

@Igor The Xenomai4 project is already working well for sunxi today.

 

03.02.2023 в 11:22, rodw сказал:

wget https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/6.2/patch-6.2-rc3-rt1.patch.gz

 

03.02.2023 в 11:22, rodw сказал:

please tell them their RT patch is broken!

 

lsdiff patch-6.2-rc3-rt1.patch

These are some basic changes. I don't see the necessary changes specific to rockchip, meson or sunxi files here.
In order for this patch to work, you need to add a lot of additional changes yourself.

 

03.02.2023 в 11:22, rodw сказал:

Linuxcnc has recently been added to the Debian repositories and is available in Bookworm and Sid.

And this is for working only in user space.
In any case, you will have to reassemble the entire chain for industrial production. kernel -> lib & modules -> linuxcnc

[going]

11.11.2022 в 10:05, awesomebytes сказал:

Bonus: a quick test (running 10min) of cyclictest with stress on the system provided the following plot on latency (74us max, 15us min, about 20ish us average):

embedded/xenomai

 

[going]

2 часа назад, Igor сказал:

Oh, you want to take care of this?

That's all I'd like to take care of.
That is, to support the Xenomai4 assembly within the framework of the Armbian project.

And this requires a mechanism for assembling and reassembling packages, such as is customary in basic distributions.

Both will serve to popularize the assembly system.

[going]

Xenomai or rt-preempt today is a simple application of patches to the kernel source code.
This process will not cause difficulties. But then we will need to build\rebuild several additional libraries

and kernel modules and the application that uses these libraries loads and unloads modules.

Obviously, it is necessary to prescribe dependencies correctly in this entire chain.

[rodw]

Thanks for the welcome. The forum imposing a 1 post a day limit on me has not been condusive to our conversation! Hopefully that is in the past now!

You've opened up a whole can of worms now! Maybe I chose the wrong platform with my N2!

Going, I read up on all your links about Xenomai.  Very interesting.

Linuxcnc supports xenomai if the environment is detected at build time

but I don't think V 4 has been used by anyone.

 

One thing that might be helpful would be a reliable way to detect if xenomai was installed as its been pointed out the current method is rather crude.

https://github.com/LinuxCNC/linuxcnc/blob/2e75b091a23e3312b9e57a88941714701f3709b8/src/rtapi/uspace_common.h#L361

I'm all ears if you could propose a better method that also detected Xenomai 4

 

Also, is network latency improved under xenomai? We have been experiencing network latency with realtek NICs on kernels 5.10 to 6.2 on Debian x86. Compiling the pristine upstream sources seems to help but its not a 100% cure.

 

I don't think our RT needs are terribly onerous. We just need to ensure that latency is low enough that our servo-thread which usually fires every 1 millisecond does not run late.

 

Regarding preempt_rt, we've never done anything other than to enable a fully preemptible kernel before building it.

 

Anyway, thanks for the advice so far, if you have time to answer some of my questions it would be appreciated.

 

 

[going]

@rodw

Why are you asking these questions about the real-time operation of the kernel here on this site?
Here we can discuss the Armbian assembly system itself.
Here we can discuss working on the devices of images that have been provided for download.

 

If you use the Armbian build system and want it to be able to assemble a number of interdependent

packages in a complex, then ask. I can direct you right now and I can provide my development branch for this.

 

But it is best to discuss the source code of the kernel and linuxcnc on the linuxcnc forum.

 

 

[rodw]

Sorry, I got off track when you mentioned Xenomai as I had not considered that path,

I'll wait until my armbian hardware arrives before asking further questions

[TRS-80]

13 hours ago, rodw said:

The forum imposing a 1 post a day limit on me has not been condusive to our conversation! Hopefully that is in the past now!

 

Please don't take it personal.  It only takes a few people pestering, looking for (free, immediate, personalized) support to get on the nerves of some people around here.  There are very few of us, against a sea of them, so that is how we decided to deal with it, to stem the tide.

 

Further, some threads move extremely slow.  Maybe not this one, but I have seen many times where someone replies months, maybe even years later.  Different people come along, move some issue forward by inches, and on it goes.  Such is the way of progress some times.

 

Welcome to the forums.

[rpardini]

User on Discord reported problems using the -rt patches with the main branch on Amlogic edge.

1) RT patches are specific to certain linux versions, and the fact most of our families use say 6.2.y branch (and we can't easily override KERNELBRANCH= due to the tangle between board/boardfamily/family_common source order conundra). Would need to either hack at the meson64_common file to change it to to "tag:6.2" to use the 6.2.0 release, or use a hook in a config file to to the sam. Suggestion: add a FORCE_KERNELBRANCH param to work around this more easily for users without requiring a config file change.

2) RT patches add in a ./localversion-rt file, which is picked up by Kconfig and included in CONFIG_LOCALVERSION, which changes the paths of _everything_ from what Armbian expects. Say from "6.2.14-meson64" to "6.2.14-rt3-meson64". The kernel builds, but packaging gags. It's possible to remove the patch that adds that file so the version matches, but Armbian definitely should be able to handle this more gracefully, either by automatically removing "localversion-*" files, or by actually honoring them.