kristsm Posted March 5, 2020 Posted March 5, 2020 (edited) Hi! I didn't find much info in the forum on how to correctly apply RT patch to the Armbian (Orange Pi Zero Plus H5 board). Here is how I did: Built Armbian as follows: 1) I downloaded patch-5.4.22-rt13.patch from https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/5.4/ and placed into ~/build/userpatches/kernel/sunxi-current ( I saw from the logs that this is the folder for the builder to search for user patches in my case) 2) Started ./compile.sh --> FULL OS image for flashing --> Show a kernel configuration menu before compilation --> orangepizeroplus (H5) as target board --> current (recommended , which is 5.4.23 as of today) --> bionic (18.04 LTS) --> standard image with console interface. 3) Build log shows: "[ o.k. ] * [c] patch-5.4.22-rt13.patch" So it seems the patch is included in the compilation. 4) In the kernel configuration screen I amended settings for low-latency kernel ( I chose the third item in the list which seems is full preemptive mode) Armbian build finishes without errors. The board starts fine, it shows Armbian 5.4.23-rt13 as the kernel version. uname -a shows that the kernel is in fact in the SMP PREEMPT mode. Now the problem is that any cyclictest with any configuration shows much worse AVG latency times compared to the system with standard Armbian 5.4.23 kernel. In most cases AVG latency times exceed 1000us , therefore its far-far away from the times I see in the forum for other systems using even the H3 chipset (sub 50us AVG latency tims is not that uncommon). What I did wrong, or what necessary/mandatory steps have I missed for having sub 50us AVG latency system? Thanks for any help in advance! Krists Edited March 5, 2020 by kristsm misstyped
kristsm Posted March 5, 2020 Author Posted March 5, 2020 Sorry - 1000us+ was for MAX latency, not average. Krists
MX_Master Posted March 5, 2020 Posted March 5, 2020 There's no PREEMPT_RT option for the arm64 inside patch-5.4.22-rt13.patch
kristsm Posted March 6, 2020 Author Posted March 6, 2020 MX_Master - thanks for the tip! How should I determine the last RT kernel patch version containing PREEMPT_RT Option for arm64? I am not an advanced programmer, but opening patch-5.4.22-rt I see there are configs for ARM64.... P.S. from packages.debian.org I also see there is a package with a name linux-image-5.4.0-4-rt-arm64-unsigned
MX_Master Posted March 6, 2020 Posted March 6, 2020 See the a/arch/arm64/Kconfig section and compare it with other popular platforms lsdiff patch-5.4.22-rt13.patch | grep x86 filterdiff -i a/arch/x86/Kconfig patch-5.4.22-rt13.patch lsdiff patch-5.4.22-rt13.patch | grep arm64 filterdiff -i a/arch/arm64/Kconfig patch-5.4.22-rt13.patch
kristsm Posted March 6, 2020 Author Posted March 6, 2020 And what exactly should I look for? Both (x86 & arm64) have: 1. /include/asm/preempt.h 2. select ARCH_SUPPORTS_RT 3. select HAVE_PREEMPT_LAZY Additionally , ARM64 have: - select ARCH_INLINE_READ_LOCK if !PREEMPT - select ARCH_INLINE_READ_LOCK_BH if !PREEMPT - select ARCH_INLINE_READ_LOCK_IRQ if !PREEMPT - select ARCH_INLINE_READ_LOCK_IRQSAVE if !PREEMPT - select ARCH_INLINE_READ_UNLOCK if !PREEMPT - select ARCH_INLINE_READ_UNLOCK_BH if !PREEMPT - select ARCH_INLINE_READ_UNLOCK_IRQ if !PREEMPT - select ARCH_INLINE_READ_UNLOCK_IRQRESTORE if !PREEMPT - select ARCH_INLINE_WRITE_LOCK if !PREEMPT - select ARCH_INLINE_WRITE_LOCK_BH if !PREEMPT - select ARCH_INLINE_WRITE_LOCK_IRQ if !PREEMPT - select ARCH_INLINE_WRITE_LOCK_IRQSAVE if !PREEMPT - select ARCH_INLINE_WRITE_UNLOCK if !PREEMPT - select ARCH_INLINE_WRITE_UNLOCK_BH if !PREEMPT - select ARCH_INLINE_WRITE_UNLOCK_IRQ if !PREEMPT - select ARCH_INLINE_WRITE_UNLOCK_IRQRESTORE if !PREEMPT - select ARCH_INLINE_SPIN_TRYLOCK if !PREEMPT - select ARCH_INLINE_SPIN_TRYLOCK_BH if !PREEMPT - select ARCH_INLINE_SPIN_LOCK if !PREEMPT - select ARCH_INLINE_SPIN_LOCK_BH if !PREEMPT - select ARCH_INLINE_SPIN_LOCK_IRQ if !PREEMPT - select ARCH_INLINE_SPIN_LOCK_IRQSAVE if !PREEMPT - select ARCH_INLINE_SPIN_UNLOCK if !PREEMPT - select ARCH_INLINE_SPIN_UNLOCK_BH if !PREEMPT - select ARCH_INLINE_SPIN_UNLOCK_IRQ if !PREEMPT - select ARCH_INLINE_SPIN_UNLOCK_IRQRESTORE if !PREEMPT + select ARCH_INLINE_READ_LOCK if !PREEMPTION + select ARCH_INLINE_READ_LOCK_BH if !PREEMPTION + select ARCH_INLINE_READ_LOCK_IRQ if !PREEMPTION + select ARCH_INLINE_READ_LOCK_IRQSAVE if !PREEMPTION + select ARCH_INLINE_READ_UNLOCK if !PREEMPTION + select ARCH_INLINE_READ_UNLOCK_BH if !PREEMPTION + select ARCH_INLINE_READ_UNLOCK_IRQ if !PREEMPTION + select ARCH_INLINE_READ_UNLOCK_IRQRESTORE if !PREEMPTION + select ARCH_INLINE_WRITE_LOCK if !PREEMPTION + select ARCH_INLINE_WRITE_LOCK_BH if !PREEMPTION + select ARCH_INLINE_WRITE_LOCK_IRQ if !PREEMPTION + select ARCH_INLINE_WRITE_LOCK_IRQSAVE if !PREEMPTION + select ARCH_INLINE_WRITE_UNLOCK if !PREEMPTION + select ARCH_INLINE_WRITE_UNLOCK_BH if !PREEMPTION + select ARCH_INLINE_WRITE_UNLOCK_IRQ if !PREEMPTION + select ARCH_INLINE_WRITE_UNLOCK_IRQRESTORE if !PREEMPTION + select ARCH_INLINE_SPIN_TRYLOCK if !PREEMPTION + select ARCH_INLINE_SPIN_TRYLOCK_BH if !PREEMPTION + select ARCH_INLINE_SPIN_LOCK if !PREEMPTION + select ARCH_INLINE_SPIN_LOCK_BH if !PREEMPTION + select ARCH_INLINE_SPIN_LOCK_IRQ if !PREEMPTION + select ARCH_INLINE_SPIN_LOCK_IRQSAVE if !PREEMPTION + select ARCH_INLINE_SPIN_UNLOCK if !PREEMPTION + select ARCH_INLINE_SPIN_UNLOCK_BH if !PREEMPTION + select ARCH_INLINE_SPIN_UNLOCK_IRQ if !PREEMPTION + select ARCH_INLINE_SPIN_UNLOCK_IRQRESTORE if !PREEMPTION So what am I missing?
sfx2000 Posted March 7, 2020 Posted March 7, 2020 On 3/5/2020 at 3:21 AM, kristsm said: I didn't find much info in the forum on how to correctly apply RT patch to the Armbian (Orange Pi Zero Plus H5 board). Curious as to why one wants to apply the patches for preempt-rt in the first place? Do you have a reason/application that would benefit? Most times, one can see an overall decrease in application performance, and to take advantage of preempt-rt, the application must call pthreads in a very specific way - which means that many times, the app must also be rebuilt. My reason for asking - if preempt-rt was so awesome, it would already be in the upstream linux kernel mainline and not require the patch. Avoiding politics around preempt-rt - many of the benefits here can be done without the patches, by selecting an appropriate scheduler - schedutil looks fairly interesting at the moment. I work on networking oriented applications that have hard deadlines, on MIPS, PowerPC, ARM (both 32bit ARMv7a and 64-bit), and never had a need to apply the preempt-rt patches. By networking oriented applications - I'm saying 20mSec framing for SIP-RTP (VoIP apps), along with 3GPP/UMTS signalling - if that ain't close to real-time, I don't know what is. 1
kristsm Posted March 9, 2020 Author Posted March 9, 2020 On 3/7/2020 at 6:03 AM, sfx2000 said: Curious as to why one wants to apply the patches for preempt-rt in the first place? Do you have a reason/application that would benefit? Most times, one can see an overall decrease in application performance, and to take advantage of preempt-rt, the application must call pthreads in a very specific way - which means that many times, the app must also be rebuilt. My reason for asking - if preempt-rt was so awesome, it would already be in the upstream linux kernel mainline and not require the patch. Avoiding politics around preempt-rt - many of the benefits here can be done without the patches, by selecting an appropriate scheduler - schedutil looks fairly interesting at the moment. I work on networking oriented applications that have hard deadlines, on MIPS, PowerPC, ARM (both 32bit ARMv7a and 64-bit), and never had a need to apply the preempt-rt patches. By networking oriented applications - I'm saying 20mSec framing for SIP-RTP (VoIP apps), along with 3GPP/UMTS signalling - if that ain't close to real-time, I don't know what is. Curious as to why you waste your time if you don't want to help?
le51 Posted March 9, 2020 Posted March 9, 2020 Hi there, This topic interest me a lot. I'm using arm sbc for sound & music applications where low latency is important. sfx2000's post reminds me other discussions on other forums (mostly computer "assisted" music oriented) where some thoughts/experiences are related to poor application performance regarding the low positive impact of a RT kernel. From my own experience (x86, arm SBC's and RPI), RT kernel or not, good or bad audio interface in USB or I2S, depending on the load (cpu, memory, audio stream), I always have crack, glitches and xruns. Without RT patch, what I get: - With a H5 board with 2Gb memory (NanoPiK1plus) I can't go under 20ms of jack audio server latency without having a to unstable config but, as is, it is still "playable". - With a NanoPiM4 and a HD USB audio interface, I have 5ms of latency, a lot of power for apps and a NvMe interface for a fast drive. That's great So, I really ask myself for going further in kernel "hacking" with the RT patch.
kristsm Posted March 10, 2020 Author Posted March 10, 2020 Tried to build Armbian Bionic today with sunxi-current ( I see the compiler have included patch 5.4.23-24 as of today, which seems the kernel should be @5.4.24?). I moved 5.4.24 rt15 patch in userpatches/kernel/sunxi-current. In kernel configuration I checked expert menu , which allowed me to choose fully preemptible kernel. The compiler goes until CC [M] kernel/kheaders.o, then it throws [error] ERROR in function compile_kernel [ compilation.sh: 378] and terminates the process. Any tips on how to move further?
Igor Posted March 10, 2020 Posted March 10, 2020 22 minutes ago, kristsm said: Any tips on how to move further? You will find tips in debug logs: output/debug/compilation.log
kristsm Posted March 10, 2020 Author Posted March 10, 2020 9 hours ago, Igor said: You will find tips in debug logs: output/debug/compilation.log Thanks! Ok, log file shows following errors: fs/aufs/i_op.c: In function ‘au_pin_hdir_set_owner’: fs/aufs/i_op.c:636:45: error: ‘struct rw_semaphore’ has no member named ‘owner’ atomic_long_set(&p->hdir->hi_inode->i_rwsem.owner, (long)task); ^ make[2]: *** [fs/aufs/i_op.o] Error 1 make[1]: *** [fs/aufs] Error 2 make: *** [fs] Error 2 make: *** Waiting for unfinished jobs.... scripts/Makefile.build:265: recipe for target 'fs/aufs/i_op.o' scripts/Makefile.build:500: recipe for target 'fs/aufs' failed Makefile:1692: recipe for target 'fs' failed
Igor Posted March 10, 2020 Posted March 10, 2020 54 minutes ago, kristsm said: Ok, log file shows following errors: fs/aufs/i_op.c: In function ‘au_pin_hdir_set_owner’: fs/aufs/i_op.c:636:45: error: ‘struct rw_semaphore’ has no member named ‘owner’ atomic_long_set(&p->hdir->hi_inode->i_rwsem.owner, (long)task); ^ make[2]: *** [fs/aufs/i_op.o] Error 1 make[1]: *** [fs/aufs] Error 2 make: *** [fs] Error 2 make: *** Waiting for unfinished jobs.... scripts/Makefile.build:265: recipe for target 'fs/aufs/i_op.o' scripts/Makefile.build:500: recipe for target 'fs/aufs' failed Makefile:1692: recipe for target 'fs' failed Now its clear what to do. Build with AUFS="no" since it looks like it doesn't want to operate with RT or fix errors in the code if you need AUFS.
sfx2000 Posted March 11, 2020 Posted March 11, 2020 On 3/9/2020 at 5:47 AM, kristsm said: Curious as to why you waste your time if you don't want to help? My time is valuable - so let's spend it wisely... Do you have a real-world use-case where there is value or not?
kristsm Posted March 11, 2020 Author Posted March 11, 2020 14 hours ago, Igor said: Now its clear what to do. Build with AUFS="no" since it looks like it doesn't want to operate with RT or fix errors in the code if you need AUFS. Thanks Igor! After AUFS disabled, the compiler threw at me another error regarding NFS, disabled that too and was greeted with completed build. Flashed the image in 64gb sd-card and after boot uname -a greeted me with the message: Linux orangepizeroplus 5.4.24-rt13-sunxi64 #trunk SMP PREEMPT_RT Wed Mar 11 07:28:42 EET 2020 aarch64 aarch64 aarch64 GNU/Linux cyclic test results with all cores running in performance mode (no mods for ISOLCPUS): sudo cyclictest -a -t -n -p80 # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 0.96 0.85 0.57 1/165 3526 T: 0 ( 3518) P:80 I:1000 C: 161625 Min: 7 Act: 10 Avg: 10 Max: 49 T: 1 ( 3519) P:80 I:1500 C: 107750 Min: 7 Act: 14 Avg: 11 Max: 49 T: 2 ( 3520) P:80 I:2000 C: 80812 Min: 7 Act: 10 Avg: 10 Max: 35 T: 3 ( 3521) P:80 I:2500 C: 64650 Min: 8 Act: 10 Avg: 10 Max: 25
Recommended Posts