PREEMPT-RT PATCH for Allwinner H5

[kristsm]

Hi!

 

I didn't find much info in the forum on how to correctly apply RT patch to the Armbian (Orange Pi Zero Plus H5 board). 

Here is how I did:

Built Armbian as follows:

1) I downloaded patch-5.4.22-rt13.patch from https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/5.4/ and placed into  ~/build/userpatches/kernel/sunxi-current ( I saw from the logs that this is the folder for the builder to search for user patches in my case)

2) Started ./compile.sh  --> FULL OS image for flashing --> Show a kernel configuration menu before  compilation --> orangepizeroplus (H5) as target board --> current (recommended , which is 5.4.23 as of today) --> bionic (18.04 LTS) --> standard image with console interface.

3) Build log shows:  "[ o.k. ] * [c] patch-5.4.22-rt13.patch"

So it seems the patch is included in the compilation.

4) In the kernel configuration screen I amended settings for low-latency kernel ( I chose the third item in the list which seems is full preemptive mode)

 

Armbian build finishes without errors.

The board starts fine, it shows Armbian 5.4.23-rt13 as the kernel version.  

uname -a shows that the kernel is in fact in the SMP PREEMPT mode.

 

Now the problem is that any cyclictest with any configuration shows much worse AVG latency times compared to the system with standard Armbian 5.4.23 kernel. In most cases AVG latency times exceed 1000us , therefore its far-far away from the times I see in the forum for other systems using even the H3 chipset (sub 50us AVG latency tims is not that uncommon).

 

What I did wrong, or what necessary/mandatory steps have I missed for having sub 50us AVG latency system?

 

Thanks for any help in advance!

Krists

 

 

 

Edited March 5, 2020 by kristsm
misstyped

[kristsm]

Sorry - 1000us+ was for MAX latency, not average.

 

Krists

 

 

[MX_Master]

There's no PREEMPT_RT option for the arm64 inside patch-5.4.22-rt13.patch

[kristsm]

MX_Master - thanks for the tip!

 

How should I determine the last RT kernel patch version containing PREEMPT_RT Option for arm64? I am not an advanced programmer, but opening patch-5.4.22-rt I see there are configs for ARM64....

P.S. from packages.debian.org I also see there is a package with a name linux-image-5.4.0-4-rt-arm64-unsigned

 

 

 

[MX_Master]

See the a/arch/arm64/Kconfig section and compare it with other popular platforms 

lsdiff patch-5.4.22-rt13.patch | grep x86
filterdiff -i a/arch/x86/Kconfig patch-5.4.22-rt13.patch

lsdiff patch-5.4.22-rt13.patch | grep arm64
filterdiff -i a/arch/arm64/Kconfig patch-5.4.22-rt13.patch

 

[kristsm]

And what exactly should I look for? 

Both (x86 & arm64) have:

1. /include/asm/preempt.h

2. select ARCH_SUPPORTS_RT

3. select HAVE_PREEMPT_LAZY

 

Additionally , ARM64 have:

-       select ARCH_INLINE_READ_LOCK if !PREEMPT
-       select ARCH_INLINE_READ_LOCK_BH if !PREEMPT
-       select ARCH_INLINE_READ_LOCK_IRQ if !PREEMPT
-       select ARCH_INLINE_READ_LOCK_IRQSAVE if !PREEMPT
-       select ARCH_INLINE_READ_UNLOCK if !PREEMPT
-       select ARCH_INLINE_READ_UNLOCK_BH if !PREEMPT
-       select ARCH_INLINE_READ_UNLOCK_IRQ if !PREEMPT
-       select ARCH_INLINE_READ_UNLOCK_IRQRESTORE if !PREEMPT
-       select ARCH_INLINE_WRITE_LOCK if !PREEMPT
-       select ARCH_INLINE_WRITE_LOCK_BH if !PREEMPT
-       select ARCH_INLINE_WRITE_LOCK_IRQ if !PREEMPT
-       select ARCH_INLINE_WRITE_LOCK_IRQSAVE if !PREEMPT
-       select ARCH_INLINE_WRITE_UNLOCK if !PREEMPT
-       select ARCH_INLINE_WRITE_UNLOCK_BH if !PREEMPT
-       select ARCH_INLINE_WRITE_UNLOCK_IRQ if !PREEMPT
-       select ARCH_INLINE_WRITE_UNLOCK_IRQRESTORE if !PREEMPT
-       select ARCH_INLINE_SPIN_TRYLOCK if !PREEMPT
-       select ARCH_INLINE_SPIN_TRYLOCK_BH if !PREEMPT
-       select ARCH_INLINE_SPIN_LOCK if !PREEMPT
-       select ARCH_INLINE_SPIN_LOCK_BH if !PREEMPT
-       select ARCH_INLINE_SPIN_LOCK_IRQ if !PREEMPT
-       select ARCH_INLINE_SPIN_LOCK_IRQSAVE if !PREEMPT
-       select ARCH_INLINE_SPIN_UNLOCK if !PREEMPT
-       select ARCH_INLINE_SPIN_UNLOCK_BH if !PREEMPT
-       select ARCH_INLINE_SPIN_UNLOCK_IRQ if !PREEMPT
-       select ARCH_INLINE_SPIN_UNLOCK_IRQRESTORE if !PREEMPT
+       select ARCH_INLINE_READ_LOCK if !PREEMPTION
+       select ARCH_INLINE_READ_LOCK_BH if !PREEMPTION
+       select ARCH_INLINE_READ_LOCK_IRQ if !PREEMPTION
+       select ARCH_INLINE_READ_LOCK_IRQSAVE if !PREEMPTION
+       select ARCH_INLINE_READ_UNLOCK if !PREEMPTION
+       select ARCH_INLINE_READ_UNLOCK_BH if !PREEMPTION
+       select ARCH_INLINE_READ_UNLOCK_IRQ if !PREEMPTION
+       select ARCH_INLINE_READ_UNLOCK_IRQRESTORE if !PREEMPTION
+       select ARCH_INLINE_WRITE_LOCK if !PREEMPTION
+       select ARCH_INLINE_WRITE_LOCK_BH if !PREEMPTION
+       select ARCH_INLINE_WRITE_LOCK_IRQ if !PREEMPTION
+       select ARCH_INLINE_WRITE_LOCK_IRQSAVE if !PREEMPTION
+       select ARCH_INLINE_WRITE_UNLOCK if !PREEMPTION
+       select ARCH_INLINE_WRITE_UNLOCK_BH if !PREEMPTION
+       select ARCH_INLINE_WRITE_UNLOCK_IRQ if !PREEMPTION
+       select ARCH_INLINE_WRITE_UNLOCK_IRQRESTORE if !PREEMPTION
+       select ARCH_INLINE_SPIN_TRYLOCK if !PREEMPTION
+       select ARCH_INLINE_SPIN_TRYLOCK_BH if !PREEMPTION
+       select ARCH_INLINE_SPIN_LOCK if !PREEMPTION
+       select ARCH_INLINE_SPIN_LOCK_BH if !PREEMPTION
+       select ARCH_INLINE_SPIN_LOCK_IRQ if !PREEMPTION
+       select ARCH_INLINE_SPIN_LOCK_IRQSAVE if !PREEMPTION
+       select ARCH_INLINE_SPIN_UNLOCK if !PREEMPTION
+       select ARCH_INLINE_SPIN_UNLOCK_BH if !PREEMPTION
+       select ARCH_INLINE_SPIN_UNLOCK_IRQ if !PREEMPTION
+       select ARCH_INLINE_SPIN_UNLOCK_IRQRESTORE if !PREEMPTION

 

 

So what am I missing?

 

 

[sfx2000]

On 3/5/2020 at 3:21 AM, kristsm said:

I didn't find much info in the forum on how to correctly apply RT patch to the Armbian (Orange Pi Zero Plus H5 board). 

 

Curious as to why one wants to apply the patches for preempt-rt in the first place? Do you have a reason/application that would benefit? Most times, one can see an overall decrease in application performance, and to take advantage of preempt-rt, the application must call pthreads in a very specific way - which means that many times, the app must also be rebuilt.

 

My reason for asking - if preempt-rt was so awesome, it would already be in the upstream linux kernel mainline and not require the patch. Avoiding politics around preempt-rt - many of the benefits here can be done without the patches, by selecting an appropriate scheduler - schedutil looks fairly interesting at the moment.

 

I work on networking oriented applications that have hard deadlines, on MIPS, PowerPC, ARM (both 32bit ARMv7a and 64-bit), and never had a need to apply the preempt-rt patches.

 

By networking oriented applications - I'm saying 20mSec framing for SIP-RTP (VoIP apps), along with 3GPP/UMTS signalling - if that ain't close to real-time, I don't know what is.

[kristsm]

On 3/7/2020 at 6:03 AM, sfx2000 said:

 

Curious as to why one wants to apply the patches for preempt-rt in the first place? Do you have a reason/application that would benefit? Most times, one can see an overall decrease in application performance, and to take advantage of preempt-rt, the application must call pthreads in a very specific way - which means that many times, the app must also be rebuilt.

 

My reason for asking - if preempt-rt was so awesome, it would already be in the upstream linux kernel mainline and not require the patch. Avoiding politics around preempt-rt - many of the benefits here can be done without the patches, by selecting an appropriate scheduler - schedutil looks fairly interesting at the moment.

 

I work on networking oriented applications that have hard deadlines, on MIPS, PowerPC, ARM (both 32bit ARMv7a and 64-bit), and never had a need to apply the preempt-rt patches.

 

By networking oriented applications - I'm saying 20mSec framing for SIP-RTP (VoIP apps), along with 3GPP/UMTS signalling - if that ain't close to real-time, I don't know what is.

 

Curious as to why you waste your time if you don't want to help?

 

 

[le51]

Hi there,

 

This topic interest me a lot. I'm using arm sbc for sound & music applications where low latency is important.

sfx2000's post reminds me other discussions on other forums (mostly computer "assisted" music oriented) where some thoughts/experiences are related to poor application performance regarding the low positive impact of a RT kernel.

From my own experience (x86, arm SBC's and RPI), RT kernel or not, good or bad audio interface in USB or I2S, depending on the load (cpu, memory, audio stream), I always have crack, glitches and xruns.

 

Without RT patch, what I get:

- With a H5 board with 2Gb memory (NanoPiK1plus) I can't go under 20ms of jack audio server latency without having a to unstable config but, as is, it is still "playable".

- With a NanoPiM4 and a HD USB audio interface, I have 5ms of latency, a lot of power for apps and a NvMe interface for a fast drive. That's great

 

So, I really ask myself for going further in kernel "hacking" with the RT patch.

 

 

[kristsm]

Tried to build Armbian Bionic today with sunxi-current ( I see the compiler have included patch 5.4.23-24 as of today, which seems the kernel should be @5.4.24?).  I moved 5.4.24 rt15 patch in userpatches/kernel/sunxi-current.

In kernel configuration I checked expert menu , which allowed me to choose fully preemptible kernel.

 

The compiler goes until CC [M]  kernel/kheaders.o, then it throws [error] ERROR in function compile_kernel [ compilation.sh: 378] and terminates the process.

 

Any tips on how to move further? 

[Igor]

22 minutes ago, kristsm said:

Any tips on how to move further? 

You will find tips in debug logs: output/debug/compilation.log

[kristsm]

9 hours ago, Igor said:

You will find tips in debug logs: output/debug/compilation.log

Thanks!

 

Ok, log file shows following errors:

 

fs/aufs/i_op.c: In function ‘au_pin_hdir_set_owner’:
fs/aufs/i_op.c:636:45: error: ‘struct rw_semaphore’ has no member named ‘owner’
  atomic_long_set(&p->hdir->hi_inode->i_rwsem.owner, (long)task);
                                             ^
make[2]: *** [fs/aufs/i_op.o] Error 1
make[1]: *** [fs/aufs] Error 2
make: *** [fs] Error 2
make: *** Waiting for unfinished jobs....
scripts/Makefile.build:265: recipe for target 'fs/aufs/i_op.o'
scripts/Makefile.build:500: recipe for target 'fs/aufs' failed
Makefile:1692: recipe for target 'fs' failed

 

 

 

 

[Igor]

54 minutes ago, kristsm said:

Ok, log file shows following errors:

 

fs/aufs/i_op.c: In function ‘au_pin_hdir_set_owner’:
fs/aufs/i_op.c:636:45: error: ‘struct rw_semaphore’ has no member named ‘owner’
  atomic_long_set(&p->hdir->hi_inode->i_rwsem.owner, (long)task);
                                             ^
make[2]: *** [fs/aufs/i_op.o] Error 1
make[1]: *** [fs/aufs] Error 2
make: *** [fs] Error 2
make: *** Waiting for unfinished jobs....
scripts/Makefile.build:265: recipe for target 'fs/aufs/i_op.o'
scripts/Makefile.build:500: recipe for target 'fs/aufs' failed
Makefile:1692: recipe for target 'fs' failed

Now its clear what to do. Build with AUFS="no" since it looks like it doesn't want to operate with RT or fix errors in the code if you need AUFS.

[sfx2000]

On 3/9/2020 at 5:47 AM, kristsm said:

Curious as to why you waste your time if you don't want to help?

 

My time is valuable - so let's spend it wisely...

 

Do you have a real-world use-case where there is value or not?

[kristsm]

14 hours ago, Igor said:

Now its clear what to do. Build with AUFS="no" since it looks like it doesn't want to operate with RT or fix errors in the code if you need AUFS.

Thanks Igor!

After AUFS disabled, the compiler threw at me another error regarding NFS, disabled that too and was greeted with completed build. Flashed the image in 64gb sd-card and after boot uname -a greeted me with the message:

Linux orangepizeroplus 5.4.24-rt13-sunxi64 #trunk SMP PREEMPT_RT Wed Mar 11 07:28:42 EET 2020 aarch64 aarch64 aarch64 GNU/Linux 

 

cyclic test results with all cores running in performance mode (no mods for ISOLCPUS):

 

sudo cyclictest -a -t -n -p80
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.96 0.85 0.57 1/165 3526

T: 0 ( 3518) P:80 I:1000 C: 161625 Min:      7 Act:   10 Avg:   10 Max:      49
T: 1 ( 3519) P:80 I:1500 C: 107750 Min:      7 Act:   14 Avg:   11 Max:      49
T: 2 ( 3520) P:80 I:2000 C:  80812 Min:      7 Act:   10 Avg:   10 Max:      35
T: 3 ( 3521) P:80 I:2500 C:  64650 Min:      8 Act:   10 Avg:   10 Max:      25