linux-headers-rk3399 is broken for building out-of-tree modules - with fix

[s_frit]

This report applies to NanoPi-4 RK3399 boards (NanoPC T4, NanoPi M4, NanoPi NEO4) running Armbian Bionic (Friendlyarm-based legacy 4.4.y kernel). In other words, boards targeted by Armbian rk3399.conf and Friendlyarm kernel but *NOT* baords using rockchip64.conf and Ayufan kernel.

 

There is a bug in the friendlyarm linux 4.4 version of /scripts/package/builddeb that causes the linux-headers package to be incomplete. Attempts to build out-of-tree kernel modules to fail with errors of the form "fatal error: ../../arm/include/asm/opcodes.h: No such file or directory". For example here is the output trying to build a simple kernel module from this tutorial:

Spoiler

$ make

make -C /lib/modules/4.4.172-rk3399/build M=/home/ross/Documents/lkm_example modules

make[1]: Entering directory '/usr/src/linux-headers-4.4.172-rk3399'

CC [M] /home/ross/Documents/lkm_example/lkm_example.o

In file included from ./arch/arm64/include/asm/sysreg.h:25:0,

from ./arch/arm64/include/asm/cputype.h:99,

from ./arch/arm64/include/asm/cachetype.h:19,

from ./arch/arm64/include/asm/cache.h:19,

from include/linux/cache.h:5,

from include/linux/printk.h:8,

from include/linux/kernel.h:13,

from include/linux/list.h:8,

from include/linux/module.h:9,

from /home/ross/Documents/lkm_example/lkm_example.c:2:

./arch/arm64/include/asm/opcodes.h:5:10: fatal error: ../../arm/include/asm/opcodes.h: No such file or directory

#include <../../arm/include/asm/opcodes.h>

^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

compilation terminated.

scripts/Makefile.build:283: recipe for target '/home/ross/Documents/lkm_example/lkm_example.o' failed

make[2]: *** [/home/ross/Documents/lkm_example/lkm_example.o] Error 1

Makefile:1499: recipe for target '_module_/home/ross/Documents/lkm_example' failed

make[1]: *** [_module_/home/ross/Documents/lkm_example] Error 2

make[1]: Leaving directory '/usr/src/linux-headers-4.4.172-rk3399'

Makefile:3: recipe for target 'all' failed

make: *** [all] Error 2

 

 

@hbmaennchen encountered the same problem when compiling an existing module here: https://forum.armbian.com/topic/8693-headers-44162/

 

This is a known bug that has been encountered in other kernel header distributions, for example here and here.

 

The fix was applied to mainline in 2016 and is a one line change to the scripts/package/builddeb script. The fix was posted to LKML in March 2016 and was committed to mainline here:

 

https://github.com/torvalds/linux/commit/962475ac2f9662fe69a9f86220ab31bbbf3911d2#diff-03db9657e7716d9ca27a586068430b37

 

Please could this patch be applied to the Armbian build process?

 

I have no experience with Armbian build but if you need me to generate the patch in the correct format and submit a pull request I can do so, just let me know that you'd like me to do it.

 

Update: my system information

NanoPC T4

Ubuntu 18.04.1 LTS bionic

4.4.172-rk3399

header package installed: linux-headers-rk3399

 

Update: steps to reproduce

 

Using a NanoPi-4 RK3399 board (NanoPC T4, NanoPi M4, NanoPi NEO4) running Armbian Bionic (Friendlyarm-based legacy 4.4.y kernel)

 

First install the linux-headers package:

$ sudo apt-get install linux-headers-rk3399

 

Then, place the attached files lkm_example.c and Makefile into a directory (e.g. named lkm_example). cd to that directory, and type `make`. You should then get the error in the spoiler above.

 

The expected behavior is for the make to succeed without error (which should happen if you are running any other board with the appropriate linux-headers-... package, e.g. the same board with the dev kernel, and linux-headers-dev-rk3399 package installed.

 

 

 

Makefile

lkm_example.c

[s_frit]

For completeness, the issue is only present in friendlyarm 4.4 repo:

https://github.com/friendlyarm/kernel-rockchip/blob/nanopi4-linux-v4.4.y/scripts/package/builddeb#L325

 

Which is the source for the default kernel for 'rk3399' family boards such as nanopc-t4 (https://github.com/armbian/build/blob/master/config/sources/rk3399.conf).

 

It looks like the fix has already been applied to both the 4.4 and mainline ayufan-rock64 repositories (which are the source for 'rockchip64' boards):

https://github.com/ayufan-rock64/linux-kernel/blob/release-4.4/scripts/package/builddeb#L325

https://github.com/ayufan-rock64/linux-mainline-kernel/blob/master/scripts/package/builddeb#L158

(not sure whether Armbian is sourcing the release-4.4 branch though).

 

[s_frit]

I have made progress by applying the attached patch and rebuilding linux-headers-rk3399.deb

I then installed the deb manually with `sudo dpkg -i ./linux-headers-rk3399_5.74_arm64.deb` then `sudo apt-get install -f`

 

However now I'm getting a different error when compiling test kernel module attached above "/bin/sh: 1: ./scripts/recordmcount: not found":

Spoiler

$ make

make -C /lib/modules/4.4.172-rk3399/build M=/home/ross/Documents/lkm_example modules

make[1]: Entering directory '/usr/src/linux-headers-4.4.172-rk3399'

CC [M] /home/ross/Documents/lkm_example/lkm_example.o

/bin/sh: 1: ./scripts/recordmcount: not found

scripts/Makefile.build:283: recipe for target '/home/ross/Documents/lkm_example/lkm_example.o' failed

make[2]: *** [/home/ross/Documents/lkm_example/lkm_example.o] Error 127

make[2]: *** Deleting file '/home/ross/Documents/lkm_example/lkm_example.o'

Makefile:1499: recipe for target '_module_/home/ross/Documents/lkm_example' failed

make[1]: *** [_module_/home/ross/Documents/lkm_example] Error 2

make[1]: Leaving directory '/usr/src/linux-headers-4.4.172-rk3399'

Makefile:3: recipe for target 'all' failed

make: *** [all] Error 2

 

I have found some references to needing to "make scripts" in the headers directory, however I've been unable to find a way to do that successfully. Any hints would be appreciated.

 

Update: when running `sudo make scripts` in /usr/src/linux-headers-4.4.172-rk3399 I get the following error

Spoiler

nanopct4:/usr/src/linux-headers-4.4.172-rk3399$ sudo make scripts
  CHK     scripts/mod/devicetable-offsets.h
  HOSTCC  scripts/selinux/genheaders/genheaders
scripts/selinux/genheaders/genheaders.c:13:10: fatal error: classmap.h: No such file or directory
 #include "classmap.h"
          ^~~~~~~~~~~~
compilation terminated.
scripts/Makefile.host:91: recipe for target 'scripts/selinux/genheaders/genheaders' failed
make[3]: *** [scripts/selinux/genheaders/genheaders] Error 1
scripts/Makefile.build:484: recipe for target 'scripts/selinux/genheaders' failed
make[2]: *** [scripts/selinux/genheaders] Error 2
scripts/Makefile.build:484: recipe for target 'scripts/selinux' failed
make[1]: *** [scripts/selinux] Error 2
Makefile:590: recipe for target 'scripts' failed
make: *** [scripts] Error 2
 

 

This appears to be a similar issue to here: 

https://forum.armbian.com/topic/7253-rock64-cannot-install-wireguard-module-on-ubuntu-1804/

I will try the suggested fix once I have had some sleep.

 

fix-linux-headers-rk3399-pkg.patch

[Igor]

2 hours ago, s_frit said:

I will try the suggested fix once I have had some sleep.

Thanks! https://github.com/armbian/build/commit/4348bf68a002d30c9cb4d166f52bc5ce4057a73b

[s_frit]

Following on from my previous status, where `make scripts` was failing,  I found that selinux security related headers were absent from the linux-headers package. The best (most minimal) fix for this I found was to modify builddeb to add the security include files to the linux-headers package. The patch is an additional one line change (in addition to the fix-linux-headers-rk3399-pkg.patch above):

 

+(cd $srctree; find security/*/include -type f) >> "$objtree/debian/hdrsrcfiles"

 

With this change, both `make scripts` and my test kernel module compile successfully. And the test kernel module loads/unloads correctly.

I think this is the good change because it only adds include files, and this addition is already done for sunxi-dev and rockchip-dev (maybe elsewhere too?) e.g.:

https://github.com/armbian/build/blob/master/patch/kernel/sunxi-dev/general-packaging-4.20-dev.patch#L153

https://github.com/armbian/build/blob/master/patch/kernel/rockchip-dev/general-packaging-4.20-dev.patch#L153

 

@Igor I have attached a new patch file called fix-linux-headers-rk3399-pkg-v2.patch this patch includes the change from the previous patch. Please use the new -v2 patch to overwrite the existing fix-linux-headers-rk3399-pkg.patch in Armbian build. Once this propagates to the package repository I will re-run my tests.

 

For future reference, here is how I performed the test:

 

First, make sure that the existing linux-headers package is uninstalled:

Spoiler

nanopct4:/$ cd /usr/src

nanopct4:/usr/src$ sudo apt list --installed | grep linux-headers

nanopct4:/usr/src$ sudo apt-get remove linux-headers-rk3399

nanopct4:/usr/src$ sudo rm -rf linux-headers-4.4.172-rk3399/

 

Then install the new (working) linux-headers package that I built using Armbian build (once Igor updates the repo, just install linux-headers in the usual way).

Spoiler

nanopct4:/usr/src$ sudo dpkg -i ~/linux-headers-rk3399_5.74_arm64-v2.deb

nanopct4:/usr/src$ sudo apt-get install -f

 

Run `make scripts` as per https://docs.armbian.com/User-Guide_Advanced-Features/#how-to-build-a-wireless-driver

Spoiler

nanopct4:/usr/src$ cd linux-headers-4.4.172-rk3399/

nanopct4:/usr/src/linux-headers-4.4.172-rk3399$ sudo make scripts

 

`make scripts` should complete with no error.

 

Then running `make` in lkm_example directory (described in my first post) should work. And you can follow the instructions in the tutorial to load/unload it from the kernel.

 

fix-linux-headers-rk3399-pkg-v2.patch

[Igor]

2 hours ago, s_frit said:

Once this propagates to the package repository I will re-run my tests.

make scripts finishes without error now:

root@nanopct4:/usr/src/linux-headers-4.4.172-rk3399# make scripts
  CHK     scripts/mod/devicetable-offsets.h

 

[s_frit]

On 2/5/2019 at 10:55 AM, Igor said:

make scripts finishes without error now:

root@nanopct4:/usr/src/linux-headers-4.4.172-rk3399# make scripts
  CHK     scripts/mod/devicetable-offsets.h

 

 

I just upgraded to the 4.4.174 headers from stable (with apt upgrade from the official armbian apt repo, not using side-loaded packages), and now there is a new problem with `make scripts`:

 

ross@nanopct4:/usr/src/linux-headers-4.4.174-rk3399$ sudo make scripts
scripts/kconfig/conf  --silentoldconfig Kconfig
net/Kconfig:83: can't open file "net/wireguard/Kconfig"
scripts/kconfig/Makefile:37: recipe for target 'silentoldconfig' failed
make[2]: *** [silentoldconfig] Error 1
Makefile:576: recipe for target 'silentoldconfig' failed
make[1]: *** [silentoldconfig] Error 2
make: *** No rule to make target 'include/config/auto.conf', needed by 'scripts'.  Stop.

 

@Igor Is it possible that this is fixed with the [Wireguard fix] commit 53fcb9852 ?

 

[s_frit]

Workaround for latest `make scripts` issue is described here: 

 

[Lope]

To reproduce issue
 

git clone --depth 1 https://github.com/gordboy/rtl8812au.git
cd rtl8812au
make -j 4

#you'll see that it fails to do modpost with the stock 4.19.20 kernel.
#
#Previously I had compiled a custom kernel for 4.19.13.
#I downgraded to my 4.19.13 custom kernel and was able to complete the above without any issue.
#Kernel headers seem to be broken.

#If you're going to run the next step, you should backup your armbian 8812au.ko kernel module.
#next step I couldn't get to
sudo make install

 

[Lope]

My (directly) above post got moved here from the Armbian H5 forum

Please note:
1. I experienced the issue on a NanoPi Neo2 v1.0 (H5) board.
2. I did not experience the issue on a Rockchip SoC.

 

Nowhere does this thread mention Neo2 or H5, so I'm mentioning this here so it can at least come up in a search.

Maybe my above post got moved here in error?

[Igor]

11 hours ago, Lope said:

Maybe my above post got moved here in error?

No, the problem is shared. (workaround, mentioned few posts back, should work)