2 2
zador.blood.stained

[Framework] Build script improvement suggestions #1

Recommended Posts

Rebuilding kernel several times in a row, made some changes, that would be nice to have in build script by default, and also some thoughts on custom kernel patches.

 

1.Option to not overwrite kernel config before compilation.

Example: setting KERNEL_KEEP_CONFIG="yes" in build.sh

Changes (common.sh):

# use proven config
if [ "$KERNEL_KEEP_CONFIG" != "yes" ]; then cp $SRC/lib/config/$LINUXCONFIG.config $SOURCES/$LINUXSOURCE/.config; fi

2. Option to use ccache for kernel compilation (at least). ccache can be set by creating symlinks for cross-compiling tools, but having an option in build script is more useful and easier IMHO.

Example: setting USE_CCACHE="yes" in build.sh

Changes (in common.sh):

if [ "$USE_CCACHE" = "yes" ]
then
make $CTHREADS ARCH=arm CROSS_COMPILE="ccache arm-linux-gnueabihf-" all zImage 2>&1 | dialog --backtitle "$backtitle" --progressbox "Compiling kernel ..." 20 70
else
make $CTHREADS ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- all zImage 2>&1 | dialog --backtitle "$backtitle" --progressbox "Compiling kernel ..." 20 70
fi

Also here stderr of compilation process is redirected to stdout so compiler warnings do not tear dialog window (2>&1).

Compiling kernel with ccache right now, will report if it breaks anything.

Looks like it doesn't like invoking make deb-pkg after compilation and tries to compile everything again, will test later other than this 4.3 related bug compilation works.

 

3. Automatically pick up and apply kernel patches from custom directory.

Something like this:

# in kernel souces root dir
for patch in $SRC/patches/$KERNEL_TYPE/*.patch
do
  patch -p1 < $patch
done

KERNEL_TYPE can be "next" or "legacy" to separate patches for different kernel branches. This can be extended to using device specific directories if necessary.

Share this post


Link to post
Share on other sites

testing ccache...

 

very impressive speed!

 

with my VM (4 cores/4GB) kernel 4.2.5 compilation + debs:

 

*without ccache: 40-45 minutes

*with ccache: 10 minutes

 

 

 

btw in "compile.sh" I have added:

export BOARD="lamobo-r1"
export BRANCH="next"
export USE_CCACHE="yes"
also I have move to CROSS_COMPILE="ccache arm-....-" in "common.sh" as often as possible (of course, not including clean-up etc...)

 

 

Share this post


Link to post
Share on other sites

Added 1. and 2. (need to fix few other things than commit). Tnx.

 

Looks like it doesn't like invoking make deb-pkg after compilation and tries to compile everything again, will test later.

 

It's possible that deb-pkg is broken. This used to work fine?

 

I will also research this topic further ...

 

ADD #1: 

We need to compile deb-pkg the same way and it's fine
CROSS_COMPILE="ccache arm-linux-gnueabihf-"

 

ADD #2

 

Kernel 4.2.5 compilation + debs. Xeon E3 1231v3 @3.8Ghz, enough RAM & Crucial BX100

 

Compilation speed: 1 min 40s with ccache and 6 min 20s without

Share this post


Link to post
Share on other sites

Yes, I added ccache to make deb-pkg, but for me it compiles all sources again (second run with ccache should be much faster, but I didn't have time to finish it yesterday).

It should probably be noted in documentation or near USE_CCACHE option, that users need to set up at least ccache dir and size before enabling this option.

Share this post


Link to post
Share on other sites

Perhaps this way:

if [ "$USE_CCACHE" = "yes" ]; then USE_CCACHE="ccache"; else USE_CCACHE=""; fi

make $CTHREADS ARCH=arm CROSS_COMPILE="$USE_CCACHE arm-linux-gnueabihf-" zImage modules 2>&1 | dialog --backtitle "$backtitle" --progressbox "Compiling kernel $USE_CCACHE ..." 20 80
if [ $? -ne 0 ] || [ ! -f arch/arm/boot/zImage ]; then
		display_alert "Kernel was not built" "@host" "err"
	    exit 1
fi
make $CTHREADS ARCH=arm CROSS_COMPILE="$USE_CCACHE arm-linux-gnueabihf-" dtbs 2>&1 | dialog --backtitle "$backtitle" --progressbox "Compiling DTB $USE_CCACHE ..." 20 80
if [ $? -ne 0 ]; then
		display_alert "DTBs was not build" "@host" "err"
	    exit 1
fi

Share this post


Link to post
Share on other sites

Using bindeb-pkg target instead of deb-pkg seems to build kernel 4.3 without recompiling (which is logical since deb-pkg target is designed for making kernel source package amongst others and it executes "make clean" at the start).

Also tested ccache on u-boot compilation - works, but

if [ "$USE_CCACHE" = "yes" ]; then USE_CCACHE="ccache"; else USE_CCACHE=""; fi

needs to be called only once somewhere before u-boot and kernel compilation.

Share this post


Link to post
Share on other sites

OK, I'll place it to the proper place ... and fixed pipe exit code.

if [ ${PIPESTATUS[0]} -ne 0 ] || [ ! -f arch/arm/boot/zImage ]; then
		display_alert "Kernel was not built" "@host" "err"
	    exit 1
fi

Share this post


Link to post
Share on other sites

Igor, saw your commits on github.

1. KERNEL_KEEP_CONFIG definitely should be disabled by default, otherwise there will be no allwinner specific base config on fresh installations or major version upgrades. This option is for (re)building with different set of options/modules.

2. "dtbs" target is specific to Device Tree based kernels, building legacy kernels most likely will fail at this stage.

3. "bindeb-pkg" is not temporary but is a new default option for kernels that support this target: https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/scripts/package/builddeb?id=3716001bcb7f5822382ac1f2f54226b87312cc6b

 

I guess that having USE_CCACHE enabled by default is okay even with default config ($HOME/.ccache, limit 1G), but it has some scenarios where it won't work as expected (recompiling kernel with changed config or manually pached include files): https://lists.samba.org/archive/ccache/2014q1/001172.html

Share this post


Link to post
Share on other sites

1. Yes, you are right. Fixed

2. I made few tests with old kernel and it does not break ...

3. Aha, so just my comment is deprecated? ;)

 

Let's see how things will go fuhter. I only made few runs.

Share this post


Link to post
Share on other sites

https://github.com/igorpecovnik/lib -b master (not default yet)

 

1. Redesigned patching process: "per kernel family", picking patches from dirs, fixed and cleaned patches

AFTER

https://github.com/igorpecovnik/lib/blob/master/patching.sh

BEFORE

https://github.com/igorpecovnik/lib/blob/second/patching.sh

2. Added user patches dirs, one for kernel, one for u-boot, outside of /lib 

https://github.com/igorpecovnik/lib/blob/master/documentation/geek-faq.md#user-patching

3. Added new devices with "WIP" status: Lemaker Guitar and H3 Orange

4. Cleaning sources (WIP)

5. Many bug fixes + probably produced new bugs ...  need help with testing & fixing  :P

 

Now it's time for (new) suggestions   :D

Share this post


Link to post
Share on other sites
Just saw your commits while browsing github :) 
 
I thought about new suggestions:
 
1. Renaming some scripts. Using "compile.sh" to launch process, "common.sh" to compile u-boot & kernel, "main.sh" - i have no idea where and when it is used without searching through other scripts... it is counter-intuitive  :)
 
2. Using apt-cacher-ng to speed up building all images (or rebuilding one image) by caching downloaded packages locally.
  • should work without extra configuration on host system
  • requires adding extra parameter to debootstrap (first stage) - mirror address, i.e.
debootstrap --include=openssh-server,debconf-utils$sysvinit --arch=armhf --foreign $RELEASE $DEST/cache/sdcard/ http://localhost:3142/httpredir.debian.org/debian

or something like this. Obviously needs separate mirror address for ubuntu.

  • requires adding extra argument to apt-get (update and install commands):
 -o Acquire::http::Proxy "http://localhost:3142"

obviously it's easier to do if number of apt-get invocations is minimal, right now there are at least 3 separate install command - locales, console packages, rest of Armbian, + there are extra for building desktop images in another script.

 

3. With "enough RAM" it may be beneficial to even use ramdisk instead of loop device, but I think this is not for near future...

 

 

I'll try to test kernel compilation as soon as I sort out some PC hardware issues or free enough disk space to move to another PC.

Share this post


Link to post
Share on other sites

Was looking through changelog between branches, noticed this:

# configure MIN / MAX speed for cpufrequtils
sed -e "s/MIN_SPEED=\"0\"/MIN_SPEED=\"$CPUMIN\"/g" -i $DEST/cache/sdcard/etc/init.d/cpufrequtils
sed -e "s/MAX_SPEED=\"0\"/MAX_SPEED=\"$CPUMAX\"/g" -i $DEST/cache/sdcard/etc/init.d/cpufrequtils

I think it's not a good idea to alter init.d script because on package upgrade any changes may be lost. cpufreq settings should be defined in /etc/default/cpufrequtils

➜  ~  % cat /etc/default/cpufrequtils
ENABLE="true"
GOVERNOR="ondemand"
MAX_SPEED=960000
MIN_SPEED=720000

Also small note here:

Available frequencies on mainline for A20 based boards (sun7i-a20.dtsi):

960000, 912000, 864000, 720000, 528000, 312000, 144000 

for A10 boards except A10 Lime (sun4i-a10.dtsi):

1008000, 912000, 864000, 624000 

A10 Lime doesn't have 1008000 from this list.

Share this post


Link to post
Share on other sites

Igor,

 

FYI

 

testing a fresh install of "-b master"

 

got a kernel 4.4.0-rc3 to compile (I guess your target is 4.3 4.2.6)

 

and then

 

 arch/arm/boot/dts/Makefile:639: *** commands commence before first target.   x

                             x make: *** [dtbs] Error 2

 

 

 

EDIT:

 

added

BOARD="lamobo-r1"
BRANCH="next"
KERNELTAG="v4.3"
RELEASE="trusty"
BUILD_DESKTOP="No"

I don't know what it is about fresh installs but

root@server1404:~/armtest# KERNELTAG="v"`wget -qO-  https://www.kernel.org/finger_banner | grep "The latest st" | awk '{print $NF}'`
root@server1404:~/armtest# echo $KERNELTAG
v4.2.6
???

 

 

EDIT #2:

 

whatever KERNELTAG I apply (empty, v4.3, v4.2.6) I end with

 

error: pathspec 'v4.2.6' did not match any file(s) known to git.

[ o.k. ] Patching kernel [ sunxi-next 4.4.0rc3 ]

 

I can see tags from http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/refs/tags/

 

EDIT #3:

fetch_from_github (){
#--------------------------------------------------------------------------------------------------------------------------------
# Download or updates sources from Github
#--------------------------------------------------------------------------------------------------------------------------------
if [ -d "$SOURCES/$2" ]; then
        cd $SOURCES/$2
        if [[ "$3" != "" ]]; then
                PULL=$(git checkout $FORCE -q $3)

is $FORCE related to $FORCE_CHECKOUT ?

so $FORCE = "-f" ?

 

ok, fgrep is my friend in this puzzle

if [ "$FORCE_CHECKOUT" = "yes" ]; then FORCE="-f"; else FORCE=""; fi
ok so fetch_from_github works with

***************
LINUXKERNEL=git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git --depth 1
LINUXSOURCE=linux-mainline
LINUXCONFIG=linux-sunxi-next
LINUXDEFAULT=master
LINUXFAMILY=sunxi
FIRMWARE=
***************

fetch_from_github "$LINUXKERNEL" "$LINUXSOURCE" "$LINUXDEFAULT"

in my case

fetch_from_github "git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git --depth 1" "linux-mainline" "master"

 

what is "--depth 1" doing here ? (ok applies only to "git clone" not "git checkout")

 

 

EDIT #4:

 

ok there is an issue with --depth 1

 

http://stackoverflow.com/questions/6941889/is-it-safe-to-shallow-clone-with-depth-1-create-commits-and-pull-updates-aga

 

if you start a fresh install it only clone the HEAD (aka 4.4-rc3 tag)

 

so no other tags... including 4.2.6 and 4.3

 

ok an old Armbian tool mystery is solved

root@server1404:~/armtest/sources# git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux-mainline
Cloning into 'linux-mainline'...
remote: Counting objects: 4855800, done.
remote: Compressing objects: 100% (417582/417582), done.
remote: Total 4855800 (delta 335700), reused 0 (delta 0)
Receiving objects: 100% (4855800/4855800), 1.10 GiB | 3.24 MiB/s, done.
Resolving deltas: 100% (4002996/4002996), done.
Checking connectivity... done.
Checking out files: 100% (52217/52217), done.
root@server1404:~/armtest/sources/linux-mainline# git tag
v2.6.11
[...]
v4.2
[...]
v4.2.1
v4.2.2
v4.2.3
v4.2.4
v4.2.5
v4.2.6
v4.3
[...]
v4.4-rc1
v4.4-rc2
v4.4-rc3

root@server1404:~/armtest/sources/linux-mainline# git checkout -f v4.3
Note: checking out 'v4.3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 6a13feb... Linux 4.3

root@server1404:~/armtest/sources/linux-mainline# git checkout -f v4.2.6
Checking out files: 100% (10091/10091), done.
Previous HEAD position was 6a13feb... Linux 4.3
HEAD is now at 1c02865... Linux 4.2.6

So for a fresh install of linux-mainline

 

I guess --depth 20 is probably needed to have kernel 4.2 4.3 and 4.4-rcX

 

 

EDIT #5:

 

probably a smarter proposal "--depth 1 --no-single-branch" so the tags are retrieved and the checkout can move from tag to tag

root@server1404:~/armtest/sources# git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux-mainline-test --depth 1 --no-single-branch
Cloning into 'linux-mainline-test'...
remote: Counting objects: 808871, done.
remote: Compressing objects: 100% (350845/350845), done.
remote: Total 808871 (delta 690067), reused 551343 (delta 451473)
Receiving objects: 100% (808871/808871), 431.61 MiB | 3.60 MiB/s, done.
Resolving deltas: 100% (690067/690067), done.
Checking connectivity... done.
Checking out files: 100% (52217/52217), done.
root@server1404:~/armtest/sources# cd linux-mainline-test
root@server1404:~/armtest/sources/linux-mainline-test# git checkout -f v4.2.6
Checking out files: 100% (16472/16472), done.
Note: checking out 'v4.2.6'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 1c02865... Linux 4.2.6
root@server1404:~/armtest/sources/linux-mainline-test# git checkout -f v4.3
Previous HEAD position was 1c02865... Linux 4.2.6
HEAD is now at 6a13feb... Linux 4.3

cool, I have learnt something with git :-)

 

EDIT #6:

 

so I am compiling kernel 4.2.6 as planned

 

I am ending with

arch/arm/boot/dts/Makefile:586: *** commands commence before first target.   x
                                           x make: *** [dtbs] Error 2 

EDIT #7:

 

deleted "lib" directory and re-launched compiler.sh

 

=> DTS compiles ok but it is the "stable" lib tree

 

so again

 

git clone https://github.com/igorpecovnik/lib-b master

 

same issue

 Compiling DTB ccache ...                                                     x
                                           xqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqx
                                           x   CHK     include/config/kernel.release                                      x
                                           x   CHK     include/generated/uapi/linux/version.h                             x
                                           x   CHK     include/generated/utsrelease.h                                     x
                                           x make[1]: `include/generated/mach-types.h' is up to date.                     x
                                           x   CHK     include/generated/timeconst.h                                      x
                                           x   CHK     include/generated/bounds.h                                         x
                                           x   CHK     include/generated/asm-offsets.h                                    x
                                           x   CALL    scripts/checksyscalls.sh                                           x
                                           x arch/arm/boot/dts/Makefile:586: *** commands commence before first target.   x
                                           x make: *** [dtbs] Error 2     

 

Share this post


Link to post
Share on other sites

I guess --depth 20 is probably needed to have kernel 4.2 4.3 and 4.4-rcX

 

I solve this git handling part other way. Will submit changes today. 

 

make[1]: `include/generated/mach-types.h' is up to date.

That's due to one broken patch. Already fixed.

 

Tnx!

Share this post


Link to post
Share on other sites

Igor,

 

"--depth 1 --no-single-branch" seems far better than "--depth 20" because the download is faster (>1 MB/s but more files to download vs. 100kB/s) and there is less computation to do (as seen while testing the 2 options)

 

ok, I will retry "-b master" thanks!

Share this post


Link to post
Share on other sites

the "userpatches" feature is pretty promising...

 

branch "second" is working fine with fresh install + "--depth 1 --no-single-branch" +

BOARD="lamobo-r1"
BRANCH="next"
KERNELTAG="v4.2.6"
BUILD_DESKTOP="No"

and RELEASE="trusty" or "wheezy" or "jessie"

 

 

branch "master" still choke on "arch/arm/boot/dts/Makefile:586: *** commands commence before first target. "

Share this post


Link to post
Share on other sites

I finally managed to make separate "build server" for myself  :)

Ready to test master branch as soon as Igor updates it (I installed Ubuntu 15.04, so only mainline kernel for now). Also want to propose some enhancements (I'll make a pull request):

 

Adding user patches is cool, but I think I can make some changes so you can override (apply user patch instead of existing in lib/patch) or "mask" (prevent patch in lib/patch from applying) them.

Also I want to add make-style parameters handling, so you can run for example "sudo ./compile.sh KERNEL_ONLY=yes BOARD=cubietruck BRANCH=next" and avoid selecting board and branch manually or editing compile.sh each time.

Share this post


Link to post
Share on other sites

There will be no updates, at least not for a while.

 

I just accidentally delete the work of the whole week :( a hundred hours went to trash.  :( :( :(

 

I was playing too much with git handling that I completely crashed my working git. Of course I had no backup for this and I only could salvage few files which I just pushed up.

Share this post


Link to post
Share on other sites

                                            ┌───────────────────────┤ Choose a board ├────────────────────────┐                                             
                                            │                                                                 │                                             
                                            │  Supported:                                                     │                                             
                                            │                                                                 │                                             
                                            │    cubieboard4           A80 octa core 2Gb SoC Wifi             │                                             
                                            │    aw-som-a20            A20 dual core SoM                      │                                             
                                            │    cubieboard            A10 single core 1Gb SoC                │                                             
                                            │    cubieboard2           A20 dual core 1Gb SoC                  │                                             
                                            │    cubietruck            A20 dual core 2Gb SoC Wifi             │                                             
                                            │    lime-a10              A10 single core 512Mb SoC              │                                             
                                            │    lime                  A20 dual core 512Mb SoC                │                                             
                                            │    lime2                 A20 dual core 1Gb SoC                  │                                             
                                            │    micro                 A20 dual core 1Gb SoC                  │                                             
                                            │    pcduino3nano          A20 dual core 1Gb SoC                  │                                             
                                            │    bananapim2            A31 quad core 1Gb SoC Wifi             │                                             
                                            │    bananapi              A20 dual core 1Gb SoC                  │                                             
                                            │    bananapipro           A20 dual core 1Gb SoC Wifi             │                                             
                                            │    lamobo-r1             A20 dual core 1Gb SoC Switch           │                                             
                                            │    orangepi              A20 dual core 1Gb SoC Wifi USB hub     │                                             
                                            │    orangepimini          A20 dual core 1Gb SoC Wifi             │                                             
                                            │    orangepiplus          H3 quad core 1Gb SoC Wifi USB hub      │                                             
                                            │    cubox-i               Freescale iMx dual/quad core Wifi      │                                             
                                            │    udoo                  Freescale iMx dual/quad core Wifi      │                                             
                                            │    udoo-neo              Freescale iMx singe core Wifi          │                                             
                                            │    guitar                S500 Lemaker Guitar Action quad core   │                                             
                                            │    odroid                Exynos5422 XU3/XU4 octa core           │                                             
                                            │                                                                 │                                             
                                            │                                                                 │                                             
                                            │                                                                 │                                             
                                            │                                                                 │                                             
                                            │                                                                 │                                             
                                            │                <Ok>                    <Cancel>                 │                                             
                                            │                                                                 │                                             
                                            └─────────────────────────────────────────────────────────────────┘                                             

This far I got:

 

- added few new boards

- proper git sources switching

- completely reorganized configuration

- legacy / next / dev versions

- 80% of code cleaned.

 

Starting tomorrow from here:

https://github.com/igorpecovnik/lib

Share this post


Link to post
Share on other sites

A break would be the best / most productive. I'll see how I'll feel in the morning ;)

 

If you wants to jump in - I'll work on the configuration.sh + few things in main.sh for a while.

 

... better sources handling is a open part or even only the code cleaning, where is needed, cpu speed fixes ... $GOVERNOR will be now in config.

Share this post


Link to post
Share on other sites

Just a thought about install_packet function...

I think installing packages all at once should be faster than installing them one by one (extra dependencies calculation, reading/writing dpkg database, processing pre/postinstall triggers). Maybe rework it to display progress from apt-get in simple dialog text box?

Only downside is - if one package cannot be installed (wrong package name for example), whole process fails.

Share this post


Link to post
Share on other sites
 
Changelog:

Restarting script with sudo if needed (compile.sh)

Check for superuser rights is no longer needed (main.sh)
 
Script parameters handling (main.sh)
Handling BUILD_ALL option needs extra refactoring, so for now this parameter is not supported
 
Better handling of KERNEL_KEEP_CONFIG option (common.sh)
 
Run make oldconfig before make menuconfig (common.sh)
Handling of KERNEL_CONFIGURE option for oldconfig (common.sh)
 
Advanced patching is now more advanced than it probably needs to be (patching.sh)
 
TODO:
install_packet iteration to batch, better displaying of progress ?
renaming common.sh -> compilation.sh, compile.sh -> make.sh, updating source and documentation for this change
 

Edit: made some fixes

 

Testing patching process: for mainline stable kernel "$LINUXFAMILY-$BRANCH" parses to "sun7i-next" or "sun4i-next" instead of "sunxi-next". Is this a bug or a feature?  :)

 

Edit 2: pushed advanced_patching changes. Mainline patches are still broken due to wrong $LINUXFAMILY

Share this post


Link to post
Share on other sites

A bug ;)

 

This is also an exception. When added to configuration it should work properly, like for .configs

https://github.com/igorpecovnik/lib/blob/master/configuration.sh#L269-L270

 

https://github.com/zador-blood-stained/lib/blob/master/patching.sh#L147

This part can be dumped. I made patches, fbtft sources are not changing anymore. It has been merged into mainline for some time.

 

Tnx.

 

I'll try later to merge this together

 

EDIT:
adv.patches doesn't collect my custom patches even they are in proper dir.

Share this post


Link to post
Share on other sites

Hm. I tested it briefly on cubietruck sun7i kernel, it got all patches from /lib and it skipped fbtft patch when I created empty file userpatches/kernel/sun7i-default/fbtft_for_older.patch

I'm waiting for your solution to passing correct LINUXFAMILY to patching function to do proper tests.

What exactly did you test?

 

Edit: Just checked. Works like intended for me

➜  ~  % cd armbian/userpatches/kernel/sun7i-default
➜  sun7i-default  % _ cp ~/armbian/lib/patch/kernel/small_lcd_drivers.patch test-patch.patch
➜  sun7i-default  % ~
➜  ~  % armbian
➜  armbian  % ./compile.sh BOARD=cubietruck KERNEL_ONLY=yes BRANCH=default
....
....
[ o.k. ] ... test-patch.patch sun7i-default 3.4.104 [ succeeded ]

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.
2 2