[Framework] Build script improvement suggestions #1

[Igor]

OK, check my update. 

 

Not working for me. BASH issue? I think this part is critical:

 

https://github.com/igorpecovnik/lib/blob/master/patching.sh#L54-L56

[zador.blood.stained]

Fixed. Sorting/removing duplicates was broken.

 

Edit:

[ o.k. ] Compiling uboot. Please wait. [ 2016.01rc2 ]

Looks like displaying u-boot tag is broken. Local version is based on 2015.10

[Igor]

Cool ! 

 

I gave up on this "advanced" git handling, now changing kernel and u-boot sources. Each tags will have own subdir, all with shallow clone of course.

 

u-boot-odroid/unttaged

u-boot-odroid/2015.10

u-boot-odroid/2015.04

 

... kernel likewise. According to this, I'll fix the compilation part so ... 

[zador.blood.stained]

Fixed dtb compilation (removed extra space in 0013-lamobo_r1_dts.patch at the beginning of the line).

Mainline kernel compilation is working.

[ o.k. ] Kernel building done [ @host ]
[ o.k. ] File name [ linux-image-next-sunxi_5.00_armhf.deb ]
[ o.k. ] Runtime [ 4 min ]

BTW, you can add my repo as another remote and merge commits or even branches from it, this way it will be easier for me to merge your changes back into my working copy (and it may be faster and easier for you).

 

Edit: some config names are broken after configuration update:

[ o.k. ] Compiling kernel [ @host ]
cp: cannot stat ‘/home/zador/armbian/lib/config/linux-sun7i-default.config’: No such file or directory

[Igor]

This lamobo fix was "lost in the crash" together with others small fixes yet to be rediscovered

 

BTW, you can add my repo as another remote and merge commits or even branches from it, this way it will be easier for me to merge your changes back into my working copy (and it may be faster and easier for you).

 

Agreed. I used to merge others snippets via Github pull request. My git handling knowledge need some upgrading I see it's simple but I will do a tests before. ASAP

 

I fixed download / patching part ... will update when I fix this .config part and made few test runs.

 

EDIT: I added another remote to my branch and made some merging w/o problems but I got quite some conflicts which needs slow manual resolving. 

[zador.blood.stained]

/home/zador/armbian/lib/main.sh: line 264: [: ==: unary operator expected

Line 264:

[ $CLEAN_LEVEL == 4 ] && cleaning "$CLEAN_LEVEL"

Line 292:

[ $CLEAN_LEVEL != 4 ] && cleaning "$CLEAN_LEVEL" 

Something here is not right. I believe line 264 is obsolete and needs to be removed, line 292 should be changed to

[ "$CLEAN_LEVEL" != "4" ] && cleaning "$CLEAN_LEVEL" 

[Igor]

All cleaning but removing sources must be done after git operations, right?

 

"" is needed and 264 too

[Igor]

I am done until evening. 

Some ideas: 

- I need to see new kernel features, so make olddefconfig / oldconfig must be an option and always out of dialog

- this dialog on/off option would be cool also a blank, no console output, only log

- when mostly done we can alter/adjust script names 

[zador.blood.stained]

At line 264 it complains about "==" operator, which is specific to advanced test ("[[").

Hm, missed true meaning of clean level 4, "dell all output" in compile.sh is not descriptive enough in this case 

[zador.blood.stained]

My logic with oldconfig/olddefconfig is:

if KERNEL_CONFIGURE is true, then process is designed to be interactive and it will ask about new options before bringing menu configuration dialog;

otherwise process is not interactive, and it won't ask users about some new kernel options if they don't want to mess with kernel configuration at all, safe defaults will be assumed for new options.

 

If you don't like this, you can change it to always be oldconfig, it's not 100% needed.

 

Make *config right now doesn't use dialog as far as I see.

 

- this dialog on/off option would be cool also a blank, no console output, only log

I'll try to implement this next.

[Igor]

Hm, missed true meaning of clean level 4, "dell all output" in compile.sh is not descriptive enough in this case 

 

I know.  More and better comments are needed. I made few test runs and it looks good ... busy with life until evening.

[zador.blood.stained]

For clean level - instead of numbers it can be a list (comma or space separated) of things to clean, i.e. CLEAN_LEVEL="debs,srcdir,make", and it can be implemented like this:

if [[ $CLEAN_LEVEL == *debs* ]] ; then rm -rf "$DEST/debs/"; fi

[Igor]

OK, more logical. Fine with me.

[zador.blood.stained]

I don't know how to implement this with the new cleaning logic

# Choosing kernel if debs are present
if [[ $BRANCH == "next" ]]; then
	MYARRAY=($(ls -1 $DEST/debs/linux-image* | awk '/next/' | sed ':a;N;$!ba;s/\n/;/g'))
	else
	MYARRAY=($(ls -1 $DEST/debs/linux-image* | awk '!/next/' | sed ':a;N;$!ba;s/\n/;/g'))
fi
if [[ ${#MYARRAY[@]} != "0" && $KERNEL_ONLY != "yes" ]]; then choosing_kernel; fi

Either it needs separate parameter in compile.sh or it needs to be removed or reworked somehow.

 

Edit: I left it commented in main.sh

[wildcat_paris]

Guys,

 

You have done a tremendeous update for Armbian kernel compiling on current beta "master" branch

 

I have tested

BOARD="lamobo-r1"
BRANCH="dev"
KERNELTAG="v4.4-rc4"
PROGRESS_LOG_TO_FILE=yes

 

Everything compiled fine. uBoot 16.01-rc2 kernel 4.4-rc4

 

to install, I had to remove  my [patched kernel 4.3.0 uboot 15.10] *.deb files { next-sunxi } to install "Armbian 5.0 kernel"  debs named { dev-sunxi } => file conflicts with the next/dev packages

 

Reboot, I ended with no lamobo-r1 tools "swconfig"... lost the labomo-r1.dts file after a hard reboot... bad karma

 

ok, I have my "emergency sdcard", the labomo-r1 is the heart of my fiber Internet + ethernet/fiber/wifi home network (YES! I have fiber between my bedroom and my main room)

 

I had to extract all [ kernel 4.3 uboot 15.10 ] *.deb to replace the files. I got lucky u-boot 16.01-rc2 did not complained of the trick at reboot.

 

then removed dev-sunxi debs and reapplied the next-sunxi... and my system is back

 

 

beta testing :-)          

[zador.blood.stained]

wildcat_paris

So, I didn't exactly understand, dev kernel didn't work for you?

[wildcat_paris]

wildcat_paris

So, I didn't exactly understand, dev kernel didn't work for you?

 

long story short...

 

 

moving from next-sunxi to dev-sunxi is missing cleaning of next-sunxi first OR just to keep deb package named as "next-sunxi" not  "dev-sunxi" (I guess keeping next-sunxi in the debian package name is easier. Because dev is a a bit ahead of next and not really something different)

 

Also for lamobo-r1 the tools in /usr/local/bin are missing (or have been relocated and network config at reboot just failed)

 

 

So yes, it failed, but the on going modifications of "Armbian tools" are very promising.

 

Kindly,

 

Guillaume

[zador.blood.stained]

I think it's more right to declare relationship between kernel packages so they cannot be installed at the same time.

If I understand this right, it should be something like this in the kernel package control file:

Provides: linux-image
Conflicts: linux-image
Replaces: linux-image

Right now it doesn't have "Conflicts" and "Replaces" fields.

➜  ~  % apt-cache show linux-image-next-sunxi
Package: linux-image-next-sunxi
Status: install ok installed
Priority: optional
Section: kernel
Installed-Size: 14180
Maintainer: Igor Pecovnik <****l.com>
Architecture: armhf
Source: linux-4.3.0-sunxi
Version: 4.6
Provides: linux-image, linux-image-2.6, linux-modules-4.3.0-sunxi
Suggests: linux-firmware-image-next-sunxi
Description: Linux kernel, version 4.3.0-sunxi
 This package contains the Linux kernel, modules and corresponding other
 files, version: 4.3.0-sunxi.
Description-md5: 117a3231b09bf7ccdf0ae5d4ec6b8c5f
Homepage: http://www.kernel.org/

Edit:

but the on going modifications of "Armbian tools" are very promising.

This is only the beginning  I mean, beginning of improvement, not of introducing tons of bugs, I hope 

[wildcat_paris]

 

Igor wrote:

This lamobo fix was "lost in the crash" together with others small fixes yet to be rediscovered

 

@Igor, no binary tools like swconfig, so yes, confirmed

[zador.blood.stained]

@wildcat_paris
I'm kind of lost, are you talking about sunxi tools (bin2fex, fex2bin, nand-part, meminfo) or swconfig only?
swconfig is supposed to be extracted from archive, it's precompiled, as far as I see.
I tested new changes in my branch today before pushing it, and sunxi tools (or some other tools, I didn't bother to check) failed to compile, but I managed to fix it by installing libc6-dev-armhf-cross
Please check if you have it installed on your build host and install it if it's missing.

Edit: Since I made some changes to installing packages on host system and I was testing on minimal Trusty running in a container, I'm not sure if those build errors weren't caused by me.

Edit 2: Nevermind, checked it on "normal" desktop Trusty, it's installed by default.

Edit 3: Still it was installed by "Recommends" relationship, not by "Depends" 

[wildcat_paris]

@zador.blood.stained

 

 

all of the binary tools in /usr/local/bin/ brought by armbian including bin2fex, fex2bin, nand-part, swconfig

gr@server1404:~$ sudo apt-get install libc6-dev-armhf-cross
[...]
libc6-dev-armhf-cross is already the newest version.

 

I will be testing the "next" tool instead of "dev", keep you in touch. I will extract the *.deb to check the files before applying the packages.

[zador.blood.stained]

Then it must be packaging error, not compiling. I didn't touch it (yet) and so didn't test it. Maybe will check later.

[wildcat_paris]

btw I would have suggested to add in lib/patching.sh in advanced_patch ()

mkdir -p "$SRC/userpatches/$dest/$family/$device"
mkdir -p "$SRC/userpatches/$dest/$family"

so next-time the user can fill the proper directory with the patches without having to add logs and read patching.sh

[zador.blood.stained]

I already did something like that, prepare_host function duplicates directory structure from lib/patch to userpatches

[wildcat_paris]

btw I would suggest to add in main.sh

 

 

        if [ "$USEALLCORES" = "yes" ]; then
                CTHREADS="-j$(($CPUS + $CPUS/2))";
        else
                if [ "$USEALLCORES" = "single" ]; then
                        CTHREADS="-j1";
                else
                        CTHREADS="-j${CPUS}";
                fi
        fi

because when there is an compil error, you don't need to look upper in the logs to find the problem

and with ccache, it is as fast as possible

 

 

edit:

also in configuration.sh, else it cannot be used by

[ -z $USEALLCORES ] && USEALLCORES="yes" # Use all CPU cores for compiling

EDIT 2:

I mean a quick switch between ALLCORES and a single core is useful to catch compile errorr faster. CCACHE is here to help with single core compile... if possible

[wildcat_paris]

I already did something like that, prepare_host function duplicates directory structure from lib/patch to userpatches

 

I my mind, userpatches is to avoid messing with lib/patch (minimizing changes in lib directory, so being more "portable" / configurable, I don't like much git conflit when "git pull" the armbian lib (git saying "save your changes" or you cannot update)

[zador.blood.stained]

It doesn't duplicate files, it creates only empty directories.

In my mind it is one of the reasons too.

[wildcat_paris]

little suggestion (bash only):

trap '{ echo "Hey, you pressed Ctrl-C.  Time to quit." ; exit 1; }' INT

 would be useful as CTRL-C can break one script and the others can keep going on.

[zador.blood.stained]

Yes, thanks. I already have this on my todo list for unmounting loop devices after debootstrap process.

[wildcat_paris]

I already did something like that, prepare_host function duplicates directory structure from lib/patch to userpatches

 

one more detail @zador.blood.stained

 

you think like one of the "Armbian lib" tool writer/dev

 

I think like an end user wishing to patch  the kernel as quickly as possible, messing as little as possible with "lib" *.sh and folders, getting my kernel debian files to update my device.

 

So having as little as possible to understand the internal of the "Armbian lib"

 

Thanks for you help