[can wait][generic] patch in /scripts/something/ for compile.sh to solve some issues

[wildcat_paris]

Hello,

 

I understand plainly zador.blood.stain for keeping the one-liner version of compile.sh

 

I am dumb guy with low memory in brain and I prefer a config file per board (I am using one file for XU4, one for OPi+2E, one for Lamobo-R1, one generic file for playing) instead of a file with plenty of one liners for compile.sh

 

But I have *just* used the plain original (updated) version of Armbian lib and I felt coming back to the Age of ICE (everything is root and sudo, no possible proper CTRL-C of compile.sh to restore all files with user id&group $HOME/.ccache is root, impossible to use RSA keys for forked version of lib (to push/pull) -- unless using the RSA key with root user... etc.)

 

for the backlog of my "failed" pull requests

https://github.com/igorpecovnik/lib/pull/387

 

I would propose to add a script in /scripts/patchToUseConfigFiles/ (I will find a better directory name or any directory in mind)

which would:

- patch the user's local compile.sh (I will maintain the patch for compile.sh)

- install one config file for general use and one example for a specific board (to use ./compile.sh or ./compile xu4 or any board you have in mind)

- create and maintain the appropriate documentation how to use the patched compile.sh

 

so well it will take a directory in "lib" + 4 files (+a file in lib.docs)

 

is it something possible? The Age of ICE is very cold, even if I am fan of Vlad, Trump & Angela. Democracy is overrated and a pure theoretical concept.

(for fans of the new UK PM, read this or this because out of EU, UK people won't even get the very limited social boundaries from the EU regulations from international agreements. EU is dangerous but getting out of EU is even more scary)

[borsti67]

I would like to see that compile.sh does not need to be tampered with by the user.

Especially for variables: These should be set within the scripts ONLY when not defined yet, so one can easily customize.

 

So compile.sh should contain something like this at the very beginning:

[[ -f user_compile.conf ]] && source user_compile.conf

(user_compile.conf would contain variable definitions like "SDSIZE=6000" - then it should not be necessary to change anything in compile.sh manually?)

 

In scripts like configuration.sh the lines should read

[[ -z $SDSIZE ]] && SDSIZE="4000" # SD image size in MB

This way the user has full control over script parameters without needing to patch files.

[wildcat_paris]

Especially for variables: These should be set within the scripts ONLY when not defined yet, so one can easily customize.

 

So compile.sh should contain something like this at the very beginning:

[[ -f user_compile.conf ]] && source user_compile.conf

 

the *very beginning of compile.sh* is not possible as a user may need to override the variables listed at the beginning of compile.sh (or my understanding of bash is far too limited, the same as a goldfish)

KERNEL_ONLY=""						# leave empty to select each time, set to "yes" or "no" to skip dialog prompt
KERNEL_CONFIGURE="no"					# want to change my default configuration
CLEAN_LEVEL="make,debs"					# comma-separated list of clean targets: "make" = make clean for selected kernel and u-boot,
							# "debs" = delete packages in "./output/debs" for current branch and family,
							# "alldebs" - delete all packages in "./output/debs", "images" = delete "./output/images",
							# "cache" = delete "./output/cache", "sources" = delete "./sources"
# user
DEST_LANG="en_US.UTF-8"					# sl_SI.UTF-8, en_US.UTF-8
CONSOLE_CHAR="UTF-8"

# advanced
KERNEL_KEEP_CONFIG="no"					# overwrite kernel config before compilation
EXTERNAL="yes"						# build and install extra applications and drivers
DEBUG_MODE="no"					# wait that you make changes to uboot and kernel source and creates patches
FORCE_CHECKOUT="yes"					# ignore manual changes to source
BUILD_ALL="no"						# cycle through available boards and make images or kernel/u-boot packages.
							# set KERNEL_ONLY to "yes" or "no" to build all kernels/all images

# build script version to use
LIB_TAG=""	

But as you point out this very well. probably yes, thoses variables would need to be put in an external config file entirely.

 

so it is easy to have a common user_compile.conf

and files named user_compile_xxx.conf (xxx = config per board)

 

so we have the best of one-liner version (variables on the command line) of compile.sh and config file based (user is free to select what's best for its use)

[borsti67]

the *very beginning of compile.sh* is not possible as a user may need to override the variables listed at the beginning of compile.she based (user is free to select what's best for its use)

It IS possible because consequently also THESE variables would be tested if already defined before.

So if you'd like to set "BUILD_ALL=yes" you would do so in user_compile.conf, not in compile.sh. Of course also compile.sh needs to be changed the same way as I described in the sample above (this applies to ALL Armbian scripts where a variable is defined).

Another advantage would be that ANY variable could be set either in the config or as a parameter, even those which are currently not supported (as DISKSIZE, for example).

 

Regarding your proposal

and files named user_compile_xxx.conf (xxx = config per board)

I would recommend not to do this by default, because the boards (or other reasons for having multiple config files) is changing every now and then.

But as my proposed script would be executed during runtime, you could easily add your additional scripts like this:

# user_compile.config:
DISKSIZE="5000"
BOARD="myownboard"

# ... more variables ...

[[ -f user_${BOARD}.config ]] && source user_${BOARD}.config

# ... anything else ...

in this example you set the board name to "myownboard" (you could do this also via commandline) and the script checks if another script named "user_myownboard.config" file exists - if so, it will be sourced. Within that one you would set up your board-specific variables and patches.

[wildcat_paris]

@borsti67

 

in this example you set the board name to "myownboard" (you could do this also via commandline) and the script checks if another script named "user_myownboard.config" file exists - if so, it will be sourced. Within that one you would set up your board-specific variables and patches.

 

I haven't thought of this point but it is interesting.

 

I had in mind the

"./compile.sh" using default user_compile.config

"./compile.sh  xu4" using user_compile_xu4.config

 

Zador.blood.stain prefers

# compile.sh KERNEL_ONLY="yes" DEST_LANG="ru_RU.UTF-8" BOARD="OPi+2E" SYSTEM="jessie"

or maybe the opposite

# KERNEL_ONLY="yes" DEST_LANG="ru_RU.UTF-8" BOARD="OPi+2E" SYSTEM="jessie" compile.sh

 

I don't like because I have no memory in my goldfish brain nor time to type or copy/paste

(and Zador.blood.stain is clearly 10x more skilled than me in bash and many other areas)

 

@All

the idea IS not to replace the existing framework to build all the boards (which is great)

BUT it is a counterpart of the "userpatches" concept where you can set-up files/patch/config for Lamobo-R1/XU4/OPi+2E

example:

- I need to build kernel only for Lamobo-R1 (I have custom patches&config for the kernel)

- I need to build full image for OPi2+2E with all standard settings

- I need to build full image for XU4 but I need to set specific details (boot partition size, use newer settings like UUID etc. )

 

And last but not least:

- limit the use of root user (and then use RSA key for forked version of "lib" in github)

- CTRL-C to stop cleanly compile.sh (trap CTRL-C and launch clean-up before ending the script)

- clean the environment (chown all the related files of "lib" including $HOME/.ccache because sudo inside compile.sh makes all .ccache files "root" then ccache can get broken for other uses, so I don't want this $HOME/.ccache mess ever ever again)

[borsti67]

I had in mind the

"./compile.sh" using default user_compile.config

"./compile.sh  xu4" using user_compile_xu4.config

ah, that's nice, too! 

 

And last but not least:

- CTRL-C to stop cleanly compile.sh (trap CTRL-C and launch clean-up before ending the script)

In fact I noticed that my userscripts/overlay folder got dumped after I interrupted compilation during "customize_image.sh".

Fortunately I was able to recover from my own git, but I didn't expect that at all...

[zador.blood.stained]

Zador.blood.stain prefers

# compile.sh KERNEL_ONLY="yes" DEST_LANG="ru_RU.UTF-8" BOARD="OPi+2E" SYSTEM="jessie"

or maybe the opposite

# KERNEL_ONLY="yes" DEST_LANG="ru_RU.UTF-8" BOARD="OPi+2E" SYSTEM="jessie" compile.sh

I don't have to remember anything, "smart" autocomplete from history in zsh does it for me automatically  , and persistent options are defined in userpatches/lib.config

 

If you want to keep config files in $SRC directory, I can add something like this at the end (I mean near the end) of compile.sh

[[ -n $1 && -f $SRC/config-$1.conf ]] && source $SRC/config-$1.conf

so you can use something like 

./compile.sh lamobo-r1
#or
./compile.sh testconfig

In fact I noticed that my userscripts/overlay folder got dumped after I interrupted compilation during "customize_image.sh".

Fortunately I was able to recover from my own git, but I didn't expect that at all...

It shouldn't do that, I'll have to recheck things

[borsti67]

It shouldn't do that, I'll have to recheck things

If it helps: It's not in the log files, but on screen I saw this message:

 

rm: cannot remove '/{...}/output/cache/sdcard/tmp/overlay': Device or resource busy

[zador.blood.stained]

If it helps: It's not in the log files, but on screen I saw this message:

rm: cannot remove '/{...}/output/cache/sdcard/tmp/overlay': Device or resource busy

Yes, it helps - so unmounting failed because some process was still using this directory, and then rm deleted it. I think it can be solved by mounting overlay directory read-only. Edit: and checking if unmount was successful before removing the directory.

[zador.blood.stained]

$HOME/.ccache is root, 

Yes, this can be a problem, but you can set

umask=002

in ~/.ccache/ccache.conf and set ~/.ccache owner group i.e.to "sudo" to fix this. Obviously you will need to clear your cache or change permissions manually to apply these changes to already existing files and directories in ccache.

 

You can check "SHARING A CACHE" section of ccache man page: http://manpages.ubuntu.com/manpages/xenial/man1/ccache.1.html

 

Edit: I forgot, you also need to set SGID bit on ~/.ccache directory ("chmod g+s ~/.ccache")

[borsti67]

I have done some changes to compile.sh:

• Variables are now defined ONLY if not passed otherwise, so they can be set elsewhere without the need to make changes in this file
• (not necessary, just for readability of code) I have added the escape-sequences for the ANSI-coloring as variables, so they can be used anywhere later. This is just in case someone prefers other colors for output and doesn't want to change each line individually. At the moment it only works within compile.sh as I didn't touch other scripts yet, but I can easily do if I should!
• in case a file named "armbian_user.conf" is found in the same folder, it will be sourced just before Armbian self-update takes place. Within this file (which is absolutely not necessary to have) the user can set overrides to variables, call other scripts and so on.
• at the end of the script, all files in $SRC will be owned by the current user (instead of root)

Please review the attached patch and apply if ok!

(in case something is missing/wrong also please tell so I can modify and create another patch)

 

[PS: I needed to change the extension to .txt to upload, .patch is not allowed...?]

 

compile_sh.patch.txt

[zador.blood.stained]

• Variables are now defined ONLY if not passed otherwise, so they can be set elsewhere without the need to make changes in this file

Yes, I know about this syntax, but IMO it's harder to read, this block should be as simpel as possible

 

• in case a file named "armbian_user.conf" is found in the same folder, it will be sourced just before Armbian self-update takes place. Within this file (which is absolutely not necessary to have) the user can set overrides to variables, call other scripts and so on.

This solution is more universal because it allows easy switching between several config files

 

• at the end of the script, all files in $SRC will be owned by the current user (instead of root)

And if you run this script while logged in as root user or from "sudo su -" shell ($SUDO_USER is not defined)?

[borsti67]

Yes, I know about this syntax, but IMO it's harder to read, this block should be as simpel as possible

 

This solution is more universal because it allows easy switching between several config files

 

And if you run this script while logged in as root user or from "sudo su -" shell ($SUDO_USER is not defined)?

 

Hm, the forum doesn't support nested quotes?

 

• may be it's a little harder to read, but as the user doesn't have to change anything here I would not consider this a problem. The name of the variable and its standard value are visible and well documented in the comments, so if the user wants to have them changed he would do it in the separate conf (in standard syntax)...
• Oops, I have overlooked this one. This is better indeed, but it will work only as long as we don't expect more than 1 parameter, correct?
• good point, I never tried. $SUDO_USER was the only reference to the "real" logged-in user I was able to find. Do you know any better method? As long as it's unsafe this way, I'll put it in [[ -z ]]...

[zador.blood.stained]

• Oops, I have overlooked this one. This is better indeed, but it will work only as long as we don't expect more than 1 parameter, correct?

No, this will work in any case, if first parameter is not name of the file but KEY=value pair (i.e. BOARD=cubietruck), it just won't do anything at this stage.

 

Anyway, I pushed some changes: this adds support for additional config files and adds a hook to run after building, so you can put something like this in userpatches/lib.config if you want to:

run_after_build()
{
 [[ -n $SUDO_USER ]] && chown -R $SUDO_USER:$SUDO_USER $SRC
}

[borsti67]

forgot patch

compile_sh.patch.txt

[borsti67]

 

No, this will work in any case, if first parameter is not name of the file but KEY=value pair (i.e. BOARD=cubietruck), it just won't do anything at this stage.

 

Anyway, I pushed some changes: this adds support for additional config files and adds a hook to run after building, so you can put something like this in userpatches/lib.config if you want to:

run_after_build()
{
 [[ -n $SUDO_USER ]] && chown -R $SUDO_USER:$SUDO_USER $SRC
}

Thanks for the hook, I think this will do very good!

 

But regarding the parameters and the inclusion script I have still some questions:

1. you placed the script AFTER Armbian self-update. This means one will not be able to provide a LIB_TAG by the additional script.
2. maybe I'm not experienced enough in bash scripting, but to my knowledge parameters don't have priority over code. This means if compile.sh contains

   LIB_TAG=""
   

   this won't work:

   LIB_TAG="v1.0" ./compile.sh
   

   ...or do I miss something? That's the main reason why I proposed the slightly more complex syntax in the patch.

[zador.blood.stained]

I know that this won't work for LIB_TAG and BUILD_ALL parameters, but BUILD_ALL is used mainly by Igor to produce images, and I don't see any use cases for dynamically changing LIB_TAG. In case you want to use your own fork of build script, you just create $SRC/.ignore_changes and use "wrapper" script that updates the code if needed and launches compile.sh afterwards.

[borsti67]

Currently I'm working only with one board, so for me it's not an issue. The idea is in case the Armbian script gets broken for a specific board only (I've had this some time ago for a BBB) you would be able to set LIB_TAG to the latest working one in the config-<thisboard>.conf file.

At the moment for me the actual master works fine, so no need here - I justed wanted to bring it to your attention in case you didn't consider this yet.

I just have changed my lib.conf to define the hook and now it's compiling.

 

Regarding my 2nd remark, there LIB_TAG was just an example, you also cannot pass BOARD and such as parameters, therefore my question. But the question is somewhat obsolete now as all of these can be set in the dedicated conf...

 

Anyway thanks for the really fast reply (and action!)

 

Do you dislike the idea of having the color definitions as variables instead of escape sequences at all or should I provide this once more later? But then over all scripts which use colored output...?

[zador.blood.stained]

Regarding my 2nd remark, there LIB_TAG was just an example, you also cannot pass BOARD and such as parameters, therefore my question. But the question is somewhat obsolete now as all of these can be set in the dedicated conf...

You can't pass them as environment variables (BOARD=cubietruck ./compile.sh), but you can pass command-line arguments (./compile.sh BOARD=cubietruck), it is handled here in main.sh

 

Do you dislike the idea of having the color definitions as variables instead of escape sequences at all or should I provide this once more later? But then over all scripts which use colored output...?

"Keep it simple" and "function over fashion". This is a build script and not a christmas tree  , I don't want to spend time and resources on visual stuff while there are issues to work on. Especially in compile.sh where you need to define colors in place - I'm not against customizing diaplay_alert function later.

[borsti67]

You can't pass them as environment variables (BOARD=cubietruck ./compile.sh), but you can pass command-line arguments (./compile.sh BOARD=cubietruck), it is handled here in main.sh

/me = dumb.

 

"Keep it simple" and "function over fashion". This is a build script and not a christmas tree  , I don't want to spend time and resources on visual stuff while there are issues to work on. Especially in compile.sh where you need to define colors in place - I'm not against customizing diaplay_alert function later.

I've had that already in another project, so it was/is just c'n'p for the variable definitions (which should be set in compile.sh IMHO), the rest is search-and-replace.

 

Compilation finished in the meantime: Inclusion of the conf worked as expected, and also the after-build hook!

[wildcat_paris]

@zador.blood.stain @borsti67

 

Guys, you rock!