armbian-next development

[going]

1 час назад, balbes150 сказал:

15 часов назад, rpardini сказал:

What I automated is the boring part, so the human only has to use the eye at the end of it. At least for trivial rebases.

It is part of the Python patching system, but only used if invoked manually by a developer.

Judging by what I got as a result of building the kernel and studying the log of the patch process, this is not the case. the system interferes with patches and changes them itself, without my permission. the build system should not interfere with patches and break their work. patch processing can be performed exclusively in interactive mode, as a separate function. during the build process, patches should not change in any way.

Which of you two should I believe?

 

Here I confirm that the script that analyzes patches, separates them and tries to apply them in parts, as well as the script

that will create patches or commit to the code in the source repository should never be called during work the build system itself.
Such scripts should be located in the tools folder, they should be called as independent and self-sufficient executable files.

[atone]

Image built with Armbian NEXT® boots on BPi M5! Most important things seem to work correctly (or at least on par with master). Well done!

 

Just to give another perspective: I'm a .NET developer and Windows user. I first heard of Armbian on 2022-10-21 (and downloaded and flashed an official image), and built my first custom image from master 12 days ago. So NEXT cannot be so bad

[balbes150]

37 минут назад, going сказал:

a source package

If I understood your thought correctly.

Creating source packages doesn't make sense. The build system itself is "the sources for all packages, the kernel, the loader, and so on." Therefore, it makes no sense to duplicate this work. Our task is to prepare the source code (get the right version of the source code, apply patches, form a configuration, etc.) and assemble a ready-made binary package. If  use a classic scheme with a source package, it will require the user to come to the same set of complex steps (installing the necessary packages for build, configuring them correctly, etc.), which will kill the main trump card of the Armbioan assembly system - ease of use for the average user. I performed a few simple steps and got the result (my kernel, bootloader, etc.).

[balbes150]

19 минут назад, going сказал:

Which of you two should I believe?

You don't believe anyone. Only your personal experience and your own verification. IMHO during assembly, patches should not be processed in any way at all, only "dumb application as is".

 

[Igor]

2 hours ago, balbes150 said:

Sorry, I can't get into jira, if it suits me, I can create a question in GIT.

I forgot  sorry. Then Git.
 

24 minutes ago, balbes150 said:

Creating source packages doesn't make sense.

For kernel sources it makes - as we make changes to it. Main distributions changes kernel a lot less then we do and they all provide source packages of that modified kernel. If we are talking about https://github.com/armbian/build/tree/master/packages/extras-buildpkgs then this is optional. I don't think anyone would seek source package for those few things we are building there.

[going]

22 минуты назад, balbes150 сказал:

Creating source packages doesn't make sense. The build system itself is "the sources for all packages, the kernel, the loader, and so on." Therefore, it makes no sense to duplicate this work. Our task is to prepare the source code (get the right version of the source code, apply patches, form a configuration, etc.) and assemble a ready-made binary package. If  use a classic scheme with a source package, it will require the user to come to the same set of complex steps (installing the necessary packages for build, configuring them correctly, etc.), which will kill the main trump card of the Armbioan assembly system - ease of use for the average user. I performed a few simple steps and got the result (my kernel, bootloader, etc.).

 

Thank you Oleg. This is the first firm opinion on whether we should create source packages or not.

I want to ask everyone who reads this post to write a short message:
1) I don't need the source packages.
2) I need the source package. I use them.

It depends on whether I will try to implement it.

[rpardini]

1 hour ago, going said:

1) I don't need the source packages.

 

Definitely not for Kernel.

Consider a new user, that wants to patch something really simple, for example, add a missing LED to a board via DT. (that's how I got started.)

What is easier / more productive: download kernel-source-xxx package, make it work somehow, see a complete source without any history/commits, not understand why things are done that way... etc.

Or: go to armbian/build repo, see history & documentation & happy friendly people collaborating & a tool that guides them to produce a new build & a good place to PR his small patch so that it lives later....

[rpardini]

3 hours ago, balbes150 said:

It suits me (joke).

 

It suits a lot of people -- not only you.

I am (seriously) taking in the sensible parts of your feedback and implementing. (it would be easier if you were not also insulting, I get distracted by the insults when parsing your text).

 

I already got most of the "junk" out ("junkless", for the "Oleg" scenario: fully native arm64 assembly, no Docker) while still keeping the capacity to have a "junkfull" Docker image with everything (without having to sync 2 pieces). It's _a lot_ of if's.

Next, (pun!), I will handle the logging of the patch apply to output what we had before (just patch name, OK or not, colors), and hide the details in the logfile. (eg; rejects, etc).

Then reduce the logging to file of things we don't need & fix the formatting of the single .txt log as has already been discussed a few pages back.

Also https://armbian.atlassian.net/browse/AR-1451 so logs actually show up on screen.

 

So it's just more code / more if's -- and more scenarios to test.

 

The whole reason I wanted to avoid _adding more code_ is that the more code I add, the more code to review, and the longer it will take to merge.

But I'd rather have everyone onboard, or, if that's not possible, have most people onboard, even if takes a bit longer.

 

[schwar3kat]

@rpardini When a file has been changed in the local branch being built then master armbian-build provides a useful warning about not being able to update with a dialog to abort, continue or view changes before building.  

Personally I find this function to be a very useful reminder and would like to see it in armbian-next.

[schwar3kat]

@rpardini armbian-next - repeat build string at the end of the build process leaves out build options, BOARD=...
and NO_APT_CACHER=... (although armbian build master also leaves out NO_APT_CACHER ) and possibly others.

I would probably expect that all defined options would be included?

Original test build string (with desktop and base configuration selected through dialog):

./compile.sh BOARD=pine64 BRANCH=current RELEASE=jammy BUILD_MINIMAL=no BUILD_DESKTOP=yes KERNEL_CONFIGURE=no CREATE_PATCHES=no NO_APT_CACHER=yes

 

Repeat build string returned at the end of the build process ("BOARD=pine64" and "NO_APT_CACHER=yes" are missing):
[✨] Repeat Build Options [ ./compile.sh   BRANCH=current RELEASE=jammy BUILD_MINIMAL=no BUILD_DESKTOP=yes KERNEL_ONLY=no KERNEL_CONFIGURE=no DESKTOP_ENVIRONMENT=xfce DESKTOP_ENVIRONMENT_CONFIG_NAME=config_base DESKTOP_APPGROUPS_SELECTED="3dsupport browsers chat desktop_tools editors email internet multimedia office programming remote_desktop" COMPRESS_OUTPUTIMAGE=sha,gpg,img ]

I would also not expect COMPRESS_OUTPUTIMAGE=sha,gpg,img to be included (although no harm in including it).

[schwar3kat]

@rpardini Regarding patches encoded big5 with Chinese in the code comments and in tables and etc.   Are you considering  a way to accept these.
These don't work with current armbian-next, and direct conversion to UTF-8 breaks the one that I tried for master and next.

Example: the huge wifi-4002-add-realtek-8723cs.patch (kernel rockchip64 has six identical and there are others with variations)

I have created a version of the rockchip64 patch with translated comments and deleted Chinese where this was just a Chinese translation in the same quoted string as the English.  In table like structures where it wasn't clear if the string could be left out, I replaced with question marks. 

This patch builds on both master and next, but I have no way of testing it on hardware.

The problem here might be that the Chinese in the table like structures might have a purpose (print or error strings come to mind).

Maybe detect big5/foreign-chars and use the old method or something similar?
 

[rpardini]

2 hours ago, schwar3kat said:

The problem here might be that the Chinese in the table like structures might have a purpose (print or error strings come to mind).

 

Yeah, parsing the patches includes being able to decode them.

Patches should match the encoding of the target patched source; In this case, it's UTF-8.

In my view, those patches are mangled (not just "in big5"): the original encoding was lost due to copy/pasting, and they're already (in master) putting frakked up characters in the source code.

So it can't get worse, but now I warn you about the error.

 

 

[rpardini]

3 hours ago, rpardini said:

So it can't get worse

 

A lie. I just found a bug where I mangled perfectly valid UTF-8, during mbox parsing, for no good reason. Fixing...

 

 

[schwar3kat]

On 1/16/2023 at 2:35 AM, rpardini said:

but now I warn you about the error

 

I'm guessing that ">" on the end of the line in the logs means that the patch had no issues and applied correctly?

[rpardini]

18 hours ago, schwar3kat said:

I'm guessing that ">" on the end of the line in the logs means that the patch had no issues and applied correctly?

 

Err... no, current patching log is a terrible mess and very hard to tell wtf is going on.

I am making a complete review of the patching logging. Objective is to produce a simple "table-like" output like we had before, on screen; then have details/rejects/etc on the logfile when investigating.

Sorry for the bad experience, it's gonna be much better "real soon now".

PS:  For now, the least-horrible way to see the Patching info is looking at the produced .md file in output/logs. Render it to UTF-8 HTML for full glory.

[Igor]

Anyone reading this volunteer in helping us getting together some article we can ship to some a bit serious magazines. Equally useful if someone knows someone that this might be interesting topic to cover?

Technical article about changes, improvements and benefits

https://armbian.atlassian.net/browse/AR-1497

[schwar3kat]

@rpardini I'm not sure how far you have progressed cached stuff on next.  Hook function changes?
 

A change inside a hook function in the board config file is not causing the bsp change to be detected.
The hook is post_family_tweaks_bsp.

Example: orangepi-r1plus-lts.conf.  I just changed some of the values in the two files that are written into the build and the changes don't make it into the build.  The bsp cache is used.

[TRS-80]

On 1/17/2023 at 7:58 PM, Igor said:

Anyone reading this volunteer in helping us getting together some article we can ship to some a bit serious magazines.

 

I just saw this.  Earlier I had read this, but I don't think those are the same.

 

Therefore, I can try and work on this.  Did anyone start anything yet?

 

@rpardini, as you obviously know the work best, could you be so kind as to indulge me a summary of the main features?  Not to discount your work, but for the average reader I think a 20 (50? more?) item long commit log does not make for the most interesting article.

 

Maybe we can hash something out in lanefu's HedgeDoc instance again, like that one time?  I think that worked pretty well.

[rpardini]

On 2/5/2023 at 3:03 AM, TRS-80 said:

 

@rpardini, as you obviously know the work best, could you be so kind as to indulge me a summary of the main features?  Not to discount your work, but for the average reader I think a 20 (50? more?) item long commit log does not make for the most interesting article.

 

Maybe we can hash something out in lanefu's HedgeDoc instance again, like that one time?  I think that worked pretty well.

 

Yep, good idea. Commit log is at 400+ commits... I've slides (in Markdown/HedgeDoc) on my own instance, that I've copied to Lane's at https://docs.lane-fu.com/dAGOaCTLQ0SgPnQsHz6LWw?both -- it's in slide style, first few ones are outdated, but can serve as base: https://docs.lane-fu.com/dAGOaCTLQ0SgPnQsHz6LWw?both -- also, if you have patience to watch the developer meeting videos, most secrets will be revealed. I was hoping to do one last consolidated/updated presentation before the switch, but time is pressing...

[rpardini]

On 2/3/2023 at 9:38 AM, schwar3kat said:

 

Example: orangepi-r1plus-lts.conf.  I just changed some of the values in the two files that are written into the build and the changes don't make it into the build.  The bsp cache is used.

 

 

Yep, not only are changes to hooks ignored, but changes to core code are _also_ ignored. For now the only way to get around it is to remove the .deb in question from output/debs (or CLEAN_LEVEL=debs, although that cleans too much).

Meanwhile I'm working on almost exactly the same subject but for packages like kernel and u-boot, and bsps will come after that. Essentially I'll be hashing the bash code of drivers/patches/config + core+hooks and including that into the .deb Version: field.

[schwar3kat]

13 hours ago, rpardini said:

and bsps will come after that.

Excellent...  just making sure that the issue is on your radar. 

I noticed a distcc.sh in the compilation folder.  I'm guessing/hoping that this will mean easy distributed compilation.

[rpardini]

22 hours ago, schwar3kat said:

I noticed a distcc.sh in the compilation folder.  I'm guessing/hoping that this will mean easy distributed compilation.

 

Yep, I worked on that a few months ago. We've something like `./compile.sh distccd` which starts a distcc daemon, complete with zeroconf etc, using the same toolchains Armbian would.

Something else (forget what) enables it to be used on a controller node. You do need a bigger controller machine, say 8-cores, to completely max out some two/three 4-core distccd's, due to the high preprocessing load.

The network between the machines has to be very good, too (gigabit+). All in all is a "fun" way to use all those SBCs laying around doing nothing, but don't expect something miraculous...

One day I'll finish / document this...

 

[schwar3kat]

I was just looking into trying to set this up manually a couple of days ago, but life got in the way and I didn't take it very far.
Just a fun idea to try out and maybe get a little more speed when stuff isn't cached.

[schwar3kat]

@rpardini

My apologies for not testing much on next/main the last two weeks.  (13+ hour workdays with two hour commute is a bit rough).

New main branch.   Error on first run.  The build works on second run.
Running on Linux Mint Vera (Jammy) without docker:

 

$ git clone --depth 1 https://github.com/armbian/build
Cloning into 'build'...
remote: Enumerating objects: 8900, done.
remote: Counting objects: 100% (8900/8900), done.
remote: Compressing objects: 100% (5308/5308), done.
remote: Total 8900 (delta 3651), reused 7386 (delta 3146), pack-reused 0
Receiving objects: 100% (8900/8900), 83.75 MiB | 5.65 MiB/s, done.
Resolving deltas: 100% (3651/3651), done.
Updating files: 100% (12015/12015), done.

 

$ cd build

$ ./compile.sh BOARD=orangepi-r1plus-lts BRANCH=current RELEASE=bullseye BUILD_MINIMAL=no BUILD_DESKTOP=no KERNEL_ONLY=no KERNEL_CONFIGURE=no  CREATE_PATCHES=no COMPRESS_OUTPUTIMAGE=img

......

[\U0001f331] Getting ORAS manifest [ ORAS manifest from ghcr.io/armbian/cache-uboot/uboot-orangepi-r1plus-lts-current:2022.07-Se092-Pbb0c-B680b ]
[\U0001f331] Downloading required [ ORAS tooling ]
parse error: Invalid numeric literal at line 1, column 11
[\U0001f4a5] Error  4 occurred in SUBSHELL [ SUBSHELL at /mnt/data/build-armbian/armbian-main/build/lib/functions/general/oci-oras.sh:141 ]
[\U0001f4a5] Error 4 occurred in main shell [ at /mnt/data/build-armbian/armbian-main/build/lib/functions/general/oci-oras.sh:141
     oras_get_artifact_manifest() --> lib/functions/general/oci-oras.sh:141
   is_artifact_available_in_remote_cache() --> lib/functions/artifacts/artifacts-obtain.sh:279
   artifact_uboot_is_available_in_remote_cache() --> lib/functions/artifacts/artifact-uboot.sh:107
   artifact_is_available_in_remote_cache() --> lib/functions/artifacts/artifacts-obtain.sh:24
                do_with_logging() --> lib/functions/logging/section-logging.sh:72
       obtain_complete_artifact() --> lib/functions/artifacts/artifacts-obtain.sh:169
       build_artifact_for_image() --> lib/functions/artifacts/artifacts-obtain.sh:209
    main_default_build_packages() --> lib/functions/main/build-packages.sh:49
   full_build_packages_rootfs_and_image() --> lib/functions/main/default-build.sh:4
          do_with_default_build() --> lib/functions/main/default-build.sh:17
         cli_standard_build_run() --> lib/functions/cli/cli-build.sh:17
        armbian_cli_run_command() --> lib/functions/cli/utils-cli.sh:126
                 cli_entrypoint() --> lib/functions/cli/entrypoint.sh:164
                           main() --> compile.sh:52
 ]
[\U0001f4a5] Cleaning up [ please wait for cleanups to finish ]

 

Sorry vnc copy messed up the icons.
Logs attached.

log-build-fad6555a-d76c-4966-b374-91b1812b012a.log log-build-fad6555a-d76c-4966-b374-91b1812b012a.log.ans summary-build-fad6555a-d76c-4966-b374-91b1812b012a.md

[rpardini]

34 minutes ago, schwar3kat said:

New main branch.   Error on first run.  The build works on second run.

 

34 minutes ago, schwar3kat said:

Downloading required [ ORAS tooling ]

 

Thanks @schwar3kat ! This has been elusive, but its clear that when the tooling is 1st downloaded, the parsing fails.

2nd run the tooling is already there so doesnt fail. 

Will try to address!

[schwar3kat]

@rpardini

Next/main. Creating patches.
 

When I create a patch with armbian/build master (deprecated) using  CREATE_PATCHES=yes, I get a patch in the output/patch folder in mbox format.
When I create a patch with armbian/build main using  CREATE_PATCHES=yes ARTIFACT_IGNORE_CACHE=yes, I only get a diff file in the output/patch folder.

Is this intentional, a bug, or have I perhaps not set something up?  Can I get a mbox patch?

[rpardini]

5 hours ago, schwar3kat said:

BUILD_PATCHES=yes

 

Even after ruminating through the codebase for years, this is new to me. Will investigate..

[schwar3kat]

20 hours ago, rpardini said:

Will investigate..

Hold off investigating for now. 
This might be my fault.  I usually set global git username and user email.  I think that I may only have set them locally on the master branch folder.
I will retest when I get a chance.

Edit:  Tested.  Damn, I messed up.  Of course it can't do mbox without email address.
Apologies for wasting your time.

[ALIGMSTEN]

Hi,

 

Simply noting here, realise the busyness, right now, some config packages missing by the looks of it.

 

[🌱] Configuration prepared for BOARD build [ orangepizero2.conf ]
[✨] Repeat Build Options (early) [ ./compile.sh BOARD=orangepizero2 BRANCH=edge RELEASE=bullseye BUILD_MINIMAL=yes BUILD_DESKTOP=no KERNEL_ONLY=no KERNEL_CONFIGURE=no COMPRESS_OUTPUTIMAGE=sha,img ]

 

 

armbian-firstlogin timezone misgivings.

 

Please use this account for your daily work from now on.

/usr/lib/armbian/armbian-firstlogin: line 111: curl: command not found
/usr/lib/armbian/armbian-firstlogin: line 114: nmcli: command not found
/usr/lib/armbian/armbian-firstlogin: line 147: curl: command not found
/usr/lib/armbian/armbian-firstlogin: line 147: jq: command not found
Detected timezone:

Set user language based on your location? [Y/n] Y

 

Gladly attend to it, confirm file with pointers, so Im sure please!

[rpardini]

6 hours ago, schwar3kat said:

I usually set global git username and user email. 

 

This has been handled for other cases where `git format-patch` was used (kernel drivers cache, I think) so should also be fixed: it should use MAINTAINER and MAINTAINEREMAIL (default to Igor) if not set on git config. git config might not exist at all for example if using Docker -- so this is smth that should be handled anyway. Can you open a GitHub issue? I am losing track of stuff by splitting between forum and GH....