Initial easy setup proposal

[botfap]

You have a build framework that works pretty good, is it out of the question to have an extremely simple configuration framework? Maybe framework is more advanced than needed, probably just a script? 

 

It it doesn’t need to be part of the normal build process but could just simply inject a configuration profile into a completed rootfs, image or actual sd card

 

Just the basics:

Single network manager profile

hostname

default user and pass

serial console config 

log2ram status

remote login status (ssh, xrdp)

 

I don’t mind writing a simple v0.1 to get started 

 

 

 

[tkaiser]

19 hours ago, botfap said:

is it out of the question to have an extremely simple configuration framework? Maybe framework is more advanced than needed, probably just a script?

 

Absolutely support this. But IMO discussion should be moved out of this thread in an own thread over at https://forum.armbian.com/forum/12-armbian-build-framework/

 

@Igor can you please move last two posts to something like an 'Initial easy setup proposal' thread? I think a lot of users might benefit from a way to easily adjust the settings @botfap talked about prior to first boot.

[botfap]

Simple structure like this to start?

 

amrbian-image-config image.img function [options]  
    n|hostname
        name
    e|ethernet
        d|dhcp
        s|static
            ip mask [gw] [dns] [search]
    w|wifi
        d|dhcp
            ssid crypto key
        s|static
            ssid crypto key ip mask [gw] [dns] [search]
    u|user
        name password
    l|log2ram
        e|enable
        d|disable
    r|remote
        s|ssh
            e|enable
            d|disable
        r|rdp
            e|enable
            d|disable
    h|help
        print

 

[lanefu]

Do you all envision this as a bolt-on that just kind of runs last after image build around the user-patches phase, or more tightly integrated into the build process?

 

At a glance it seems more like a potential extension of armbian-config that could also be executed at anytime if needed a well.

[botfap]

I envisage it more as a troubleshooting tool at this point for headless boards but it could be used for a lot more I suppose

 

Initial version to test: https://github.com/botfap/armbian-image-config.git

 

Working for injecting hostname and wifi and ethernet network profiles. DHCP only for now but I will update it later

 

[lrrr]

I would do something similar to Armbian SDK and userpatches/customize-image.sh.

 

Add script to setup chroot (mounts,qemu-static,etc) using pre-built image.  For example, if userpatches/post-image-customize.sh exists, copy to chroot and exec or drop to chroot for manual configuration.  Then, cleanup.

 

It would be more versatile as you could exec binaries on the image, like dpkg-reconfigure, etc.  You could even use armbian-config but hardware detection would be based on host hardware, not the target.

[lanefu]

I envisage it more as a troubleshooting tool at this point for headless boards but it could be used for a lot more I suppose
 
Initial version to test: https://github.com/botfap/amrbian-image-config.git
 
Working for injecting hostname and wifi and ethernet network profiles. DHCP only for now but I will update it later
 

Ohhh! You want to edit image such as the official ones. I get it. No depencies on build. Pretty sweet actually

Sent from my SM-G950U using Tapatalk

[tkaiser]

29 minutes ago, lanefu said:

You want to edit image such as the official ones. I get it. No depencies on build. Pretty sweet actually

 

Well, I'm not entirely sure how high the percentage of Armbian users is running Linux on the machine they deal with the downloaded Armbian image. I would believe that's less than 0.1% since the majority uses something called Windows or macOS.

 

In fact those Linux users today already can simply loopmount the image and edit /boot/armbian_first_run.txt since /usr/lib/armbian/armbian-firstrun-config already exists and works.

 

I thought about tweaking this mechanism to support more options like @botfap is currently doing but making this feature accessible to those +99% of Armbian users not able to mount an ext4 partition. That would require adding a 32MB FAT32 partition priot to the rootfs partition.

 

On this partition labeled "Armbian" then the following will be:

• armbian-firstrun-config.txt (here options can be set that are then processed by /usr/lib/armbian/armbian-firstrun-confi, ideally the defaults are also contained but commented so users get an idea how an otherwise unmodified Armbian install will work)
• README.html (containing a brief overview how to access documentation, how to search the forum -- not just a stupid link to the forum but an explanation how to search the forum! -- and how to donate to the project)
• a 'documentation' directory (containing an offline version of https://docs.armbian.com)

The approach would be compatible with @botfap's work since when his tool is written in clean bash it can also run on macOS (I honestly don't care about Windows at all) and it just needs an additional mode where it generates armbian-firstrun-config.txt instead and his actual config tasks need to be merged into /usr/lib/armbian/armbian-firstrun-config

 

 

[botfap]

Update version with support for static IP addressing and template creation pushed. Can any one test pre loading a network (wifi or ethernet) config onto a clean image?

https://github.com/botfap/amrbian-image-config.git

 

@tkaiser I agree a fat partition with config and help is the best solution but putting it at the start is a pain and I suspect requires lots of framework changes

[tkaiser]

22 minutes ago, botfap said:

a fat partition with config and help is the best solution but putting it at the start is a pain and I suspect requires lots of framework changes

 

Why? FAT support is part of Armbian though deprecated. In the past there existed options to create /boot on a small FAT partition -- that's also the way DietPi used Armbian's build system for some time -- but we don't want /boot on a FAT partition since dpkg on FAT is a mess.

 

But asides that I would believe it's as easy as adding this small partition at image creation time.

[botfap]

10 hours ago, tkaiser said:

But asides that I would believe it's as easy as adding this small partition at image creation time. 

In which case its reasonably straightforward then. I have prototyped with a 32mb fat32 partition appended on to the end of a downloaded tinkerboard image. Using a modified version of the script to read in the changes at first boot. Seems to work fine independently of the Armbian first boot. I was trying to keep its mechanism separate from Armbians so as to not break things but I gather this is something you would like to see integrated into Armbian? Is that correct?

 

I think a fat32 partition is better on the end of the disk image instead of the beginning. My thought process being;  everybody expects and most people want the rootfs to be the first addressable partition on the board. I have some boards, esoteric admittedly, that will only boot a kernel from the first partition. I dont want my kernel and boot config files on a fat32 partition. The extra step of moving a 32mb volume in the rootfs resizing is minimal

[tkaiser]

2 hours ago, botfap said:

I was trying to keep its mechanism separate from Armbians so as to not break things but I gather this is something you would like to see integrated into Armbian? Is that correct?

 

Yes, I think this stuff should go into /usr/lib/armbian/armbian-firstrun-config

 

2 hours ago, botfap said:

I dont want my kernel and boot config files on a fat32 partition. The extra step of moving a 32mb volume in the rootfs resizing is minimal

 

We also don't want this stuff on a FAT partition and that's why the 'old' way to use a FAT partition for /boot is marked deprecated. Until now I didn't care about position of such a new FAT32 partition but thought it would be better to have at the beginning since I feared rootfs resizing hassles (and never looked into this in detail so if you say it's an easy task then I would prefer the FAT32 partition after rootfs and then moved as part of image resize on first boot)

[Igor]

3 hours ago, tkaiser said:

Yes, I think this stuff should go into /usr/lib/armbian/armbian-firstrun-config

 

Perhaps even make armbian-config command line friendly & more modular and get rid of this as well?

[tkaiser]

57 minutes ago, Igor said:

Perhaps even make armbian-config command line friendly & more modular and get rid of this as well?

 

I don't get it. You want to drop armbian-config? Or what is 'this' you're talking about? Or do you want to add CLI parameters to armbian-config so it could be called automatically and finish the tasks?

 

Anyway: I think what's needed is some rudimentary way to configure a few items that modify default Armbian behavior on first boot.

 

The current https://github.com/armbian/build/blob/master/packages/bsp/armbian_first_run.txt.template version is broken by design since config file would be on an ext4 partition so isn't accessible by vast majority of users anyway and the few 'tweaks' there are of not much use anyway (why should Ethernet and wireless be mutually exclusive?!). Only real interesting one there are wireless credentials.

 

IMO we should focus on @botfap's list for now.

[Igor]

36 minutes ago, tkaiser said:

Or do you want to add CLI parameters to armbian-config

 

Exactly.

 

36 minutes ago, tkaiser said:

is broken by design since config file would be on an ext4 partition so isn't accessible by vast majority

 

I know. 

If we are going to add FAT partition, we need to think over and test it profoundly. Perhaps at the end if 1st is not FAT already and remove it the rootfs resize process?

 

One is injections, the other is rework of current 1st run config which could also be done with armbian-config ...

[lrrr]

Here is a couple of more ways I was messing around with.

 

1) Create a static sized "binary" shell script on the image with a special header.  Find the header and inject script.  Simple working example.

# Something like this would be added to official SDK.  For testing add to userpatches/customize-image.sh

# add 25k binary script
ACI=/boot/customize-image.sh
printf "#ARMBIAN_CUSTOMIZE_IMAGE_PLACEHOLDER" > $ACI
OFFSET=$(du -b $ACI | awk {'print $1'})
dd if=/dev/zero of=$ACI bs=1 count=25k conv=notrunc seek=$OFFSET

# modify armbian-firstrun-config and add txt file to exec $ACI
sed -i 's/^do_firstrun_automated_user_configuration$/sh \/boot\/customize-image.sh\ndo_firstrun_automated_user_configuration\n/' /usr/lib/armbian/armbian-firstrun-config
touch /boot/armbian_first_run.txt

# Bash script run on users host.  Depends on dd and grep.

IMAGE=$1

# parse image and get offset
grep -abq "#ARMBIAN_CUSTOMIZE_IMAGE_PLACEHOLDER" $IMAGE
if [ $? -ne 0 ]; then
        echo "Unable to find placeholder in $IMAGE" && exit 1
fi

OFFSET=$(grep -Uabo "#ARMBIAN_CUSTOMIZE_IMAGE_PLACEHOLDER" $IMAGE | awk -F : {'print $1'})

# myscript.sh > /boot/customize-image.sh
cat <<-EOF > myscript.sh
#!/bin/sh -e
echo "Deleting /root/.not_logged_in_yet" >> /root/log.txt
rm -f /root/.not_logged_in_yet
echo "Resetting root password" >> /root/log.txt
chage -d 17790 root
exit 0
EOF

# inject script into image
printf "Injecting script into $IMAGE (offset=0x%x)\n" $OFFSET
dd if=myscript.sh of=$IMAGE bs=1 count=16k seek=$OFFSET conv=notrunc

 

2) Another way is to append extra blocks before first partition (after u-boot).  Have armiban-firstrun-config read the block range with dd, verify config, and process.  No example.

 

Edit: seems like this thread is going in different direction.  Have fun... but FAT?!?

[tkaiser]

37 minutes ago, Igor said:

If we are going to add FAT partition, we need to think over and test it profoundly. Perhaps at the end if 1st is not FAT already and remove it the rootfs resize process?

 

I'm fine with that. The config file should be overwritten from /dev/urandom (to prevent recovering contents later) and then the partition can simply be deleted.

 

33 minutes ago, lrrr said:

seems like this thread is going in different direction.  Have fun... but FAT?!?

 

We're thinking about a mechanism that allows average Armbian users to edit 'first boot' behaviour. Experts and users of the build system don't need anything of this since they can customize things already.

 

It's just about a somewhat standardized way to provide a config file that's suitable for Windows and macOS users as well (and that requires a filesystem accessible by those OS)

[lanefu]

Current proposal?

 

• the FAT32 config partition is at the end of the disk image
• firstrun
  • applies configuration changes
  • config file is scrubbed via shred for security purposes
  • fat32 part is destroyed
  • rootfs expansion step proceeds as normal

[Igor]

Another idea in this context would be implementing a standard DebianInstaller to the build script and provide tiny netboot images. With eMMC powered boards, this makes sense. But it's a bit more work.

[tkaiser]

1 hour ago, Igor said:

would be implementing a standard DebianInstaller

 

How should this help with headless boards especially those where no one wants to buy a serial console (e.g. ODROID HC1 and HC2)?

[botfap]

@lrrr I like the concept but its too complicated and doesn't provide what people seem to be looking for which is a way to quickly edit the settings on the sd card from a windows pc

 

@tkaiser @Igor  Thats a whole different conversation. Let me get this tool finished tonight / tomorrow as its become useful to me and others and then lets have a conversation about that. While 80% of the functionality of what you seem to be asking for is implemented in my tool, integrating that with the existing first boot is a very different task.

 

Im happy to have a look at creating such a system but I need to ask a few questions about how the Armbian first boot currently works. What we now seem to be talking about is a combined "first boot" and "pre configuration" framework along with a mechanism to display and edit this info on a standard Windows PC or Mac along with help  and info? Am I understanding correctly?

 

@lanefu Seems entirely sensible

[botfap]

As for the armbian-image-config

 

https://github.com/botfap/armbian-image-config

Tool to pre-configure basic system settings on Armbian images. Each module can be used independently on an image but using the same module twice will overwrite not combine any changes. This tool needs superuser permissions to mount and write to system files on disk images, you will need to run it with sudo or in a root shell

armbian-image-config armbian-image.img module [options]

modules:
    n|hostname [mynewhostname]
    e|ethernet [dhcp|static] [ipaddr mask] [gw] [dns] [search]
    w|wifi [dhcp|static] ssid key [ipaddr mask] [gw] [dns] [search]
    u|user [username] [password]
    l|log2ram [enabled|disabled]
    d|debug [enabled|disabled|save]
    r|remote [ssh|xrdp] [enabled|disabled]
    t|template [package|template.tgz]
    x|rootshell
    h|help

 

Image Info
Image file can be any standard Armbian image (or ubuntu 18 / debian 9). Either self built or downloaded

Replace image name with keyword "template" to generate just the config files ready to make a template package

 

Module Help and Examples

hostname or n
Sets the system hostname for the image

armbian-image-config armbian-image.img h|hostname [mynewhostname]
    n|hostname
        name

armbian-image-config armbian-image.img h myserver

ethernet or e
Sets up ethernet networking for the first detected card for either DHCP or static IP addressing

armbian-image-config armbian-image.img e|ethenet [dhcp|static] [ipaddr mask] [gw] [dns]
    e|ethernet
        d|dhcp
        s|static
            ipaddr mask [gw] [dns] [search]

armbian-image-config armbian-image.img e d
armbian-image-config armbian-image.img e s 10.0.0.50 255.255.255.0 10.0.0.254 9.9.9.9 mynet.lan

wifi or w
Sets up wifi networking for the first detected card for either DHCP or static IP addressing

armbian-image-config armbian-image.img w|wifi [dhcp|static] ssid key [ipaddr mask] [gw] [dns] [search]
    w|wifi
        d|dhcp
            ssid key
        s|static
            ssid key ip mask [gw] [dns] [search]
        s6|staticipv6
            ssid crypto key ip6 mask6 [gw6] [dns6] [search]
    
armbian-image-config armbian-image.img w d MyWifiAP MySecret
armbian-image-config armbian-image.img w s MyWifiAP MySecret 10.0.0.51 255.255.255.0 10.0.0.254 9.9.9.9 mynet.lan

user or u
Sets the username and password of the primary system user

armbian-image-config armbian-image.img u [username] [password]

log2ram or l
Enable or disable log2ram for troubleshooting startup issues

armbian-image-config armbian-image.img l [enabled|disabled]

remote or r
Enable or disable remote access via ssh and rdp

armbian-image-config armbian-image.img r [ssh|xrdp] [enabled|disabled]

debug or e
Sets the boot diagnostics to enabled / disabled or save logs from image to a local archive

armbian-image-config armbian-image.img d|debug [enabled|disabled|save]
    d|debug
        enabled
        disabled
        save

e.g. armbian-image-config armbian-image.img d save

rootshell or x
Mount the image and start a chroot shell

armbian-image-config armbian-image.img rootshell

template or t
Inject a config template into the image. Can be any files in a compressed tar archive which is extracted at root (/) on the image root file system. See also Creating templates below

armbian-image-config armbian-image.img t|template [package|template.tgz]
    t|template
        package
        template.tgz

e.g. armbian-image-config armbian-image.img t aic-template.tgz

 

Creating templates
To create a config template, consisting of the config of 1 or more modules, use the keyword "template" instead of an image file name. This will populate the "tmproot" folder with the generated config files ready for packaging. To create a package use the command "armbian-image-config template package" and a config packaged named aic-template.tgz will be created

armbian-image-config template n tvbox
armbian-image-config template w d MyWiFi wpa-psk MySecret
armbian-image-config template package

 

Development Status
Active

#### Module Progress
- hostname - done, needs testing
- ethernet - done, needs testing
- wifi - done, needs testing
- user #TODO
- log2ram - testing in botfap branch
- debug - testing in botfap branch
- template done, needs testing
- rootshell #TODO
- help - done, needs testing

[lrrr]

@botfap A script generator would do most of the work, generate a config script, find offset in image and insert.  Could be script based like yours or a statically linked binary, or dynamic linked with a GUI.

 

One could do something similar with a static sized text file, such as armbian_first_run.txt, that is padded with whitespace instead of zeros.  But armbian-firstrun-config doesn't do much.

 

@tkaiser No filesystem required to mount, just rw access to the image file.  I admit, it's hacky but doesn't require much changes to build.

 

Just throwing some ideas out.  I don't care if you make armbian fat, just add option to disable it in build so I don't have to fork code. 

 

Edit: armbian-firstrun-config does not do much but it does call (source) armbian_first_run.txt.  Any code in the file will be exec on first boot.  The txt file extension threw me off, thinking it was parsed.

Edited September 22, 2018 by lrrr
Additional info

[botfap]

The armbian-image-config script is pretty much done, I'm about to push it to github but thought I best check a couple of things first with people who are more knowledgeable about Armbian than me

 

• Apart from sudo netdev audio video dialout plugdev input bluetooth systemd-journal and ssh, are there any other groups that you want the user adding to?
• What files need deleting or modifying after initial setup but before 1st reboot? (/root/.not_logged_in_yet, ~/Xauthority)
• For debugging 1st reboot problems, is there any interesting info outside of /var/log? (assuming log2ram is disabled)
• I have only added support for WPA-PSK in the wifi setup. IS WPA-Enterprise or radius auth used enough to justify adding support for them?
• IPv6 support is currently DHCP or auto config only. Are there many people who use IPv6 ONLY, no IPv4, with static addressing?

 

 

@lrrr I understand that. I like the idea, we do something similar in our production boxes. start the rootfs at an 8mb offset and embed a 4mb hidden, compressed fat partition between uboot and rootfs containing config information. We then have a windows GUI tool to manipulate the 3 files we store in there. I just dont think its suitable for Armbian, its adds a small level of complexity that is difficult for non tech users to troubleshoot. It also requires any minor tweaks, say a change of hostname, to have to repack and reinsert data to the disk image or SD Card. I could be wrong, feedback would be appreciated

 

[lrrr]

@botfapIMO, users who cannot access ext4 fs need a GUI wizard but the tool should not be required for users who can access ext4 fs.

 

It's feature creep, all because the average Armbian user cannot find a way to access the file system.  There are ways to access ext4 from Windows (ext2fsd) and Mac (macfuse).  Also, you have VM's and Linux Live USB/CD.

 

 

[chwe]

22 hours ago, lrrr said:

There are ways to access ext4 from Windows

At least for me, these tools messed quite often things up. E.g. they weren't made for 'removable' storage, I had more than on 'unexpected blue-screen'  due to ext4 fs wasn't properly 'unmounted' before removal..  

22 hours ago, lrrr said:

Also, you have VM's and Linux Live USB/CD.

If you've to setup a VM or start a Live USB/CD for your 10-20$ SBC that sounds like a lot of work and not reliable for configuration.

 

On 9/15/2018 at 8:06 PM, lanefu said:

• the FAT32 config partition is at the end of the disk image
• firstrun
  • applies configuration changes
  • config file is scrubbed via shred for security purposes
  • fat32 part is destroyed
  • rootfs expansion step proceeds as normal

I support this approach..  maybe with a 'second boot' option. e.g. disable rootfs and expansion fat shred in the first run, so that users can test if their configuration works as expected before it get's killed, this might add some complexity maybe people should just burn a new image when they mess up the config.

[TonyMac32]

9 hours ago, chwe said:

On 9/22/2018 at 3:18 PM, lrrr said:

There are ways to access ext4 from Windows

At least for me, these tools messed quite often things up.

Agreed.

 

 

 

 

[Igor]

On 9/12/2018 at 4:55 PM, botfap said:

Just the basics:

I would propose to add additional networking setup ways:

 

- ifupdown

- systemd

 

This way we cover exceptions and I can clean up armbian-config.

[lrrr]

Used ext2fsd for years when I used to dual boot with Windows.  Never encountered any problems.  Tried it the other day before posting.  No problems accessing the sdcard and adding my custom configuration to armbian_first_run.txt. 

[chwe]

On 9/24/2018 at 7:32 AM, Igor said:

I would propose to add additional networking setup ways:

 

- ifupdown

- systemd

Do we really want this? IMO people knowing what they're doing should know how to mount an image to do such stuff.. and for the others, not sure if this just adds more support-questions like: I messed up network, and now I've no idea how to bring it back please help cause I spend x hours to set up my SBC.