Help With Understanding Firstrun and Resetting Armbian Images

[Jeremiah Cornelius]

Hello,
It's been more than two days working to solution. I have gone deep into the documentation and forum posts before asking this. Much of the information in this subforum is c 2017, and relates to /etc/init.d locations which are now obsolete, etc. Definitive assistance is much appreciated!

 

I'm building clean Armbian-based images, for a number of unsupported desktop configurations on unsupported boards. These have been fantastically functional, I I'd like to make these available as community-supported releases. I do not expect such a hybrid configuration, despite using an Armbian kernel package, and an Armbian Focal base filesystem image, to ever be supported - regardless of how robust and useful.

 

Example, I have built Khadas VIM3 images, using the Meson64 kernel image with a 32-bit userland, and enabling the Amlogic hardware tweaks for sound and cooling etc., using a Khadas uBoot, built with the Fenix tools. This allows for the broadest multi-media and drm compatibility, as well as support for the Raspberry-Pi fueled wealth of armhf binaries, electron packages, etc.

 

Distribution of these images entails cleaning all logfiles, and scrubbing runtime customization of ssh keys, networks, users, groups, caches, bash histories, etc. This is no problem - my utility scripts for these things, and optimizing resulting .img.xz images work very well.

 

My stumbling block is getting an Armbian OOB firstrun experience. I want Armbian FS resize, ssh to be initialized, MOTD displayed, root passwd to be set, new user created, and graphical startup option requested.

 

AFAIK I have all the required systemd units in /usr/lib/systemd/system/, all the right scripts in /usr/lib/armbian, etc.

I delete all users but root, disable displaymanager with systemctl and then issue: 'systemctl enable armbian-firstrun && systemctl enable armbian-firstrun-config && systemctl enable armbian-resize-filesystem', then immediately 'shutdown -h now'.

 

At following power-up, I often notice multiple OpenSSHd server restarts, which I assume indicate key regenerations. However, I have never been prompted to generate a new root passwd, create a regular user, or any of the other configuration goodies. THERE'S CLEARLY SOMETHING I'm missing here! :-)
 

Does /root need to be completely vacant? Is there a hidden flag written as a dotfile under /usr/lib or /var that I don't know about?
 

Solving this would be very helpful. I'm not real happy with the Khadas and Pine contributed images, which typically include a default user, with cruft and artifacts that are sometimes a risk, and never the right way to deploy.

Thanks!

- Jeremiah

₩₳ł₮ł₦₲ ₣ØⱤ ₮ⱧɆ ₲ł₣₮ Ø₣ ₴ØɄ₦Đ ₳₦Đ Vł₴łØ₦

 

 

Edited April 28, 2021 by Jeremiah Cornelius
Missing word.

[Jeremiah Cornelius]

After a couple of days, this seems to be a stumper.

I'd wish that this, an essential piece of Armbian configuration and operation, were documented as well as the image building process!

[Werner]

10 minutes ago, Jeremiah Cornelius said:

I'd wish that this, an essential piece of Armbian configuration and operation, were documented as well as the image building process!

You can help make this wish become reality. Everybody can contribute to make the documentation better: https://github.com/armbian/documentation

The lack of (human) resources is a big issue for the Armbian project so community contributions are more than welcome.

[Jeremiah Cornelius]

1 hour ago, Werner said:

You can help make this wish become reality. Everybody can contribute to make the documentation better: https://github.com/armbian/documentation

The lack of (human) resources is a big issue for the Armbian project so community contributions are more than welcome.

I agree. I'm very much in favor of making contributions, and not shy about being detailed. When the Linux world revolved around HOWTOs, not Reddit subs and siloed vendor forums, I was particularly active.

Here's a case where I'd gladly supply documentation, were I not faced with ambiguous outcomes, and a lack of authoritative information to begin!
Thanks!

— Jeremiah

[lanefu]

3 hours ago, Jeremiah Cornelius said:

After a couple of days, this seems to be a stumper.

I'd wish that this, an essential piece of Armbian configuration and operation, were documented as well as the image building process!

Can you just give me some quick bullet points on what you're waiting to achieve... I may have an answer.

[Jeremiah Cornelius]

4 hours ago, lanefu said:

Can you just give me some quick bullet points on what you're waiting to achieve... I may have an answer.

Thanks! Very kind.

• I am looking to re-set the firstboot state for a derived image, after customization and out-of-band additions.
• My efforts extend to re-enabling systemd unit files for armbian-firstrun, armbian-firstrun-config, and armbian-resize-filesystem.
• Results: New ssh keys are generated, and new UUID appended to /etc/armbian-image-release - no other config tasks are run (update root pw, create normal user, choose logon environment, etc)

Observation: /usr/lib/systemd/system/armbian-firstrun-config is deleted after being enabled, on next boot.

• Is this expected behavior?
• Is it an error in how symlinks under /etc/systemd are handled?
• Would an issue on the git repository be well received?

I'd like the unit disabled, and maybe a re-settable flag written to the image, preventing accidental firstrun.

 

My main question is still why this isn't working as expected. Does behavior change, if /etc/armbian-image-release already exists? There's something I'm missing!

 

When I understand this fully, I'll be in a better position to offer contributions.

 

[lanefu]

4 hours ago, Jeremiah Cornelius said:

Results: New ssh keys are generated, and new UUID appended to /etc/armbian-image-release - no other config tasks are run (update root pw, create normal user, choose logon environment, etc)

Doing this should probably trigger it.  https://github.com/armbian/build/blob/master/lib/distributions.sh#L157

Also take a look at 

https://github.com/armbian/build/blob/master/lib/distributions.sh#L372-L379

 

Quote

Observation: /usr/lib/systemd/system/armbian-firstrun-config is deleted after being enabled, on next boot.

Is this expected behavior?

Is it an error in how symlinks under /etc/systemd are handled?

Might be related to https://github.com/armbian/build/blob/master/packages/bsp/common/lib/systemd/system/armbian-firstrun-config.service#L9

 

5 hours ago, Jeremiah Cornelius said:

Would an issue on the git repository be well received?

Probably best to keep conversation here for now.  Mostly we use issues in GitHub for major bugs / problems

[lanefu]

5 hours ago, Jeremiah Cornelius said:

I'd like the unit disabled, and maybe a re-settable flag written to the image, preventing accidental firstrun.

Yeah first run has a few distributed touchpoint.. kind of needs to be consolidated and more intuitive to manage as you describe.. I know many have interest in being able to bundle in cloud-init to the image.. and making the first-riun and first-login components a little more polished would help with that.

[Jeremiah Cornelius]

16 hours ago, lanefu said:

Doing this should probably trigger it.  https://github.com/armbian/build/blob/master/lib/distributions.sh#L157

Also take a look at 

https://github.com/armbian/build/blob/master/lib/distributions.sh#L372-L379

 

Might be related to https://github.com/armbian/build/blob/master/packages/bsp/common/lib/systemd/system/armbian-firstrun-config.service#L9

 

Probably best to keep conversation here for now.  Mostly we use issues in GitHub for major bugs / problems

 

Really thanks for this. It gives me a lot more to go on.
I'll do more work in the next day or so, and be back with some updates - HOPEFULLY with some guidance to offer others, if successful!

 

The ""${SDCARD}"/root/.not_logged_in_yet" is a GREAT hint!

It's the opposite of touching a flag in the file system, so of course finding it in a configured system is impossible! I'd have taken a long time, before I got to it in the script sources.

 

— Jeremiah

[tparys]

As a possible out-of-the-box solution. Have you tried directly mounting the original image, and just rsync'ing that to your partitions?

 

# mkdir /dev/shm/{original,current}

# kpartx -av Armbian_21.02.1_Nanopim4v2_focal_current_5.10.12_desktop.img
add map loop9p1 (253:0): 0 7258112 linear 7:9 32768

# mount /dev/mapper/loop9p1 /dev/shm/original

# mount /dev/mmcblk0p1 /dev/shm/current

# rsync -av --delete /dev/shm/{original,current}/

# umount /dev/shm/original
# umount /dev/shm/current
# sync

 

I didn't try it (for obvious reasons), but that should reset pretty much all logs, passwords, user files, ssh keys, etc ..

 

The verbose option should list everything that it's doing.

 

Your mileage may vary. You might need to adjust if your build has two partitions. It may delete important things ...

[Jeremiah Cornelius]

12 minutes ago, tparys said:

As a possible out-of-the-box solution. Have you tried directly mounting the original image, and just rsync'ing that to your partitions?

 

# mkdir /dev/shm/{original,current}

# kpartx -av Armbian_21.02.1_Nanopim4v2_focal_current_5.10.12_desktop.img
add map loop9p1 (253:0): 0 7258112 linear 7:9 32768

# mount /dev/mapper/loop9p1 /dev/shm/original

# mount /dev/mmcblk0p1 /dev/shm/current

# rsync -av --delete /dev/shm/{original,current}/

# umount /dev/shm/original
# umount /dev/shm/current
# sync

 

I didn't try it (for obvious reasons), but that should reset pretty much all logs, passwords, user files, ssh keys, etc ..

 

The verbose option should list everything that it's doing.

 

Your mileage may vary. You might need to adjust if your build has two partitions. It may delete important things ...

That's a good idea. I'd have to be more selective - as there are changes to my images I'd need preserved, such as apt changes, dselect selections, dpkg options, for a start. I also do the unthinkable, and change contents of /etc/armbian-release and /etc/armbian-image-release, because the platforms aren't yet build targets, and I am playing Doctor Frankenstein with the parts. LOL

 

Thanks,
— Jeremiah

[tparys]

11 minutes ago, Jeremiah Cornelius said:

... there are changes to my images I'd need preserved ...

 

Make your changes to the image before you call rsync? /dev/shm/original is mounted rw unless you tell it otherwise.

 

13 minutes ago, Jeremiah Cornelius said:

I also do the unthinkable, and change contents of /etc/armbian-release and /etc/armbian-image-release

 

You monster

[Jeremiah Cornelius]

17 hours ago, lanefu said:

Doing this should probably trigger it.  https://github.com/armbian/build/blob/master/lib/distributions.sh#L157
 

This is it!

Touching /root/.not_logged_in_yet gets the desired, correctly reset behavior for the armbian-firstrun-config systemd unit. I knew it was some small, critical configuration I'd been missing.

I now have what's needed for a community, unsupported Armbian Khadas VIM3 image that's functionally equivalent to Odroid N2+, and have plans for similar with 32-bit apt/userspace, which enables access to an armhf ecosystem of commercial audio software, and snap images.

 

This also means that the VIM3 Twister OS image that I have some good interest in from users, can be updated to use this firstrun logic - with an addition that will hardlink the new user to the "pi" account on which Twister OS scripts and configurations depend.

 

In the background are plans for ready-to-run Armbian VIM images with Budgie desktop and Elementary OS, pre-built. Being unsupported, some of these may advance from Focal to Groovy or Hirsute.
 

Thanks!

[Jeremiah Cornelius]

2 minutes ago, tparys said:

 

You monster

Well, at least for function, and not capriciously!

— Jeremiah

[lanefu]

2 hours ago, Jeremiah Cornelius said:

n the background are plans for ready-to-run Armbian VIM images with Budgie desktop and Elementary OS, pre-built. Being unsupported, some of these may advance from Focal to Groovy or Hirsute.

 BTW have you played with https://github.com/armbian/build.  We have budgie I there, and would gladly take help there as well

..but you're doing 32bit userspace?   

[Jeremiah Cornelius]

5 minutes ago, lanefu said:

 BTW have you played with https://github.com/armbian/build.  We have budgie I there, and would gladly take help there as well

..but you're doing 32bit userspace?   

Budgie is on my list - as I get kinks worked out with clean, vanilla Armbian. I am doing hybrid arm64 kernels and armhf userspace because of audio-plugins. This worked well for me in Pinebook Pro, etc.

I'd do more with pure arm64, if I can solve some gaps with Carla, and when there's a 64-bit Linux Widevine. That's when Google decides they need a 64-bit Chromebook, I guess.

 

— Jeremiah

[lanefu]

6 hours ago, tparys said:

# mount /dev/mapper/loop9p1 /dev/shm/original

ha I love the idea of mounting to /dev/shm .. clever.. I always just made a dummy folder from /mnt

[lanefu]

FYI keep an eye on this PR from @rpardini  provides more customization opportunities. (I know this doesn't address the 32bit use space stuff, but seemed relevant to share  )

https://github.com/armbian/build/pull/2799

[tparys]

13 hours ago, lanefu said:

ha I love the idea of mounting to /dev/shm .. clever.. I always just made a dummy folder from /mnt

 

Reason I do that is that I've used this technique to backup (or obliterate) full OS images. If I rsync from an image that doesn't have that new directory, it'll wipe out the mount.

 

If I have the mount in a tmpfs at a known location, that shouldn't happen.

[rpardini]

Hey @Jeremiah Cornelius 

I've been working on something similar it seems, thanks to @lanefu for making the connection.

Although the path I took was directly modifying the Armbian build scripts, instead of modifying a live image and then re-packing it.

 

My fork of Armbian is quite complex, but I'm slowly splitting and reorganising things (and sometimes writing a metaprogramming Bash framework in the process ) 

A few points that may be of interest to you:

 

- Onboarding replaced by cloud-init, instead of any interactive onboarding, I configure cloud-init to read some YAML files from /boot.

  /boot/network-config can configure all aspects of networking (including wifi), /boot/meta-data is just that, and /boot/user-data can do almost anything, create users, passwords, import SSH keys, install packages, run scripts, resize rootfs, create SSH host keys, etc.

  cloud-init is of course focused on cloud images, but actually can be used on any environment, if you can stomach its YAML and hammer on its config.

  With cloud-init comes netplan.io (in place of NetworkManager) and some other stuff that may not be appropriate for your use-case so take with a grain of salt.

  Also my implementation is not bullet-proof yet, but it aims to work just like the Ubuntu RaspberryPi 4 "Server" image (also has cloud-init out of the box).

  Most of it is done in https://github.com/rpardini/armbian-build/blob/rpardini-fragments/userpatches/fragments/cloud-init/cloud-init.sh (including the .not_logged_in_yet thing)

  Some other related stuff in https://github.com/rpardini/armbian-build/blob/rpardini-fragments/userpatches/fragments/more_like_ubuntu_cloud.sh (which disables a bunch of stuff in the BSP,  during BSP-generation, so that reinstalls/upgrades of the bsp package do not ruin everything).

 

- A trick I call rootfs-in-rootfs, during the image build process, I capture an e2image ext4 dump of the produced ext4 rootfs and then include it in the rootfs.  A bit like having a directory full of files, creating a .zip of it, then including that .zip file inside itself  but without the compression.

  Then boot the SD card (which includes the rootfs dump inside) and run "e2image -racp /root/rootfs.ext4.e2img /dev/sda2" for example.

  The nice thing is that e2image (part of e2fsprogs) can do a "smart" copy (-c), knowing that reading is cheaper than writing, it can read both the source and destination and only write the changed blocks to the target. This allows for a very, very fast version of "nand_sata_install",  especially on the 2nd+ "flash".

  More speed comes from reading/writing some hundred-thousand whole blocks instead of possibly millions of files that rsync would need to inspect individually.

  The relevant code fragment is https://github.com/rpardini/armbian-build/blob/rpardini-fragments/userpatches/fragments/rootfs_e2img_inside_rootfs.sh

 

Hopefully these will be of some help,

Ricardo

 

 

[Jeremiah Cornelius]

14 hours ago, rpardini said:

Hey @Jeremiah Cornelius 

I've been working on something similar it seems, thanks to @lanefu for making the connection.

Although the path I took was directly modifying the Armbian build scripts, instead of modifying a live image and then re-packing it.

 

My fork of Armbian is quite complex, but I'm slowly splitting and reorganising things (and sometimes writing a metaprogramming Bash framework in the process ) 

A few points that may be of interest to you:

 

< snip >

 

Hopefully these will be of some help,

Ricardo

 

 

Oh, man. Thanks. This information is a goldmine!
I won't be able to dig in very deeply until the weekend, but I am champing at the bit, a bit. ;-)

Edited May 7, 2021 by Jeremiah Cornelius
Wrote INSIDE quote! ;-)