Improving / simplifying first-run services using systemd features

[Matthijs Kooijman]

Hi all,

 

While building a custom image that needs to do some things at the end of the first bootup and reboot, I ran into some issues with armbian-firstrun.service. It currently has Type=simple, which means dependencies will be started when armbian-firstrun is *started*, and there is no clean way to wait until after it has *finished*. Looking to fix this, I noticed some other things that could be improved in this area, such as using systemd's first-boot-complete.target and ConditionFirstBoot to more simply manage first-boot-only services.

 

I'm considering implementing some changes and submitting them in a PR, but before I do that, I'd like to get some feedback on the approach and whether it seems worth investing time into. I considered creating a github issue about this, but given it is sort-of a feature request and the github new issue wizard seems intended to redirect away from github issues, I thought to instead post here. If a github issue seems more appropriate, I'll gladly repost there.

 

In any case, to solve the particular problem I was having (a service that needs to run after the full first boot has completed, including resizing and firstrun script), here's three incremental changes that I would think would make sense (just 1. would be the minimal to solve this problem, 1. + 2. would solve it more generally, and 1. + 2. + .3 would simplify the code a bit maybe).

1. Make armbian-firstrun.service `Type=oneshot`, so you can declare `After=armbian-firstrun.service` and `After=armbian-resize-filesystem.service`. This also means that system boot is delayed until this service is completed, but it does not prevent other boot services to run in parallel, so I would not consider this an issue (if it is, then removing the `Before=getty...` stuff could be considered, which is currently pointless anyway (you will not get a login prompt until armbian-firstrun.service is started, but it will not wait for completion). Note that using `Type=simple` in armbian-firstrun.service originates from https://github.com/armbian/build/commit/ee8d396fa639cd89e08fdd6646c32308f3b25f4f, but there is no rational for this choice.

2. Give armbian-firstrun.service and armbian-resize-filesystem `Wants=first-boot-complete.target` and `Before=first-boot-complete.target` so `first-boot-complete.target` does not fire until `armbian-firstrun` is done, allowing others to just do `After=first-boot-complete.target` without having to explicitly reference specific services.

3. Give armbian-firstrun.service and armbian-resize-filesystem `ConditionFirstBoot=yes` to let systemd ensure they are only called on the first boot. This allows removing the `systemctl disable` calls in these respective scripts as well, but needs the `/etc/machine-id` file to be removed (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900366#20), but that's probably a good idea anyway (to be regenerated on first boot, currently all machines using the same image have the same machine id).

 

One thing (feature or seeming problem maybe) with using ConditionFirstBoot as suggested above, is that systemd ensures that if the first boot is interrupted, the first boot services are ran again on the subsequent boots, until they have all ran completely (to ensure they all had a chance to ran). This is implemented by (roughly) running ConditionFirstBoot services only if `/etc/machine-id` does not exist, and only writing it to disk *after* all `first-boot-complete.target` services have completed (also when they failed, I think). This means that if the boot is interrupted before *all* first-boot services are ran, all of them will be ran again on the next boot and needs these services to be ok with being ran twice (this is also currently the case when the boot is interrupted when either or both services are still running, except that if either service completes, then that particular service will not be retried, but others might be). See also https://man7.org/linux/man-pages/man5/machine-id.5.html

 

As a related optimization, if 1. above is applied I think we could also add a `Before=ssh.service` to armbian-firstrun.service and remove the sshd restart from `armbian-firstrun`, so systemd just delays sshd startup until the keys are generated (but only if armbian-firstrun is active, so on subsequent boots sshd is started as normal).

 

Also, regardless of all of the above, I see that the armbian-firstrun deletes the ssh host keys and then forces regenerating them by calling dpkg-reconfigure. However, it would seem safer to me to actually delete the ssh keys at the end of image generation somewhere (maybe in `post_debootstrap_tweaks()`?), to really rule out the possibility of machines all using the same (publicly known) keys from the image file.

 

So, how do these sound?

[TRS-80]

Don't be discouraged if people don't see this right away.  Maybe bump it again later.  Well, here, take my bump. 

 

Those warnings are to scare off the tidal waves of people seeking free support through the back door, not for actual well thought out propositions like you are making here. 

 

Did you try playing with this locally yet?

 

 

[lanefu]

On 3/14/2022 at 10:23 AM, Matthijs Kooijman said:

So, how do these sound?

All great points.   Oneshot seems appropriate. I don't think there's an outstanding  technical reason that would prevent us from changing it.  Feel free to open a draft pull request as your developing.

 

On 3/14/2022 at 10:23 AM, Matthijs Kooijman said:

Also, regardless of all of the above, I see that the armbian-firstrun deletes the ssh host keys and then forces regenerating them by calling dpkg-reconfigure. However, it would seem safer to me to actually delete the ssh keys at the end of image generation somewhere (maybe in `post_debootstrap_tweaks()`?), to really rule out the possibility of machines all using the same (publicly known) keys from the image file.

would ask that this piece be a separate PR from  one shot service changes discussed.

[ManoftheSea]

Sounds like a great use of available features.

[ManoftheSea]

https://github.com/armbian/build/pull/3642