thexman

@Mabrafs Fixed with latest kernel please read posts in this thread for more information

@StefanH. I think the issue was already fixed with 5.73 as I wrote that in an earlier posts. This is a mistake in the docs I guess. I checked myself /etc/armbian-release shows 5.73 and the image shows 5.75 so this might be a bug :D. I checked all my services yesterday and none failed on boot up with 5.75. no problem . I thought you already used apt to update and the kernel wasn't updated for some reason. Therefore I pointed on the armbian-config method. Typically apt does it. 5.74/5.75 however was not updated on my device. Or it was missing because auto updates run once per week on my machine

@StefanH. You are very welcome! I did quite a few tests (including switching to mainline but dropped it in favour of the latest legacy). I upgraded the kernel and this also fixed the issue for me permanently without the need of downgrading "systemd". I am currently on ARMBIAN 5.73 stable Ubuntu 16.04.6 LTS 3.4.113-sun8i but I will now upgrade to 5.74 as I saw there is a newer version. You can upgrade your kernel version via "sudo armbian-config" => System => Other. There you can choose the kernel you want to switch to. 4.x.x is mainline kernel and might break specific legacy board features so I would recommend 3.4.113 with version 5.73 or 5.74. But 5.73 already fixed the problem so it is up to you what version you want to choose. Update: strangely the update to 5.74 worked and after that the tool showed me 5.75 which also worked but motd still shows me 5.73. IDK what's is going on here. But I can confirm that 5.73 - 5.75 do work.

@plus90 Hey I can't confirm that OpenVPN server is affected by the latest kernel version or by the latest systemd. My OpenVPN server on 5.73 is working as expected and I can connect like before. Everything is up to date on my server and I'm not facing issues. Just to let you know that this might not be a general problem. You are welcome but I'm not assuming that my solution will be a workaround as the problem might be located somewhere else but feel free to try it as it could be easily reverted ;). If the service is up and running there is a high chance that you're facing a problem that is not related to original systemd problem.

So then also thanks @ning 😋, didn't check what was part of the actual fix!

Just want to let you know that the issue was fixed with latest kernel and now working for me. See also post here.

@Igor Can confirm that the updated fixed the issue. Updated to latest kernel, armbian + systemd. Thanks for the fix 👌

@Igor haha, well that actually can happen quite easily

@Sergio I can't tell how many end users were affected but also servers suffered. So this is definitely no OPI issue and I wouldn't untrust the platform. Take a look in this thread for further reading because I attached a handful of bug reports referencing to Ubuntu Launchpad concerning this issue. The origin is a systemd security update done by Ubuntu. If you would have run the same distro and kernel revision on a RPI it probably would have suffered the same issue because also x86_x64 platforms were affected. It is a annoying bug I totally agree and this shouldn't happen at all.

@Sergio The problem with SBCs (except Raspberry) is that there is a huge variety of boards and SOCs (with limited native linux manufacturer support) which have to be integrated. This also made a legacy kernel necessary in the first place. In case of the RPI you actually have one type of SOC with different revisions which is much better supported and easier to integrate. Also the legacy kernel is quite old which might in this case also be the actual reason. I assume that the issue not absolutely has something to do with the SOC or board but rather with the kernel version. Because also other virtualized providers with specific kernels suffered the issue and this has actually nothing to do with a specific SBC or SOC.

@Sergio You don't inevitably need to switch to Stretch to avoid the issue. Simply switching to mainline on Xenial will also work. As already said legacy kernel features will not work anymore but that would be the same for any mainline. Switching can be done through the armbian configuration tool.

@scaevola Thanks for mentioning me. Here is my script on Github to automatically dowgrade systemd to latest functioning release, just in case that someone is interested or wants to try or use it ;). No need to dig in the original thread. The readme there also describes how to use it. Upgrading to mainline kernel also works, I have tried that the other day. But be aware that some applications especially crafted for legacy might break.

@JRD McLAREN The first service that breaks should be `systemd-tmpfiles`. I have my doubts that this will get fixed on Ubuntus side, because I reported it. The legacy kernel is quite old and it looks like they somehow don't care if the new systemd breaks it or not as no one felt responsible answering anything about how or if this will be resolved. The only real solution is downgrading to 21.11 which is the last working systemd version and put systemd on hold, at least for some time and see if a newer systemd fixes the problem. Here is a link my script which does that automatically.

@JRD McLAREN Does not really fix the root cause, only SSH is fixed other services might still not work / fail to start. Also UsePrivilegeSeparation might impose a higher security risk, for more info read the man page of sshd_config.

@IgorS Ok can't say anything about that because I haven't tried. I assume video accel. in >16.04.5 does work so there might be a fix for that because all never version depend on mainline.

@IgorS Ok how you mind ;). But it has to be said that your 229-4ubuntu21.4 is older than the last working version 229.4ubuntu21.11 as I proposed. Plus you still have the version discrepancy between apt and your real disk version. I'm not sure in which package systemd-tmpfiles is included but I assume systemd so you have two different versions of one package on your disk. But I do understand that just tmpfiles is replaced by 21.4. Yes apt doesn't know and that is what I said. I have no knowledge how the systemd included components interact TBH so it might not be a problem at all. If you take the security point serious switch to mainline kernel and everything will be good at all...because you currently still rely on an old version which can't be updated. This will also be the best bet depending on what you want.

@IgorS Interesting that this approach works, however I wouldn't recommend doing this because you now have a different version of systemd-tmpfiles compared to the other systemd packages. This might cause compatibility issues. You should downgrade all systemd packages to a specific version instead of overwriting files manually. Also this might result in inconsistencies between the apt version of systemd-tempfiles and the real version residing on your disk. For more information on the bug see my post and the attached solution to downgrade to the latest working systemd revision without any issues.

@plus90 Yeah this works I also tried that the other day. You could also create the directory on start up, BUT you have to do this for all services on your device not initializing correctly. In my case that is a lot of stuff and also new installed services might break again as it affects volatile folder creation. Give my permanent solution a shot .

@Igor It's fine, I thought you were interested in the output of the log. I simply misunderstood your post. It's not necessary to look into or provide support in that case because I also didn't invest much time finding the origin. Furthermore, I'm back on legacy for the moment and stay with it and watch how the systemd updates in the future proceed. Maybe the bug will be fixed and at the moment I have no desperate need to switch to mainline. Thanks for the additional information concerning Armbianmonitor and mainline that is very useful. @scaevola You are very welcome, thanks for your words I'm happy that the scripts helped you . The issue with systemd is also very annoying for me because my OPi is a DNS server so it really sucks when a core update breaks the whole thing. I'm glad that I have a VNC server on the OPi otherwise it would be a pain in the a** because it works in headless mode (HDMI off)... I find it more convenient to simply execute one single batch file instead of searching every single of the four packages manually. IDK if there is a better way to find and get them without retrieving the download paths directly from Launchpad. A more sophisticated script would consider a specific Ubuntu versions, target platform and parse the download paths automatically. However for the moment this should be ok! Probably there is already a script where you can specify platform, package(s), system version and package version which then retrieves the correct paths and installs the package. Maybe an idea for a future project if not already existent . If your OPi Zeros are not on legacy kernel 3.x, you can check with uname -a, they probably won't be affected at all. If they have kernel version 4.x it should be fine from what I noticed.

Hey BlackWoxs, you are very welcome! I'm very happy that I could contribute something useful that is also helping other people

I can provide you with more information on the systemd-modules-load.service later on. However I'm currently back again on the legacy kernel because I noticed other stuff not working (which actually isn't crucial). But is it the case that all Armbian images with mainline kernel don't support the hardware read-outs of H3 processors? Armbianmonitor isn't working correctly on my side with 16.04 and mainline e.g. voltage, DRAM clock, temperature and so on are not retrieved correctly. The same is the case for h3consumption which doesn't work any more and a few other tools which now report either no value or wrong values (EZ Monitor, phpSysMonitor). Or is it because of the missing compatibility of 16.04 with mainline kernel as it is intended for legacy? thanks in advance for more info on that!

I finally upgraded to the mainline Kernel Linux pan 4.19.13-sunxi #5.70 SMP Sat Jan 12 15:43:21 CET 2019 armv7l armv7l armv7l GNU/Linux like described here and the issue is now also gone for me even with systemd-229-4ubuntu21.15 However I have now an issue with the service systemd-modules-load.service loaded failed failed Load Kernel Modules which won't start I don't know what the culprit of the issue is and what effects it has.

Hello Community, This is going to be a longer post because I want to summarize everything I found out and I will also attach useful resources and additions to @vfrolov tutorial. E.g. on how to get back to a working / previous state with a certain release version. I also experienced the issue discussed here on my OrangePI One with ARMBIAN 5.70 stable Ubuntu 16.04.5 LTS 3.4.113-sun8i myself. The problems first occurred with systemd-229-4ubuntu21.9 affecting several services (e.g. ssh, dnsmasq, lighttpd, openvpn,...) that on my little server refused to start - full list of affected services are listed below. With the next version of systemd-229-4ubuntu21.10 the issue was solved through a regression release. Since systemd-229-4ubuntu21.15 is out the problems finally occurs again in the same severity on my server. I have no in deep knowledge why this actually occurs or if this will ever be fixed again. However you can get more info in the resources (bug reports) I listed at the end of this post. I added more information on my actual case there and I also invite you to state your issues there so we have a better chance to get some response or fix. Basically, the origin of the issue seems to be related to systemd-tmpfiles which has changed in 21.9 and 21.15 and now breaks volatile path creation for certain services that are started on boot up and depend on these paths. See also my journalctl listing below which is a tiny snapshot of paths not accessible and prevent services to start correctly. Based on my findings the issue is caused by Ubuntu itself, however it's not clear if a next release will address the problem and what Distros are actually affected. It has already been discussed above on how to downgrade to systemd-229-4ubuntu4. However this is an quite old version of systemd (~2016) and obviously therefore not the best solution. It will work but you will skip over two years of updates. Sadly, through the repos and automatic apt updates you can only update to systemd-229-4ubuntu21.15 or downgrade to systemd-229-4ubuntu4. As @vfrolov already stated we need four packages to fix the problem with systemd. Below are the links to the respective Launchpad package repositories where you can find the downloads for all revisions of the respective armhf packages: libpam-systemd libsystemd0 systemd systemd-sysv Based on that I took my time and searched for both systemd-229-4ubuntu21.10 and systemd-229-4ubuntu21.11 all the necessary install resources. Hence I combined all necessary package download paths and created two easy commands per release to downgrade a system on either of the two versions (both working and both fixing the issue for me). Downgrade to systemd-229-4ubuntu-21.10: Navigate to any folder and execute each of this two commands below. 1. Download packages, 2. install packages manually. wget http://launchpadlibrarian.net/399247908/libpam-systemd_229-4ubuntu21.10_armhf.deb | wget http://launchpadlibrarian.net/399247910/libsystemd0_229-4ubuntu21.10_armhf.deb | wget http://launchpadlibrarian.net/399247918/systemd_229-4ubuntu21.10_armhf.deb | wget http://launchpadlibrarian.net/399247917/systemd-sysv_229-4ubuntu21.10_armhf.deb sudo dpkg -i libpam-systemd_229-4ubuntu21.10_armhf.deb libsystemd0_229-4ubuntu21.10_armhf.deb systemd_229-4ubuntu21.10_armhf.deb systemd-sysv_229-4ubuntu21.10_armhf.deb Downgrade to systemd-229-4ubuntu-21.11: And again, choose any folder and execute both commands below. wget http://launchpadlibrarian.net/401375344/libpam-systemd_229-4ubuntu21.11_armhf.deb | wget http://launchpadlibrarian.net/401375346/libsystemd0_229-4ubuntu21.11_armhf.deb | wget http://launchpadlibrarian.net/401375354/systemd_229-4ubuntu21.11_armhf.deb | wget http://launchpadlibrarian.net/401375353/systemd-sysv_229-4ubuntu21.11_armhf.deb sudo dpkg -i libpam-systemd_229-4ubuntu21.11_armhf.deb libsystemd0_229-4ubuntu21.11_armhf.deb systemd_229-4ubuntu21.11_armhf.deb systemd-sysv_229-4ubuntu21.11_armhf.deb You can also generate a bash script to download, update and remove package files automatically in case you want to try a future release and then rollback to a working state in case the issue still perists. That is no magic at all so I won't attach any resources for that here. However if someone is interested I can upload a bash script for both version for easy one line execution. Update: I decided to create a repo for the sake of completeness which includes the bash files and instruction on how to use them: systemd-downgrades on Github. My Ubuntu Version: Distributor ID: Ubuntu Description: Ubuntu 16.04.5 LTS Release: 16.04 Codename: xenial My Kernel: Linux pan 3.4.113-sun8i #2 SMP PREEMPT Sat Jan 12 15:54:26 CET 2019 armv7l armv7l armv7l GNU/Linux Output of journalctl -b 0 -u systemd-tmpfiles-setup.service: Jän 14 11:01:51 xxx systemd[1]: Starting Create Volatile Files and Directories... Jän 14 11:01:51 xxx systemd-tmpfiles[581]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/log: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/lib: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/sendsigs.omit.d: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /home: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /srv: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/lock/subsys: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/run/lighttpd: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/cache: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/cache/man: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/openvpn: Bad file descriptor Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE Jän 14 11:01:51 xxx systemd[1]: Failed to start Create Volatile Files and Directories. Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state. Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'. Affected services: # dnsmasq.service loaded failed failed dnsmasq - A lightweight DHCP and caching DNS server # lighttpd.service loaded failed failed Lighttpd Daemon # <email address hidden> loaded failed failed OpenVPN connection to server # ssh.service loaded failed failed OpenBSD Secure Shell server # systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device Nodes in /dev # systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories Resources related to the problem: Launchpad - 1804847 Launchpad - 1804603 Launchpad - 1811580