thexman Posted January 16, 2019 Posted January 16, 2019 6 hours ago, Igor said: This could be safe to ignore, but it would be nice to peek into the logs to see which module(s) are not getting loaded. sudo journalctl -u systemd-modules-load.service I can provide you with more information on the systemd-modules-load.service later on. However I'm currently back again on the legacy kernel because I noticed other stuff not working (which actually isn't crucial). But is it the case that all Armbian images with mainline kernel don't support the hardware read-outs of H3 processors? Armbianmonitor isn't working correctly on my side with 16.04 and mainline e.g. voltage, DRAM clock, temperature and so on are not retrieved correctly. The same is the case for h3consumption which doesn't work any more and a few other tools which now report either no value or wrong values (EZ Monitor, phpSysMonitor). Or is it because of the missing compatibility of 16.04 with mainline kernel as it is intended for legacy? thanks in advance for more info on that!
thexman Posted January 16, 2019 Posted January 16, 2019 (edited) 4 hours ago, BlackWoxs said: @thexman thank you very much for your work and the batch files on GitHub. Thanks to these I am now back and running fine under systemd-229-4ubuntu-21.11, without having to resort to non-standard patching/config. Much appreciated. Hey BlackWoxs, you are very welcome! I'm very happy that I could contribute something useful that is also helping other people Edited January 16, 2019 by thexman typo 1
Igor Posted January 16, 2019 Posted January 16, 2019 17 minutes ago, thexman said: I can provide you with more information on the systemd-modules-load.service later on No need. We don't have capacity to sponsor support for generic Debian problems - use Google, Debian/Ubuntu forums. Pointing you to the right direction is already a premium service which will save you some time. 23 minutes ago, thexman said: Armbianmonitor isn't working correctly on my side with 16.04 and mainline That's is known. This service along with other utilities were made specially for legacy kernel. Most of those functions does not exists and/or will never exists in a modern kernel. Use kernel which suits your needs best.
scaevola Posted January 22, 2019 Posted January 22, 2019 @thexman Thank you very much for recognizing the problem and your github bash files! They are really heplful! I have Orange Pi PC server at home, which serves as dlna server and also tvheadend and samba store. So after last update I was really upset when it occurred I can't connect with server anymore through SSH. Fortunately I installed Webmin few months ago and thanks to that I was able to get to webmin virtual console and easy repair the systemd thanks to your files. It just came to me that I've got two Opi Zeros, running as dlna players and they have auto updates on. But yesterday they worked fine. I hope this bug won't happen to them. 2
thexman Posted January 23, 2019 Posted January 23, 2019 @Igor It's fine, I thought you were interested in the output of the log. I simply misunderstood your post. It's not necessary to look into or provide support in that case because I also didn't invest much time finding the origin. Furthermore, I'm back on legacy for the moment and stay with it and watch how the systemd updates in the future proceed. Maybe the bug will be fixed and at the moment I have no desperate need to switch to mainline. Thanks for the additional information concerning Armbianmonitor and mainline that is very useful. @scaevola You are very welcome, thanks for your words I'm happy that the scripts helped you . The issue with systemd is also very annoying for me because my OPi is a DNS server so it really sucks when a core update breaks the whole thing. I'm glad that I have a VNC server on the OPi otherwise it would be a pain in the a** because it works in headless mode (HDMI off)... I find it more convenient to simply execute one single batch file instead of searching every single of the four packages manually. IDK if there is a better way to find and get them without retrieving the download paths directly from Launchpad. A more sophisticated script would consider a specific Ubuntu versions, target platform and parse the download paths automatically. However for the moment this should be ok! Probably there is already a script where you can specify platform, package(s), system version and package version which then retrieves the correct paths and installs the package. Maybe an idea for a future project if not already existent . If your OPi Zeros are not on legacy kernel 3.x, you can check with uname -a, they probably won't be affected at all. If they have kernel version 4.x it should be fine from what I noticed. 1
plus90 Posted January 26, 2019 Posted January 26, 2019 I thought I was hacked I didn't have a lot of time to research, but I wanted to share my solution to the SSH issue. I boot my Orange Pi Zero with serial connection and I run this command: sudo mkdir -p /var/run/sshd Afert that, SSH works until the next reboot
thexman Posted January 26, 2019 Posted January 26, 2019 (edited) @plus90 Yeah this works I also tried that the other day. You could also create the directory on start up, BUT you have to do this for all services on your device not initializing correctly. In my case that is a lot of stuff and also new installed services might break again as it affects volatile folder creation. Give my permanent solution a shot . Edited January 26, 2019 by thexman
IgorS Posted January 26, 2019 Posted January 26, 2019 I faced with same problem on OPiplus2e, Ubuntu Xenial, legacy kernel 3.4.113-sun8i. After systemd update to version 229-4ubuntu21.15 program /bin/systemd-tmpfiles which creates tmp directories on start stopped working. For each dir (configured in /usr/lib/tmpfiles.d) message is the same: Failed to validate path /some_path: Bad file descriptor This can be checked by executing as root command: SYSTEMD_LOG_LEVEL=debug systemd-tmpfiles --create sshd I had old backup from august last year, systemd version 229-4ubuntu21.4 I just copied this file (/bin/systemd-tmpfiles) from backup an everything start to work. What is exactly wrong with it I am not sure.
thexman Posted January 26, 2019 Posted January 26, 2019 @IgorS Interesting that this approach works, however I wouldn't recommend doing this because you now have a different version of systemd-tmpfiles compared to the other systemd packages. This might cause compatibility issues. You should downgrade all systemd packages to a specific version instead of overwriting files manually. Also this might result in inconsistencies between the apt version of systemd-tempfiles and the real version residing on your disk. For more information on the bug see my post and the attached solution to downgrade to the latest working systemd revision without any issues.
IgorS Posted January 26, 2019 Posted January 26, 2019 I'll take my chances. systemd-tmpfiles just creates and cleans directories based on coniguration, wjhile rest of systemd handles other things. Recently was, as far aa I know disovered security risk in systemd which ie possible to exploit from remote, therefore I wold like to keep last version and I really don't expect troubles because of this. Apt, in other hand cant know nothing about this, bec<use it just keep itts darabase of package versions.
thexman Posted January 26, 2019 Posted January 26, 2019 @IgorS Ok how you mind ;). But it has to be said that your 229-4ubuntu21.4 is older than the last working version 229.4ubuntu21.11 as I proposed. Plus you still have the version discrepancy between apt and your real disk version. I'm not sure in which package systemd-tmpfiles is included but I assume systemd so you have two different versions of one package on your disk. But I do understand that just tmpfiles is replaced by 21.4. Yes apt doesn't know and that is what I said. I have no knowledge how the systemd included components interact TBH so it might not be a problem at all. If you take the security point serious switch to mainline kernel and everything will be good at all...because you currently still rely on an old version which can't be updated. This will also be the best bet depending on what you want.
thexman Posted January 26, 2019 Posted January 26, 2019 @IgorS Ok can't say anything about that because I haven't tried. I assume video accel. in >16.04.5 does work so there might be a fix for that because all never version depend on mainline.
JRD McLAREN Posted January 26, 2019 Posted January 26, 2019 So .. I have the same problem .. after update. http://ix.io/1zim Fast fix to run sshd - edit /etc/ssh/sshd_config UsePrivilegeSeparation no Board / system is OpiZero H2+ ARMBIAN 5.70 stable Ubuntu 16.04.5 LTS 3.4.113-sun8i
thexman Posted January 26, 2019 Posted January 26, 2019 @JRD McLAREN Does not really fix the root cause, only SSH is fixed other services might still not work / fail to start. Also UsePrivilegeSeparation might impose a higher security risk, for more info read the man page of sshd_config.
JRD McLAREN Posted January 26, 2019 Posted January 26, 2019 Yes, you're right. I know it. But, I need to access ssh and it look's as a common systemd package problem "on Ubuntu's side"
thexman Posted January 26, 2019 Posted January 26, 2019 @JRD McLAREN The first service that breaks should be `systemd-tmpfiles`. I have my doubts that this will get fixed on Ubuntus side, because I reported it. The legacy kernel is quite old and it looks like they somehow don't care if the new systemd breaks it or not as no one felt responsible answering anything about how or if this will be resolved. The only real solution is downgrading to 21.11 which is the last working systemd version and put systemd on hold, at least for some time and see if a newer systemd fixes the problem. Here is a link my script which does that automatically.
thexman Posted January 31, 2019 Posted January 31, 2019 Just want to let you know that the issue was fixed with latest kernel and now working for me. See also post here.
plus90 Posted February 19, 2019 Posted February 19, 2019 I can't connect to the OpenVPN server on my Orange Pi Zero since this SSH issue has started. The OpenVPN server status is active (running). My version info: ARMBIAN 5.73 stable Ubuntu 16.04.5 LTS 3.4.113-sun8i @thexman, thanks for the proposed solution, I'm planning on trying that. I just wanted to report the issue for now. Thanks.
thexman Posted February 19, 2019 Posted February 19, 2019 @plus90 Hey I can't confirm that OpenVPN server is affected by the latest kernel version or by the latest systemd. My OpenVPN server on 5.73 is working as expected and I can connect like before. Everything is up to date on my server and I'm not facing issues. Just to let you know that this might not be a general problem. You are welcome but I'm not assuming that my solution will be a workaround as the problem might be located somewhere else but feel free to try it as it could be easily reverted ;). If the service is up and running there is a high chance that you're facing a problem that is not related to original systemd problem.
djurny Posted February 23, 2019 Posted February 23, 2019 - edit - How my skepticism (read: incredulity and conceit) can put me on my wrong foot sometimes . Updated to the latest version of systemd (and ancillaries) yesterday and this has fixed both ssh ('/var/run/sshd' missing) and GNU screen ('Cannot make directory '/var/run/screen': Permission denied') issues on my OrangePi Zero setup: ** 229-4ubuntu21.16 Groetjes, Spoiler Hi, I'm having the same recurring issue with the ssh service failing to startup as well on my OrangePi Zero. After lots of tinkering I found out that '/var/run' and/or '/run' was overlayed by a 'tmpfs' after/during the tmpfiles service was started. This caused the tmpfiles service to use a inode for /var/run or /run that wasn't accessible anymore due to the tmpfs overlay on '/run' or '/var/run'. (Details are lost on me, as this was quite some time ago already.) I modified the ssh service configuration to have ssh wait on the tmpfiles-setup service - see Spoiler below. The fixup worked for a while, but after a reboot over the holidays (lots of aptitude update+upgrades before the reboot), it appears that it's back to non-operational. Quite annoying, as it's a headless board. Perhaps the Spoiler below might also work for some people, as soon as I have a configuration change working, I'll share that here as well. Next search is to figure out when and by who the tmpfs overlay is done for /run and /var/run. I prefer to fix the configuration to get the ssh service to work and keep up-to-date with all packages in the meantime. Comments are welcome by the way, Groetjes, p.s. For all spoilers and fixups in this reply, use at your own risk and make sure you have backups of all files that you choose to modify. Also make sure that you have an alternative login method other than ssh to mend any typos or other collateral damage that might follow. Spoiler --- ssh.service 2018-12-01 21:40:36.728028572 +0800 +++ ssh.service.fixup 2018-12-01 21:41:50.837422407 +0800 @@ -1,6 +1,8 @@ [Unit] Description=OpenBSD Secure Shell server -After=network.target auditd.service +#[begin] Ansible Managed: Add dependency on tmpfiles-setup +After=network.target auditd.service systemd-tmpfiles-setup.service +#[end] Ansible Managed: Add dependency on tmpfiles-setup ConditionPathExists=!/etc/ssh/sshd_not_to_be_run [Service] Spoiler above can be applied with GNU 'patch', on '/lib/systemd/system/ssh.service'. systemd needs a 'systemctl deamon-reload' after changing the service configuration, as the dependencies have changed. Spoiler Spoiler below is not in unified diff format and has to be applied to '/lib/systemd/system/systemd-tmpfiles-setup.service' manually. [Unit] Description=Create Volatile Files and Directories Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8) DefaultDependencies=no Conflicts=shutdown.target After=local-fs.target systemd-sysusers.service #[disabled] Before=sysinit.target shutdown.target #[begin] Before=sysinit.target shutdown.target sshd.service #[end] RefuseManualStop=yes
StefanH. Posted March 28, 2019 Posted March 28, 2019 @thexman thanks a lot for your downgrade instruction! you saved me hours of trial and error!!! btw: you wrote earlier that you upgraded to the new kernel but then downgraded again. is this still the current situation or did you overcome all issues you had with the next kernel? 1
thexman Posted March 28, 2019 Posted March 28, 2019 3 hours ago, StefanH. said: @thexman thanks a lot for your downgrade instruction! you saved me hours of trial and error!!! btw: you wrote earlier that you upgraded to the new kernel but then downgraded again. is this still the current situation or did you overcome all issues you had with the next kernel? @StefanH. You are very welcome! I did quite a few tests (including switching to mainline but dropped it in favour of the latest legacy). I upgraded the kernel and this also fixed the issue for me permanently without the need of downgrading "systemd". I am currently on ARMBIAN 5.73 stable Ubuntu 16.04.6 LTS 3.4.113-sun8i but I will now upgrade to 5.74 as I saw there is a newer version. You can upgrade your kernel version via "sudo armbian-config" => System => Other. There you can choose the kernel you want to switch to. 4.x.x is mainline kernel and might break specific legacy board features so I would recommend 3.4.113 with version 5.73 or 5.74. But 5.73 already fixed the problem so it is up to you what version you want to choose. Update: strangely the update to 5.74 worked and after that the tool showed me 5.75 which also worked but motd still shows me 5.73. IDK what's is going on here. But I can confirm that 5.73 - 5.75 do work. 1
StefanH. Posted March 29, 2019 Posted March 29, 2019 I've upgraded as well (apt-get update / upgrade). Was running on old 5.38 Now /etc/armbian-release points now to 5.73 although dpkg -l lists packages like: ii linux-image-sun8i 5.75 armhf Linux kernel, version 3.4.113-sun8i reboot (especially systemd-tmpfiles-setup.service) worked fine and services like SSH were started successfully. The official documentation states that the systemd bug was solved in 5.74 https://docs.armbian.com/Release_Changelog/ Once more, thanks!
thexman Posted March 29, 2019 Posted March 29, 2019 @StefanH. I think the issue was already fixed with 5.73 as I wrote that in an earlier posts. This is a mistake in the docs I guess. I checked myself /etc/armbian-release shows 5.73 and the image shows 5.75 so this might be a bug :D. I checked all my services yesterday and none failed on boot up with 5.75. no problem . I thought you already used apt to update and the kernel wasn't updated for some reason. Therefore I pointed on the armbian-config method. Typically apt does it. 5.74/5.75 however was not updated on my device. Or it was missing because auto updates run once per week on my machine
Mabrafs Posted April 17, 2019 Posted April 17, 2019 Had the similar problem but it seems that issue have already been fixed
thexman Posted April 25, 2019 Posted April 25, 2019 @Mabrafs Fixed with latest kernel please read posts in this thread for more information
perfstr Posted May 23, 2019 Posted May 23, 2019 Thank you very much, vfrolov! I've suffered from the error messages on boot for long time and occasionally found your solution that worked for me.
Recommended Posts