7 7
GeorgeL

SSH doesn't work on Orange Pi Zero

Recommended Posts

6 hours ago, Igor said:


This could be safe to ignore, but it would be nice to peek into the logs to see which module(s) are not getting loaded.


sudo journalctl -u systemd-modules-load.service

 

I can provide you with more information on the systemd-modules-load.service later on. However I'm currently back again on the legacy kernel because I noticed other stuff not working (which actually isn't crucial). But is it the case that all Armbian images with mainline kernel don't support the hardware read-outs of H3 processors? Armbianmonitor isn't working correctly on my side with 16.04 and mainline e.g. voltage, DRAM clock, temperature and so on are not retrieved correctly. The same is the case for h3consumption which doesn't work any more and a few other tools which now report either no value or wrong values (EZ Monitor, phpSysMonitor). Or is it because of the missing compatibility of 16.04 with mainline kernel as it is intended for legacy? thanks in advance for more info on that!

Share this post


Link to post
Share on other sites
(edited)
4 hours ago, BlackWoxs said:

@thexman thank you very much for your work and the batch files on GitHub.

 

Thanks to these I am now back and running fine under systemd-229-4ubuntu-21.11, without having to resort to non-standard patching/config.

 

Much appreciated.

Hey BlackWoxs, you are very welcome! I'm very happy that I could contribute something useful that is also helping other people :)

Edited by thexman
typo

Share this post


Link to post
Share on other sites
17 minutes ago, thexman said:

I can provide you with more information on the systemd-modules-load.service later on


No need. We don't have capacity to sponsor support for generic Debian problems - use Google, Debian/Ubuntu forums. Pointing you to the right direction is already a premium service which will save you some time.
 

23 minutes ago, thexman said:

Armbianmonitor isn't working correctly on my side with 16.04 and mainline


That's is known. This service along with other utilities were made specially for legacy kernel. Most of those functions does not exists and/or will never exists in a modern kernel. Use kernel which suits your needs best. 

Share this post


Link to post
Share on other sites

@thexman

Thank you very much for recognizing the problem and your github bash files! They are really heplful!

I have Orange Pi PC server at home, which serves as dlna server and also tvheadend and samba store.  So after last update I was really upset when it occurred I can't connect with server anymore through SSH. Fortunately I installed Webmin few months ago and thanks to that I was able to get to webmin virtual console and easy repair the systemd thanks to your files.

 

It just came to me that I've got two Opi Zeros, running as dlna players and they have auto updates on. But yesterday they worked fine. I hope this bug won't happen to them.

Share this post


Link to post
Share on other sites

@Igor

It's fine, I thought you were interested in the output of the log. I simply misunderstood your post. It's not necessary to look into or provide support in that case because I also didn't invest much time finding the origin. Furthermore, I'm back on legacy for the moment and stay with it and watch how the systemd updates in the future proceed. Maybe the bug will be fixed and at the moment I have no desperate need to switch to mainline. Thanks for the additional information concerning Armbianmonitor and mainline that is very useful.

 

@scaevola

You are very welcome, thanks for your words I'm happy that the scripts helped you :). The issue with systemd is also very annoying for me because my OPi is a DNS server so it really sucks when a core update breaks the whole thing. I'm glad that I have a VNC server on the OPi otherwise it would be a pain in the a** because it works in headless mode (HDMI off)... I find it more convenient to simply execute one single batch file instead of searching every single of the four packages manually. IDK if there is a better way to find and get them without retrieving the download paths directly from Launchpad. A more sophisticated script would consider a specific Ubuntu versions, target platform and parse the download paths automatically. However for the moment this should be ok! Probably there is already a script where you can specify platform, package(s), system version and package version which then retrieves the correct paths and installs the package. Maybe an idea for a future project if not already existent :P. If your OPi Zeros are not on legacy kernel 3.x, you can check with uname -a, they probably won't be affected at all. If they have kernel version 4.x it should be fine from what I noticed.

Share this post


Link to post
Share on other sites

I thought I was hacked :D

 

I didn't have a lot of time to research, but I wanted to share my solution to the SSH issue.

I boot my Orange Pi Zero with serial connection and I run this command:

sudo mkdir -p /var/run/sshd

 

Afert that, SSH works until the next reboot :)

Share this post


Link to post
Share on other sites
(edited)

@plus90

Yeah this works I also tried that the other day. You could also create the directory on start up, BUT you have to do this for all services on your device not initializing correctly. In my case that is a lot of stuff and also new installed services might break again as it affects volatile folder creation. Give my permanent solution a shot :).

Edited by thexman

Share this post


Link to post
Share on other sites

I faced with same problem on OPiplus2e, Ubuntu Xenial, legacy kernel 3.4.113-sun8i.

After systemd update to version 229-4ubuntu21.15 program /bin/systemd-tmpfiles which creates tmp directories on start stopped working.

For each dir (configured in /usr/lib/tmpfiles.d) message is the same:

  Failed to validate path /some_path: Bad file descriptor

 

This can be checked by  executing as root command: SYSTEMD_LOG_LEVEL=debug systemd-tmpfiles --create sshd

 

I had old backup from august last year, systemd version 229-4ubuntu21.4

I just copied this file (/bin/systemd-tmpfiles) from backup an everything start to work.

What is exactly wrong with it I am not sure. 

Share this post


Link to post
Share on other sites

@IgorS

Interesting that this approach works, however I wouldn't recommend doing this because you now have a different version of systemd-tmpfiles compared to the other systemd packages. This might cause compatibility issues. You should downgrade all systemd packages to a specific version instead of overwriting files manually. Also this might result in inconsistencies between the apt version of systemd-tempfiles and the real version residing on your disk. For more information on the bug see my post and the attached solution to downgrade to the latest working systemd revision without any issues.

Share this post


Link to post
Share on other sites

I'll take my chances.

systemd-tmpfiles just creates and cleans directories based on coniguration,

wjhile rest of systemd handles other things. Recently was, as far aa I know disovered security risk in systemd which ie possible to exploit from remote, therefore I wold like to keep last version and I really don't expect troubles because of this. Apt, in other hand cant know nothing about this, bec<use it just keep itts darabase of package versions.

Share this post


Link to post
Share on other sites

@IgorS

Ok how you mind ;). But it has to be said that your 229-4ubuntu21.4 is older than the last working version 229.4ubuntu21.11 as I proposed. Plus you still have the version discrepancy between apt and your real disk version. I'm not sure in which package systemd-tmpfiles is included but I assume systemd so you have two different versions of one package on your disk. But I do understand that just tmpfiles is replaced by 21.4. Yes apt doesn't know and that is what I said. I have no knowledge how the systemd included components interact TBH so it might not be a problem at all. If you take the security point serious switch to mainline kernel and everything will be good at all...because you currently still rely on an old version which can't be updated. This will also be the best bet depending on what you want.

Share this post


Link to post
Share on other sites

@IgorS

Ok can't say anything about that because I haven't tried. I assume video accel. in >16.04.5 does work so there might be a fix for that because all never version depend on mainline.

Share this post


Link to post
Share on other sites

So .. I have the same problem .. after update.

 

http://ix.io/1zim

 

Fast fix to run sshd - edit /etc/ssh/sshd_config

UsePrivilegeSeparation no

 

Board / system is OpiZero H2+ ARMBIAN 5.70 stable Ubuntu 16.04.5 LTS 3.4.113-sun8i

Share this post


Link to post
Share on other sites

@JRD McLAREN

Does not really fix the root cause, only SSH is fixed other services might still not work / fail to start. Also UsePrivilegeSeparation might impose a higher security risk, for more info read the man page of sshd_config.

Share this post


Link to post
Share on other sites

@JRD McLAREN

The first service that breaks should be `systemd-tmpfiles`. I have my doubts that this will get fixed on Ubuntus side, because I reported it. The legacy kernel is quite old and it looks like they somehow don't care if the new systemd breaks it or not as no one felt responsible answering anything about how or if this will be resolved. The only real solution is downgrading to 21.11 which is the last working systemd version and put systemd on hold, at least for some time and see if a newer systemd fixes the problem. Here is a link my script which does that automatically.

Share this post


Link to post
Share on other sites

I can't connect to the OpenVPN server on my Orange Pi Zero since this SSH issue has started.

The OpenVPN server status is active (running).


My version info:

ARMBIAN 5.73 stable Ubuntu 16.04.5 LTS 3.4.113-sun8i

 

@thexman, thanks for the proposed solution, I'm planning on trying that. I just wanted to report the issue for now. Thanks.

Share this post


Link to post
Share on other sites

@plus90

Hey I can't confirm that OpenVPN server is affected by the latest kernel version or by the latest systemd. My OpenVPN server on 5.73 is working as expected and I can connect like before. Everything is up to date on my server and I'm not facing issues. Just to let you know that this might not be a general problem. You are welcome but I'm not assuming that my solution will be a workaround as the problem might be located somewhere else but feel free to try it as it could be easily reverted ;). If the service is up and running there is a high chance that you're facing a problem that is not related to original systemd problem.

Share this post


Link to post
Share on other sites

- edit -

How my skepticism (read: incredulity and conceit) can put me on my wrong foot sometimes :unsure:.

 

Updated to the latest version of systemd (and ancillaries) yesterday and this has fixed both ssh ('/var/run/sshd' missing) and GNU screen ('Cannot make directory '/var/run/screen': Permission denied') issues on my OrangePi Zero setup: ** 229-4ubuntu21.16
 

Groetjes,

 

Spoiler

Hi, I'm having the same recurring issue with the ssh service failing to startup as well on my OrangePi Zero.

 

After lots of tinkering I found out that '/var/run' and/or '/run' was overlayed by  a 'tmpfs' after/during the tmpfiles service was started. This caused the tmpfiles service to use a inode for /var/run or /run that wasn't accessible anymore due to the tmpfs overlay on '/run' or '/var/run'. (Details are lost on me, as this was quite some time ago already.)

 

I modified the ssh service configuration to have ssh wait on the tmpfiles-setup service - see Spoiler below. The fixup worked for a while, but after a reboot over the holidays (lots of aptitude update+upgrades before the reboot), it appears that it's back to non-operational. Quite annoying, as it's a headless board.


Perhaps the Spoiler below might also work for some people, as soon as I have a configuration change working, I'll share that here as well. Next search is to figure out when and by who the tmpfs overlay is done for /run and /var/run.

 

I prefer to fix the configuration to get the ssh service to work and keep up-to-date with all packages in the meantime.

 

Comments are welcome by the way,
Groetjes,

 

p.s. For all spoilers and fixups in this reply, use at your own risk and make sure you have backups of all files that you choose to modify. Also make sure that you have an alternative login method other than ssh to mend any typos or other collateral damage that might follow.

 

Spoiler


--- ssh.service 2018-12-01 21:40:36.728028572 +0800
+++ ssh.service.fixup 2018-12-01 21:41:50.837422407 +0800
@@ -1,6 +1,8 @@
 [Unit]
 Description=OpenBSD Secure Shell server
-After=network.target auditd.service
+#[begin] Ansible Managed: Add dependency on tmpfiles-setup
+After=network.target auditd.service systemd-tmpfiles-setup.service
+#[end] Ansible Managed: Add dependency on tmpfiles-setup
 ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

 [Service]

Spoiler above can be applied with GNU 'patch', on '/lib/systemd/system/ssh.service'. systemd needs a 'systemctl deamon-reload' after changing the service configuration, as the dependencies have changed.

 

 

Spoiler

Spoiler below is not in unified diff format and has to be applied to '/lib/systemd/system/systemd-tmpfiles-setup.service' manually.



[Unit]
Description=Create Volatile Files and Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)
DefaultDependencies=no
Conflicts=shutdown.target
After=local-fs.target systemd-sysusers.service
#[disabled] Before=sysinit.target shutdown.target
#[begin]
Before=sysinit.target shutdown.target sshd.service
#[end]
RefuseManualStop=yes


 

Share this post


Link to post
Share on other sites

@thexman

 

thanks a lot for your downgrade instruction! you saved me hours of trial and error!!!

 

btw: you wrote earlier that you upgraded to the new kernel but then downgraded again. is this still the current situation or did you overcome all issues you had with the next kernel?

Share this post


Link to post
Share on other sites
3 hours ago, StefanH. said:

@thexman

 

thanks a lot for your downgrade instruction! you saved me hours of trial and error!!!

 

btw: you wrote earlier that you upgraded to the new kernel but then downgraded again. is this still the current situation or did you overcome all issues you had with the next kernel?

 

@StefanH.

You are very welcome! I did quite a few tests (including switching to mainline but dropped it in favour of the latest legacy). I upgraded the kernel and this also fixed the issue for me permanently without the need of downgrading "systemd". I am currently on ARMBIAN 5.73 stable Ubuntu 16.04.6 LTS 3.4.113-sun8i but I will now upgrade to 5.74 as I saw there is a newer version. You can upgrade your kernel version via "sudo armbian-config" => System => Other. There you can choose the kernel you want to switch to. 4.x.x is mainline kernel and might break specific legacy board features so I would recommend 3.4.113 with version 5.73 or 5.74. But 5.73 already fixed the problem so it is up to you what version you want to choose.

 

Update: strangely the update to 5.74 worked and after that the tool showed me 5.75 which also worked but motd still shows me 5.73. IDK what's is going on here. But I can confirm that 5.73 - 5.75 do work.

Share this post


Link to post
Share on other sites

I've upgraded as well (apt-get update / upgrade). Was running on old 5.38

Now /etc/armbian-release points now to 5.73 although dpkg -l lists packages like:

 

ii  linux-image-sun8i                           5.75                       armhf                      Linux kernel, version 3.4.113-sun8i

reboot (especially systemd-tmpfiles-setup.service) worked fine and services like SSH were started successfully.

The official documentation states that the systemd bug was solved in 5.74

https://docs.armbian.com/Release_Changelog/

 

Once more, thanks!

Share this post


Link to post
Share on other sites

@StefanH.

I think the issue was already fixed with 5.73 as I wrote that in an earlier posts. This is a mistake in the docs I guess. I checked myself /etc/armbian-release shows 5.73 and the image shows 5.75 so this might be a bug :D. I checked all my services yesterday and none failed on boot up with 5.75.

 

no problem ;). I thought you already used apt to update and the kernel wasn't updated for some reason. Therefore I pointed on the armbian-config method. Typically apt does it. 5.74/5.75 however was not updated on my device. Or it was missing because auto updates run once per week on my machine ;)

Share this post


Link to post
Share on other sites

Thank you very much, vfrolov! I've suffered from the error messages on boot for long time and occasionally found your solution that worked for me.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
7 7