5 5
GeorgeL

SSH doesn't work on Orange Pi Zero

Recommended Posts

9 hours ago, vfrolov said:

did it!

 

# apt list | grep systemd |grep "["

libpam-systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
libsystemd0/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
systemd-sysv/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]


# apt list -a systemd

systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
systemd/xenial,xenial 229-4ubuntu4 armhf

# apt-get install libpam-systemd=229-4ubuntu4 libsystemd0=229-4ubuntu4 systemd=229-4ubuntu4 systemd-sysv=229-4ubuntu4
 

Could you please share exact steps you did that?

Share this post


Link to post
Share on other sites

I ask for help!
First of all, I do not speak English and have had the translation done by google.
My hardware = orangepipcplus, my system is Armbian_5.59_Orangepipcplus_Ubuntu_xenial_default_3.4.113_desktop.
The system is loaded in the eMMC.

After a firmware update & upgrade (from armbian-config) or also sudo apt update & upgrade, it is no longer possible to connect via SSH from the WIN7 computer to the OPI. I have made several attempts. It's always the same. The connection to the network is rejected. The connection via XRDP is still possible.

What can I do to reuse the SSH connection after the update & upgrade?
Yours sincerely
hajupiko49

Share this post


Link to post
Share on other sites

hi, smuschevich. sorry for inconvenience.

 

The basic idea is simple: return the package that worked before the updates

1. we need to find out which packages from the "SYSTEMD" family are currently installed on the computer

2. what was the previous version(s) that worked, unlike the new one

3. and finally, install packages of the correct version

 

The first task is performed by the command:

apt list | grep systemd | grep "\["

where:

  apt list --- gives the full list of packages of your distribution

  grep systemd --- selects only packages containing the word "systemd" in the name

  >>: you may use the command "apt list *systemd*" instead of apt list | grep systemd 

  grep "\[" --- selects only packages containing special labels, for example, "[installed]"

in my case the output was:

  libpam-systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
  libsystemd0/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
  systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
  systemd-sysv/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]

[установлен] == [installed]

it shows that I'm interested in 4 packages: libpam-systemd,  libsystemd0, systemd, systemd-sysv, - these are all of the seme version - very good!

 

The seccond task is performed by the command:

apt list -a systemd

in my case the output was:

  systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
  systemd/xenial,xenial 229-4ubuntu4 armhf

I assumed that the previous version was "229-4ubuntu4"

 

The third task is performed by the command:

apt-get install libpam-systemd=229-4ubuntu4 libsystemd0=229-4ubuntu4 systemd=229-4ubuntu4 systemd-sysv=229-4ubuntu4

 

Do not forget to pray hard during the third step!  Good luck :-))

Share this post


Link to post
Share on other sites
6 hours ago, hajupiko49 said:

I ask for help!
First of all, I do not speak English and have had the translation done by google.
My hardware = orangepipcplus, my system is Armbian_5.59_Orangepipcplus_Ubuntu_xenial_default_3.4.113_desktop.
The system is loaded in the eMMC.

After a firmware update & upgrade (from armbian-config) or also sudo apt update & upgrade, it is no longer possible to connect via SSH from the WIN7 computer to the OPI. I have made several attempts. It's always the same. The connection to the network is rejected. The connection via XRDP is still possible.

What can I do to reuse the SSH connection after the update & upgrade?
Yours sincerely
hajupiko49

1. Use XRDP or HDMI monitor, as well as the USB keyboard and mouse

2. launch the terminal window using the mouse, or the keyboard by pressing Control+Alt+T

3. Execute commands:

    sudo mkdir /var/run/sshd

    sudo service sshd start

 

But the problem is much wider than what you have had encountered with

Share this post


Link to post
Share on other sites

Herzlichen Dank für den Ratschlag, nun kann ich wieder meine ssh Verbindung nutzen. Nach einem Neustart war es wieder wie zuvor mit der Fehlermeldung. Nach der erneuten Eingabe der Befehlsfolge stand die SSH Verbindung wieder.

Da mir das zu umständlich ist, habe ich die zwei Zeilen in meine rc.local eingefügt. Nun kann ich die SSH Verbindung auch nach einem Neustart sofort verwenden.

Damit entfällt eine Neuinstallation des Systems was mir auch sehr angenehm ist. Ich habe ja einen Tag zu arbeiten um alle Programme wieder im eMMC zu haben.

Mit freundlichen Grüßen

hajupiko49

 

Thanks for the advice, now I can use my ssh connection again. After a restart, it was again as before with the error message. After re-entering the command sequence, the SSH connection was restored.
Since this is too complicated for me, I have inserted the two lines into my rc.local. Now I can use the SSH connection immediately after a reboot.
This eliminates a reinstallation of the system which is also very pleasant to me. I have one day to work to have all programs back in the eMMC.
Yours sincerely
hajupiko49

Share this post


Link to post
Share on other sites
8 hours ago, smuschevich said:

Hi, vfrolov, thank you very much for your answer!

 

In the end I decided to install Armbian Bionic - all problems have gone.

have you done upgrade after fresh install?

what version of systemd is installed now?

apt list systemd

 

thank you

Share this post


Link to post
Share on other sites
On 11/25/2018 at 6:28 PM, vfrolov said:

hi, smuschevich. sorry for inconvenience.

 

The basic idea is simple: return the package that worked before the updates

1. we need to find out which packages from the "SYSTEMD" family are currently installed on the computer

2. what was the previous version(s) that worked, unlike the new one

3. and finally, install packages of the correct version

 

The first task is performed by the command:

apt list | grep systemd | grep "["

where:

  apt list --- gives the full list of packages of your distribution

  grep systemd --- selects only packages containing the word "systemd" in the name

  >>: you may use the command "apt list *systemd*" instead of apt list | grep systemd 

  grep "[" --- selects only packages containing special labels, for example, "[installed]"

in my case the output was:

  libpam-systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
  libsystemd0/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
  systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
  systemd-sysv/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]

[установлен] == [installed]

it shows that I'm interested in 4 packages: libpam-systemd,  libsystemd0, systemd, systemd-sysv, - these are all of the seme version - very good!

 

The seccond task is performed by the command:

apt list -a systemd

in my case the output was:

  systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен]
  systemd/xenial,xenial 229-4ubuntu4 armhf

I assumed that the previous version was "229-4ubuntu4"

 

The third task is performed by the command:

apt-get install libpam-systemd=229-4ubuntu4 libsystemd0=229-4ubuntu4 systemd=229-4ubuntu4 systemd-sysv=229-4ubuntu4

 

Do not forget to pray hard during the third step!  Good luck :-))

 

 

Thank you very much for the instructions, I had the exact same issue ! My first thought was an SD card issue or even a hacker attack ! I spent the whole night trying to figure it out...

It's a shame that such update from Ubuntu breaks the system especially when ssh is my only remote access to the headless orange pi

Share this post


Link to post
Share on other sites
7 hours ago, chocomega said:

 

Thank you very much for the instructions, I had the exact same issue ! My first thought was an SD card issue or even a hacker attack ! I spent the whole night trying to figure it out...

It's a shame that such update from Ubuntu breaks the system especially when ssh is my only remote access to the headless orange pi

thank you.

please tell us, what board & OS version/release have you got

Share this post


Link to post
Share on other sites
8 minutes ago, vfrolov said:

could anyone confirm that mew version of SYSTEMD (229-4ubuntu21.10) works well ??

no, it doesn't, i used apt-get upgrade today. after an hour of frustration trying to fix SSH I found this post,  lesson learned though, never update what is already working 

Share this post


Link to post
Share on other sites
On 11/26/2018 at 11:19 PM, vfrolov said:

have you done upgrade after fresh install?

what version of systemd is installed now?

apt list systemd

 

thank you

# apt list systemd

Listing... Done

systemd/bionic-security,bionic-updates 237-3ubuntu10.9 armhf [upgradable from: 237-3ubuntu10.3]

Share this post


Link to post
Share on other sites

Workaround for ssh issue (create one shot service/daemon to make proper sshd dir):

0. Log in under root

1. Create file:

touch /etc/systemd/system/create-ssh-dir.service

 

2. Grant access:
chmod 664 /etc/systemd/system/create-ssh-dir.service


3. Edit content file:

[Unit]
Description=Creates sshd directory
Before=ssh.service

[Service]
Type = oneshot
ExecStart=mkdir -p /var/run/sshd

[Install]
WantedBy=multi-user.target

4. Enable service:
systemctl enable create-ssh-dir.service

 

5. Reboot

Share this post


Link to post
Share on other sites
On 11/29/2018 at 9:34 PM, RenatoLorenzi said:

Now I received a update from 229-4ubuntu21.9 to 229-4ubuntu21.10 and everything worked again.

I Confirm. systemd 229-4ubuntu21.10 works again. Darkness Falls.

Share this post


Link to post
Share on other sites

I have orange pi zero with Domoticz and OPi Monitor installed. Unfortunately after some upgrade SSH access broke. Each connection attempt ends up with message saying that connection is actively refused. When open OPI Monitor I can see armbian updates waiting for update, but as I lost ssh access I can't do this. Is there any other way how to get access to device and make update?

Share this post


Link to post
Share on other sites
3 hours ago, Bororo said:

Unfortunately after some upgrade SSH access broke. Each connection attempt ends up with message saying that connection is actively refused

Do you have a USB-TTL Serial dongle ? Log yourself thru that, and then check if you have those files :

-rw------- 1 root root    227 Nov 13 17:54 /etc/ssh/ssh_host_ecdsa_key
-rw-r--r-- 1 root root    181 Nov 13 17:54 /etc/ssh/ssh_host_ecdsa_key.pub
-rw------- 1 root root    411 Nov 13 17:54 /etc/ssh/ssh_host_ed25519_key
-rw-r--r-- 1 root root    101 Nov 13 17:54 /etc/ssh/ssh_host_ed25519_key.pub
-rw------- 1 root root   1675 Nov 13 17:54 /etc/ssh/ssh_host_rsa_key
-rw-r--r-- 1 root root    401 Nov 13 17:54 /etc/ssh/ssh_host_rsa_key.pub

if you don't have them, then execute "ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key" and do about the equivalent for "ecdsa" and "ed25519" ...

Share this post


Link to post
Share on other sites

today after upgrading systemd package to the version "229-4ubuntu21.15" (on 32-bit armbian) sshd and some other system services are not started again. after downgrade systemd package to version "229-4ubuntu21.11" everything is workig

Share this post


Link to post
Share on other sites

Thanks for letting us know...what's wrong with Ubuntu ?? :(

I've put on hold these packages for now with the following command:

apt-mark hold libsystemd0 libpam-systemd systemd systemd-sysv

I hope they get this sorted out especially with the recent vulnerabilities discovered in systemd !

Share this post


Link to post
Share on other sites

I made today the update from 5.60 to 5.70 and I do not have access to ssh. I do not have usb - ttl or hdmi (orange pi zero) so how can I fix ssh? 

Share this post


Link to post
Share on other sites
1 hour ago, mirko said:

I made today the update from 5.60 to 5.70 and I do not have access to ssh. I do not have usb - ttl or hdmi (orange pi zero) so how can I fix ssh? 

Either get an USB TTL adapter which is very useful anyways and is very cheap or put the SD card in your pc, chroot into the fs and fix it there like for example as suggested by downgrading particular packages.

Share this post


Link to post
Share on other sites
14 minutes ago, Werner said:

Either get an USB TTL adapter which is very useful anyways and is very cheap or put the SD card in your pc, chroot into the fs and fix it there like for example as suggested by downgrading particular packages.

thanks, I finally used arduino as a ttl usb converter, I changed the versions back to 5.60 and everything works

now I have a lesson to make a backup before every update

Share this post


Link to post
Share on other sites

Hello Community,

 

This is going to be a longer post because I want to summarize everything I found out and I will also attach useful resources and additions to vfrolovs tutorial. E.g. on how to get back to a working / previous state with a certain release version. I also experienced the issue discussed here on my OrangePI One with ARMBIAN 5.70 stable Ubuntu 16.04.5 LTS 3.4.113-sun8i myself.

 

The problems first occurred with systemd-229-4ubuntu21.9 affecting several services (e.g. ssh, dnsmasq, lighttpd, openvpn,...) that on my little server refused to start - full list of affected services are listed below. With the next version of systemd-229-4ubuntu21.10 the issue was solved through a regression release. Since systemd-229-4ubuntu21.15 is out the problems finally occurs again in the same severity on my server.

 

I have no in deep knowledge why this actually occurs or if this will ever be fixed again. However you can get more info in the resources (bug reports) I listed at the end of this post. I added more information on my actual case there and I also invite you to state your issues there so we have a better chance to get some response or fix. Basically, the origin of the issue seems to be related to systemd-tmpfiles which has changed in 21.9 and 21.15 and now breaks volatile path creation for certain services that are started on boot up and depend on these paths. See also my journalctl listing below which is a tiny snapshot of paths not accessible and prevent services to start correctly. Based on my findings the issue is caused by Ubuntu itself, however it's not clear if a next release will address the problem and what Distros are actually affected.

 

It has already been discussed above on how to downgrade to systemd-229-4ubuntu4. However this is an quite old version of systemd (~2016) and obviously therefore not the best solution. It will work but you will skip over two years of updates. Sadly, through the repos and automatic apt updates you can only update to systemd-229-4ubuntu21.15 or downgrade to systemd-229-4ubuntu4.

 

As vfrolov already stated we need four packages to fix the problem with systemd. Below are the links to the respective Launchpad package repositories where you can find the downloads for all revisions of the respective armhf packages:

 

libpam-systemd

libsystemd0

systemd

systemd-sysv

 

Based on that I took my time and searched for both systemd-229-4ubuntu21.10 and systemd-229-4ubuntu21.11 all the necessary install resources. Hence I combined all necessary package download paths and created two easy commands per release to downgrade a system on either of the two versions (both working and both fixing the issue for me).

 

Downgrade to systemd-229-4ubuntu-21.10:

Navigate to any folder and execute each of this two commands below. 1. Download packages, 2. install packages manually.

wget http://launchpadlibrarian.net/399247908/libpam-systemd_229-4ubuntu21.10_armhf.deb | wget http://launchpadlibrarian.net/399247910/libsystemd0_229-4ubuntu21.10_armhf.deb | wget http://launchpadlibrarian.net/399247918/systemd_229-4ubuntu21.10_armhf.deb | wget http://launchpadlibrarian.net/399247917/systemd-sysv_229-4ubuntu21.10_armhf.deb

sudo dpkg -i libpam-systemd_229-4ubuntu21.10_armhf.deb libsystemd0_229-4ubuntu21.10_armhf.deb systemd_229-4ubuntu21.10_armhf.deb systemd-sysv_229-4ubuntu21.10_armhf.deb

 

Downgrade to systemd-229-4ubuntu-21.11:

And again, choose any folder and execute both commands below.

wget http://launchpadlibrarian.net/401375344/libpam-systemd_229-4ubuntu21.11_armhf.deb | wget http://launchpadlibrarian.net/401375346/libsystemd0_229-4ubuntu21.11_armhf.deb | wget http://launchpadlibrarian.net/401375354/systemd_229-4ubuntu21.11_armhf.deb | wget http://launchpadlibrarian.net/401375353/systemd-sysv_229-4ubuntu21.11_armhf.deb

sudo dpkg -i libpam-systemd_229-4ubuntu21.11_armhf.deb libsystemd0_229-4ubuntu21.11_armhf.deb systemd_229-4ubuntu21.11_armhf.deb systemd-sysv_229-4ubuntu21.11_armhf.deb

 

You can also generate a bash script to download, update and remove package files automatically in case you want to try a future release and then rollback to a working state in case the issue still perists. That is no magic at all so I won't attach any resources for that here. However if someone is interested I can upload a bash script for both version for easy one line execution.

 

Update: I decided to create a repo for the sake of completeness which includes the bash files and instruction on how to use them: systemd-downgrades on Github.

 

My Ubuntu Version:

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial

 

My Kernel:

Linux pan 3.4.113-sun8i #2 SMP PREEMPT Sat Jan 12 15:54:26 CET 2019 armv7l armv7l armv7l GNU/Linux

 

Output of journalctl -b 0 -u systemd-tmpfiles-setup.service:

Jän 14 11:01:51 xxx systemd[1]: Starting Create Volatile Files and Directories...
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/log: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/lib: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/sendsigs.omit.d: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /home: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /srv: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/lock/subsys: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/run/lighttpd: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/cache: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/cache/man: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/openvpn: Bad file descriptor
Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Jän 14 11:01:51 xxx systemd[1]: Failed to start Create Volatile Files and Directories.
Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state.
Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.

 

Affected services:

# dnsmasq.service loaded failed failed dnsmasq - A lightweight DHCP and caching DNS server
# lighttpd.service loaded failed failed Lighttpd Daemon
# <email address hidden> loaded failed failed OpenVPN connection to server
# ssh.service loaded failed failed OpenBSD Secure Shell server
# systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device Nodes in /dev
# systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories

 

Resources related to the problem:

Launchpad - 1804847

Launchpad - 1804603

Launchpad - 1811580

Edited by thexman
Added link to bash scripts on Github

Share this post


Link to post
Share on other sites

I finally upgraded to the mainline Kernel

Linux pan 4.19.13-sunxi #5.70 SMP Sat Jan 12 15:43:21 CET 2019 armv7l armv7l armv7l GNU/Linux

like described here and the issue is now also gone for me even with systemd-229-4ubuntu21.15

However I have now an issue with the service

systemd-modules-load.service loaded failed failed Load Kernel Modules

which won't start I don't know what the culprit of the issue is and what effects it has.

Edited by thexman
typo

Share this post


Link to post
Share on other sites
8 hours ago, thexman said:

which won't start I don't know what the culprit of the issue is and what effects it has.


This could be safe to ignore, but it would be nice to peek into the logs to see which module(s) are not getting loaded.

sudo journalctl -u systemd-modules-load.service

 

Share this post


Link to post
Share on other sites

@thexman thank you very much for your work and the batch files on GitHub.

 

Thanks to these I am now back and running fine under systemd-229-4ubuntu-21.11, without having to resort to non-standard patching/config.

 

Much appreciated.

Share this post


Link to post
Share on other sites
5 5