vfrolov Posted November 22, 2018 Posted November 22, 2018 motion, apache, sshd, openvpn, fail2ban, apcupsd, smbd/nmbd, screen - everything is fine
smuschevich Posted November 23, 2018 Posted November 23, 2018 9 hours ago, vfrolov said: did it! # apt list | grep systemd |grep "[" libpam-systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] libsystemd0/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] systemd-sysv/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] # apt list -a systemd systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] systemd/xenial,xenial 229-4ubuntu4 armhf # apt-get install libpam-systemd=229-4ubuntu4 libsystemd0=229-4ubuntu4 systemd=229-4ubuntu4 systemd-sysv=229-4ubuntu4 Could you please share exact steps you did that?
RenatoLorenzi Posted November 23, 2018 Posted November 23, 2018 I have not found an upstream solution yet. Has anyone found anything different? https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1804603
hajupiko49 Posted November 25, 2018 Posted November 25, 2018 I ask for help! First of all, I do not speak English and have had the translation done by google. My hardware = orangepipcplus, my system is Armbian_5.59_Orangepipcplus_Ubuntu_xenial_default_3.4.113_desktop. The system is loaded in the eMMC. After a firmware update & upgrade (from armbian-config) or also sudo apt update & upgrade, it is no longer possible to connect via SSH from the WIN7 computer to the OPI. I have made several attempts. It's always the same. The connection to the network is rejected. The connection via XRDP is still possible. What can I do to reuse the SSH connection after the update & upgrade? Yours sincerely hajupiko49
vfrolov Posted November 25, 2018 Posted November 25, 2018 hi, smuschevich. sorry for inconvenience. The basic idea is simple: return the package that worked before the updates 1. we need to find out which packages from the "SYSTEMD" family are currently installed on the computer 2. what was the previous version(s) that worked, unlike the new one 3. and finally, install packages of the correct version The first task is performed by the command: apt list | grep systemd | grep "\[" where: apt list --- gives the full list of packages of your distribution grep systemd --- selects only packages containing the word "systemd" in the name >>: you may use the command "apt list *systemd*" instead of apt list | grep systemd grep "\[" --- selects only packages containing special labels, for example, "[installed]" in my case the output was: libpam-systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] libsystemd0/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] systemd-sysv/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] [установлен] == [installed] it shows that I'm interested in 4 packages: libpam-systemd, libsystemd0, systemd, systemd-sysv, - these are all of the seme version - very good! The seccond task is performed by the command: apt list -a systemd in my case the output was: systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] systemd/xenial,xenial 229-4ubuntu4 armhf I assumed that the previous version was "229-4ubuntu4" The third task is performed by the command: apt-get install libpam-systemd=229-4ubuntu4 libsystemd0=229-4ubuntu4 systemd=229-4ubuntu4 systemd-sysv=229-4ubuntu4 Do not forget to pray hard during the third step! Good luck :-))
vfrolov Posted November 25, 2018 Posted November 25, 2018 6 hours ago, hajupiko49 said: I ask for help! First of all, I do not speak English and have had the translation done by google. My hardware = orangepipcplus, my system is Armbian_5.59_Orangepipcplus_Ubuntu_xenial_default_3.4.113_desktop. The system is loaded in the eMMC. After a firmware update & upgrade (from armbian-config) or also sudo apt update & upgrade, it is no longer possible to connect via SSH from the WIN7 computer to the OPI. I have made several attempts. It's always the same. The connection to the network is rejected. The connection via XRDP is still possible. What can I do to reuse the SSH connection after the update & upgrade? Yours sincerely hajupiko49 1. Use XRDP or HDMI monitor, as well as the USB keyboard and mouse 2. launch the terminal window using the mouse, or the keyboard by pressing Control+Alt+T 3. Execute commands: sudo mkdir /var/run/sshd sudo service sshd start But the problem is much wider than what you have had encountered with
smuschevich Posted November 26, 2018 Posted November 26, 2018 Hi, vfrolov, thank you very much for your answer! In the end I decided to install Armbian Bionic - all problems have gone.
hajupiko49 Posted November 26, 2018 Posted November 26, 2018 Herzlichen Dank für den Ratschlag, nun kann ich wieder meine ssh Verbindung nutzen. Nach einem Neustart war es wieder wie zuvor mit der Fehlermeldung. Nach der erneuten Eingabe der Befehlsfolge stand die SSH Verbindung wieder. Da mir das zu umständlich ist, habe ich die zwei Zeilen in meine rc.local eingefügt. Nun kann ich die SSH Verbindung auch nach einem Neustart sofort verwenden. Damit entfällt eine Neuinstallation des Systems was mir auch sehr angenehm ist. Ich habe ja einen Tag zu arbeiten um alle Programme wieder im eMMC zu haben. Mit freundlichen Grüßen hajupiko49 Thanks for the advice, now I can use my ssh connection again. After a restart, it was again as before with the error message. After re-entering the command sequence, the SSH connection was restored. Since this is too complicated for me, I have inserted the two lines into my rc.local. Now I can use the SSH connection immediately after a reboot. This eliminates a reinstallation of the system which is also very pleasant to me. I have one day to work to have all programs back in the eMMC. Yours sincerely hajupiko49
vfrolov Posted November 26, 2018 Posted November 26, 2018 8 hours ago, smuschevich said: Hi, vfrolov, thank you very much for your answer! In the end I decided to install Armbian Bionic - all problems have gone. have you done upgrade after fresh install? what version of systemd is installed now? apt list systemd thank you
chocomega Posted November 27, 2018 Posted November 27, 2018 On 11/25/2018 at 6:28 PM, vfrolov said: hi, smuschevich. sorry for inconvenience. The basic idea is simple: return the package that worked before the updates 1. we need to find out which packages from the "SYSTEMD" family are currently installed on the computer 2. what was the previous version(s) that worked, unlike the new one 3. and finally, install packages of the correct version The first task is performed by the command: apt list | grep systemd | grep "[" where: apt list --- gives the full list of packages of your distribution grep systemd --- selects only packages containing the word "systemd" in the name >>: you may use the command "apt list *systemd*" instead of apt list | grep systemd grep "[" --- selects only packages containing special labels, for example, "[installed]" in my case the output was: libpam-systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] libsystemd0/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] systemd-sysv/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] [установлен] == [installed] it shows that I'm interested in 4 packages: libpam-systemd, libsystemd0, systemd, systemd-sysv, - these are all of the seme version - very good! The seccond task is performed by the command: apt list -a systemd in my case the output was: systemd/xenial-security,xenial-updates,xenial-security,xenial-updates,now 229-4ubuntu21.9 armhf [установлен] systemd/xenial,xenial 229-4ubuntu4 armhf I assumed that the previous version was "229-4ubuntu4" The third task is performed by the command: apt-get install libpam-systemd=229-4ubuntu4 libsystemd0=229-4ubuntu4 systemd=229-4ubuntu4 systemd-sysv=229-4ubuntu4 Do not forget to pray hard during the third step! Good luck :-)) Thank you very much for the instructions, I had the exact same issue ! My first thought was an SD card issue or even a hacker attack ! I spent the whole night trying to figure it out... It's a shame that such update from Ubuntu breaks the system especially when ssh is my only remote access to the headless orange pi
vfrolov Posted November 27, 2018 Posted November 27, 2018 7 hours ago, chocomega said: Thank you very much for the instructions, I had the exact same issue ! My first thought was an SD card issue or even a hacker attack ! I spent the whole night trying to figure it out... It's a shame that such update from Ubuntu breaks the system especially when ssh is my only remote access to the headless orange pi thank you. please tell us, what board & OS version/release have you got
vfrolov Posted November 27, 2018 Posted November 27, 2018 could anyone confirm that mew version of SYSTEMD (229-4ubuntu21.10) works well ??
José Roberto Canuto Posted November 27, 2018 Posted November 27, 2018 8 minutes ago, vfrolov said: could anyone confirm that mew version of SYSTEMD (229-4ubuntu21.10) works well ?? no, it doesn't, i used apt-get upgrade today. after an hour of frustration trying to fix SSH I found this post, lesson learned though, never update what is already working
smuschevich Posted November 27, 2018 Posted November 27, 2018 On 11/26/2018 at 11:19 PM, vfrolov said: have you done upgrade after fresh install? what version of systemd is installed now? apt list systemd thank you # apt list systemd Listing... Done systemd/bionic-security,bionic-updates 237-3ubuntu10.9 armhf [upgradable from: 237-3ubuntu10.3]
Sann-X! Posted November 28, 2018 Posted November 28, 2018 Workaround for ssh issue (create one shot service/daemon to make proper sshd dir): 0. Log in under root 1. Create file: touch /etc/systemd/system/create-ssh-dir.service 2. Grant access: chmod 664 /etc/systemd/system/create-ssh-dir.service 3. Edit content file: [Unit] Description=Creates sshd directory Before=ssh.service [Service] Type = oneshot ExecStart=mkdir -p /var/run/sshd [Install] WantedBy=multi-user.target 4. Enable service: systemctl enable create-ssh-dir.service 5. Reboot 1
RenatoLorenzi Posted November 29, 2018 Posted November 29, 2018 Now I received a update from 229-4ubuntu21.9 to 229-4ubuntu21.10 and everything worked again.
vfrolov Posted December 2, 2018 Posted December 2, 2018 On 11/29/2018 at 9:34 PM, RenatoLorenzi said: Now I received a update from 229-4ubuntu21.9 to 229-4ubuntu21.10 and everything worked again. I Confirm. systemd 229-4ubuntu21.10 works again. Darkness Falls.
Bororo Posted December 15, 2018 Posted December 15, 2018 I have orange pi zero with Domoticz and OPi Monitor installed. Unfortunately after some upgrade SSH access broke. Each connection attempt ends up with message saying that connection is actively refused. When open OPI Monitor I can see armbian updates waiting for update, but as I lost ssh access I can't do this. Is there any other way how to get access to device and make update?
martinayotte Posted December 15, 2018 Posted December 15, 2018 3 hours ago, Bororo said: Unfortunately after some upgrade SSH access broke. Each connection attempt ends up with message saying that connection is actively refused Do you have a USB-TTL Serial dongle ? Log yourself thru that, and then check if you have those files : -rw------- 1 root root 227 Nov 13 17:54 /etc/ssh/ssh_host_ecdsa_key -rw-r--r-- 1 root root 181 Nov 13 17:54 /etc/ssh/ssh_host_ecdsa_key.pub -rw------- 1 root root 411 Nov 13 17:54 /etc/ssh/ssh_host_ed25519_key -rw-r--r-- 1 root root 101 Nov 13 17:54 /etc/ssh/ssh_host_ed25519_key.pub -rw------- 1 root root 1675 Nov 13 17:54 /etc/ssh/ssh_host_rsa_key -rw-r--r-- 1 root root 401 Nov 13 17:54 /etc/ssh/ssh_host_rsa_key.pub if you don't have them, then execute "ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key" and do about the equivalent for "ecdsa" and "ed25519" ...
ning Posted January 4, 2019 Posted January 4, 2019 https://forum.armbian.com/topic/8933-warninglegacy-image-failed-to-start-sshd-after-system-upgrade/?tab=comments#comment-66872
vfrolov Posted January 11, 2019 Posted January 11, 2019 today after upgrading systemd package to the version "229-4ubuntu21.15" (on 32-bit armbian) sshd and some other system services are not started again. after downgrade systemd package to version "229-4ubuntu21.11" everything is workig
chocomega Posted January 12, 2019 Posted January 12, 2019 Thanks for letting us know...what's wrong with Ubuntu ?? I've put on hold these packages for now with the following command: apt-mark hold libsystemd0 libpam-systemd systemd systemd-sysv I hope they get this sorted out especially with the recent vulnerabilities discovered in systemd !
mirko Posted January 14, 2019 Posted January 14, 2019 I made today the update from 5.60 to 5.70 and I do not have access to ssh. I do not have usb - ttl or hdmi (orange pi zero) so how can I fix ssh?
Werner Posted January 14, 2019 Posted January 14, 2019 1 hour ago, mirko said: I made today the update from 5.60 to 5.70 and I do not have access to ssh. I do not have usb - ttl or hdmi (orange pi zero) so how can I fix ssh? Either get an USB TTL adapter which is very useful anyways and is very cheap or put the SD card in your pc, chroot into the fs and fix it there like for example as suggested by downgrading particular packages.
mirko Posted January 14, 2019 Posted January 14, 2019 14 minutes ago, Werner said: Either get an USB TTL adapter which is very useful anyways and is very cheap or put the SD card in your pc, chroot into the fs and fix it there like for example as suggested by downgrading particular packages. thanks, I finally used arduino as a ttl usb converter, I changed the versions back to 5.60 and everything works now I have a lesson to make a backup before every update
thexman Posted January 14, 2019 Posted January 14, 2019 Hello Community, This is going to be a longer post because I want to summarize everything I found out and I will also attach useful resources and additions to @vfrolov tutorial. E.g. on how to get back to a working / previous state with a certain release version. I also experienced the issue discussed here on my OrangePI One with ARMBIAN 5.70 stable Ubuntu 16.04.5 LTS 3.4.113-sun8i myself. The problems first occurred with systemd-229-4ubuntu21.9 affecting several services (e.g. ssh, dnsmasq, lighttpd, openvpn,...) that on my little server refused to start - full list of affected services are listed below. With the next version of systemd-229-4ubuntu21.10 the issue was solved through a regression release. Since systemd-229-4ubuntu21.15 is out the problems finally occurs again in the same severity on my server. I have no in deep knowledge why this actually occurs or if this will ever be fixed again. However you can get more info in the resources (bug reports) I listed at the end of this post. I added more information on my actual case there and I also invite you to state your issues there so we have a better chance to get some response or fix. Basically, the origin of the issue seems to be related to systemd-tmpfiles which has changed in 21.9 and 21.15 and now breaks volatile path creation for certain services that are started on boot up and depend on these paths. See also my journalctl listing below which is a tiny snapshot of paths not accessible and prevent services to start correctly. Based on my findings the issue is caused by Ubuntu itself, however it's not clear if a next release will address the problem and what Distros are actually affected. It has already been discussed above on how to downgrade to systemd-229-4ubuntu4. However this is an quite old version of systemd (~2016) and obviously therefore not the best solution. It will work but you will skip over two years of updates. Sadly, through the repos and automatic apt updates you can only update to systemd-229-4ubuntu21.15 or downgrade to systemd-229-4ubuntu4. As @vfrolov already stated we need four packages to fix the problem with systemd. Below are the links to the respective Launchpad package repositories where you can find the downloads for all revisions of the respective armhf packages: libpam-systemd libsystemd0 systemd systemd-sysv Based on that I took my time and searched for both systemd-229-4ubuntu21.10 and systemd-229-4ubuntu21.11 all the necessary install resources. Hence I combined all necessary package download paths and created two easy commands per release to downgrade a system on either of the two versions (both working and both fixing the issue for me). Downgrade to systemd-229-4ubuntu-21.10: Navigate to any folder and execute each of this two commands below. 1. Download packages, 2. install packages manually. wget http://launchpadlibrarian.net/399247908/libpam-systemd_229-4ubuntu21.10_armhf.deb | wget http://launchpadlibrarian.net/399247910/libsystemd0_229-4ubuntu21.10_armhf.deb | wget http://launchpadlibrarian.net/399247918/systemd_229-4ubuntu21.10_armhf.deb | wget http://launchpadlibrarian.net/399247917/systemd-sysv_229-4ubuntu21.10_armhf.deb sudo dpkg -i libpam-systemd_229-4ubuntu21.10_armhf.deb libsystemd0_229-4ubuntu21.10_armhf.deb systemd_229-4ubuntu21.10_armhf.deb systemd-sysv_229-4ubuntu21.10_armhf.deb Downgrade to systemd-229-4ubuntu-21.11: And again, choose any folder and execute both commands below. wget http://launchpadlibrarian.net/401375344/libpam-systemd_229-4ubuntu21.11_armhf.deb | wget http://launchpadlibrarian.net/401375346/libsystemd0_229-4ubuntu21.11_armhf.deb | wget http://launchpadlibrarian.net/401375354/systemd_229-4ubuntu21.11_armhf.deb | wget http://launchpadlibrarian.net/401375353/systemd-sysv_229-4ubuntu21.11_armhf.deb sudo dpkg -i libpam-systemd_229-4ubuntu21.11_armhf.deb libsystemd0_229-4ubuntu21.11_armhf.deb systemd_229-4ubuntu21.11_armhf.deb systemd-sysv_229-4ubuntu21.11_armhf.deb You can also generate a bash script to download, update and remove package files automatically in case you want to try a future release and then rollback to a working state in case the issue still perists. That is no magic at all so I won't attach any resources for that here. However if someone is interested I can upload a bash script for both version for easy one line execution. Update: I decided to create a repo for the sake of completeness which includes the bash files and instruction on how to use them: systemd-downgrades on Github. My Ubuntu Version: Distributor ID: Ubuntu Description: Ubuntu 16.04.5 LTS Release: 16.04 Codename: xenial My Kernel: Linux pan 3.4.113-sun8i #2 SMP PREEMPT Sat Jan 12 15:54:26 CET 2019 armv7l armv7l armv7l GNU/Linux Output of journalctl -b 0 -u systemd-tmpfiles-setup.service: Jän 14 11:01:51 xxx systemd[1]: Starting Create Volatile Files and Directories... Jän 14 11:01:51 xxx systemd-tmpfiles[581]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/log: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/lib: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/sendsigs.omit.d: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /home: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /srv: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/lock/subsys: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/run/lighttpd: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/cache: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/cache/man: Bad file descriptor Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/openvpn: Bad file descriptor Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE Jän 14 11:01:51 xxx systemd[1]: Failed to start Create Volatile Files and Directories. Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state. Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'. Affected services: # dnsmasq.service loaded failed failed dnsmasq - A lightweight DHCP and caching DNS server # lighttpd.service loaded failed failed Lighttpd Daemon # <email address hidden> loaded failed failed OpenVPN connection to server # ssh.service loaded failed failed OpenBSD Secure Shell server # systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device Nodes in /dev # systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories Resources related to the problem: Launchpad - 1804847 Launchpad - 1804603 Launchpad - 1811580
thexman Posted January 15, 2019 Posted January 15, 2019 (edited) I finally upgraded to the mainline Kernel Linux pan 4.19.13-sunxi #5.70 SMP Sat Jan 12 15:43:21 CET 2019 armv7l armv7l armv7l GNU/Linux like described here and the issue is now also gone for me even with systemd-229-4ubuntu21.15 However I have now an issue with the service systemd-modules-load.service loaded failed failed Load Kernel Modules which won't start I don't know what the culprit of the issue is and what effects it has. Edited January 15, 2019 by thexman typo
Igor Posted January 16, 2019 Posted January 16, 2019 8 hours ago, thexman said: which won't start I don't know what the culprit of the issue is and what effects it has. This could be safe to ignore, but it would be nice to peek into the logs to see which module(s) are not getting loaded. sudo journalctl -u systemd-modules-load.service
BlackWoxs Posted January 16, 2019 Posted January 16, 2019 @thexman thank you very much for your work and the batch files on GitHub. Thanks to these I am now back and running fine under systemd-229-4ubuntu-21.11, without having to resort to non-standard patching/config. Much appreciated.
Recommended Posts