ning Posted November 28, 2018 Share Posted November 28, 2018 due to systemd updates to a new version, maybe it's not compact with old kernel. after system upgrade, it fails to start sshd. after read code, insystemd-tmpfile chase_symlinks() return -EBADF from kernel, I guess legacy kernel not compact with newer systemd. Link to comment Share on other sites More sharing options...
Sergey_zh Posted January 12, 2019 Share Posted January 12, 2019 Hello, After upgrading the Linux - SSH stopped working. I tried reinstalling the OS and and it works fine before running apt upgrade. Board Orange pi Lite Armbian_5.65_Orangepilite_Ubuntu_xenial_default_3.4.113_desktop Spoiler Boot error messages: [FAILED] Failed to start Create Static Device Nodes in /dev. See 'systemctl status systemd-tmpfiles-setup.service' for details. [FAILED] Failed to start OpenBSD Secure Shell server. See 'systemctl status ssh.service' for details. $ systemctl status systemd-tmpfiles-setup.service ● systemd-tmpfiles-setup.service - Create Volatile Files and Directories Loaded: loaded (/lib/systemd/system/systemd-tmpfiles-setup.service; static; v Active: failed (Result: exit-code) since Sat 2019-01-12 17:15:12 UTC; 36min a Docs: man:tmpfiles.d(5) man:systemd-tmpfiles(8) Process: 539 ExecStart=/bin/systemd-tmpfiles --create --remove --boot --exclud Main PID: 539 (code=exited, status=1/FAILURE) $ systemctl status ssh.service ● ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enab Active: failed (Result: start-limit-hit) since Sat 2019-01-12 17:15:23 UTC; 2 Process: 1218 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=255) My knowledge of Linux is enough only to reinstall and not to update the OS (( Link to comment Share on other sites More sharing options...
Tido Posted January 12, 2019 Share Posted January 12, 2019 did you sesrch the Forum? read here : https://forum.armbian.com/topic/8933-warninglegacy-image-failed-to-start-sshd-after-system-upgrade/?tab=comments#comment-66872 Link to comment Share on other sites More sharing options...
olivluca Posted January 19, 2019 Share Posted January 19, 2019 Being hit by this issue today (though I'm not sure since other network services are starting normally and I don't have currently access to a console), I wonder why systemd is being offered as an upgrade if it's going to cause this problem. Link to comment Share on other sites More sharing options...
olivluca Posted January 19, 2019 Share Posted January 19, 2019 Do you think that blocking the update of systemd will solve the problem? Link to comment Share on other sites More sharing options...
olivluca Posted January 19, 2019 Share Posted January 19, 2019 Just found this thread that explains the problem and the possible solutions 1 Link to comment Share on other sites More sharing options...
Sergio Posted January 23, 2019 Share Posted January 23, 2019 On 1/12/2019 at 9:02 PM, Sergey_zh said: Hello, After upgrading the Linux - SSH stopped working. I tried reinstalling the OS and and it works fine before running apt upgrade. Board Orange pi Lite Armbian_5.65_Orangepilite_Ubuntu_xenial_default_3.4.113_desktop Reveal hidden contents Boot error messages: [FAILED] Failed to start Create Static Device Nodes in /dev. See 'systemctl status systemd-tmpfiles-setup.service' for details. [FAILED] Failed to start OpenBSD Secure Shell server. See 'systemctl status ssh.service' for details. $ systemctl status systemd-tmpfiles-setup.service ● systemd-tmpfiles-setup.service - Create Volatile Files and Directories Loaded: loaded (/lib/systemd/system/systemd-tmpfiles-setup.service; static; v Active: failed (Result: exit-code) since Sat 2019-01-12 17:15:12 UTC; 36min a Docs: man:tmpfiles.d(5) man:systemd-tmpfiles(8) Process: 539 ExecStart=/bin/systemd-tmpfiles --create --remove --boot --exclud Main PID: 539 (code=exited, status=1/FAILURE) $ systemctl status ssh.service ● ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enab Active: failed (Result: start-limit-hit) since Sat 2019-01-12 17:15:23 UTC; 2 Process: 1218 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=255) My knowledge of Linux is enough only to reinstall and not to update the OS (( Hi! Use the "UsePrivilegeSeparation no" parameter in the sshd_config file or create the required directory "/var/run/sshd" before starting the service by systemd script. Link to comment Share on other sites More sharing options...
Sergey_zh Posted January 24, 2019 Share Posted January 24, 2019 Thank you Sergio, this solution works perfectly! Link to comment Share on other sites More sharing options...
ning Posted January 27, 2019 Author Share Posted January 27, 2019 https://github.com/systemd/systemd/blob/master/src/basic/fs-util.c#L804-L815 it looks like error in this part!! fd = open("/", O_CLOEXEC|O_NOFOLLOW|O_PATH); fstat(fd, &previous_stat) returns error, and errno is EBADF I don't understand. why? when I try to use O_PATH with open, build error. interesting. Link to comment Share on other sites More sharing options...
ning Posted January 27, 2019 Author Share Posted January 27, 2019 in kernel 3.4.113 file_table.c: fget/fget_light will check if (!(file->f_mode & FMODE_PATH)) else return NULL; need to patch this part to enable fstat on PATH fds. Link to comment Share on other sites More sharing options...
ning Posted January 27, 2019 Author Share Posted January 27, 2019 while mainline kernel doesn't check FMODE_PATH, in fstat call path. other path still check FMODE_PATH. I have an idea to fix this by patch sunxi8i kernel split fget/fget_light to __fget(fd, mode)/__fget_light(fd, mode) and old fget == __fget(fd, FMODE_PATH), fget_light == __fget_light(fd, FMODE_PATH) on fstat call path, use __fget_light(fd,0) but I don't have build env, so past my idea here. Link to comment Share on other sites More sharing options...
Sergio Posted January 29, 2019 Share Posted January 29, 2019 How to disable all automatic updates in ARMBIAN 5.38 stable Ubuntu 16.04.4 LTS 3.4.113-sun8i on Orange PC Plus? First, ssh stopped working, then lighttpd and xrdp !!! Link to comment Share on other sites More sharing options...
Sergio Posted January 29, 2019 Share Posted January 29, 2019 2 hours ago, Sergio said: How to disable all automatic updates in ARMBIAN 5.38 stable Ubuntu 16.04.4 LTS 3.4.113-sun8i on Orange PC Plus? First, ssh stopped working, then lighttpd and xrdp !!! He asked, he answered) 1. sudo nano /etc/apt/apt.conf.d/02periodic // Enable the update/upgrade script (0=disable) APT::Periodic::Enable "0"; // Do "apt-get update" automatically every n-days (0=disable) APT::Periodic::Update-Package-Lists "0"; // Do "apt-get upgrade --download-only" every n-days (0=disable) APT::Periodic::Download-Upgradeable-Packages "0"; // Run the "unattended-upgrade" security upgrade script // every n-days (0=disabled) // Requires the package "unattended-upgrades" and will write // a log in /var/log/unattended-upgrades APT::Periodic::Unattended-Upgrade "0"; // Do "apt-get autoclean" every n-days (0=disable) APT::Periodic::AutocleanInterval "0"; 2. sudo nano /etc/apt/apt.conf.d/50unattended-upgrades // Automatically upgrade packages from these (origin:archive) pairs Unattended-Upgrade::Allowed-Origins { // "${distro_id}:${distro_codename}"; // "${distro_id}:${distro_codename}-security"; // Extended Security Maintenance; doesn't necessarily exist for // every release and this system may not have it installed, but if // available, the policy for updates is such that unattended-upgrades // should also install from here by default. // "${distro_id}ESM:${distro_codename}"; // "${distro_id}:${distro_codename}-updates"; // "${distro_id}:${distro_codename}-proposed"; // "${distro_id}:${distro_codename}-backports"; }; Link to comment Share on other sites More sharing options...
Igor Posted January 29, 2019 Share Posted January 29, 2019 This not exactly a solution. Next manual apt update and upgrade will break it again due to the bug in systemd. In a package that is outside our control. We could only provide a hack via our repository. Link to comment Share on other sites More sharing options...
ning Posted January 29, 2019 Author Share Posted January 29, 2019 46 minutes ago, Igor said: This not exactly a solution. Next manual apt update and upgrade will break it again due to the bug in systemd. In a package that is outside our control. We could only provide a hack via our repository. do you mean fix/update legacy kernel is not a solution? Link to comment Share on other sites More sharing options...
Sergio Posted January 29, 2019 Share Posted January 29, 2019 55 minutes ago, Igor said: This not exactly a solution. Next manual apt update and upgrade will break it again due to the bug in systemd. In a package that is outside our control. We could only provide a hack via our repository. What kind of hack are we talking about? Link to comment Share on other sites More sharing options...
ning Posted January 29, 2019 Author Share Posted January 29, 2019 @Igor same question, what kind of hack? do you see my PR on github? Link to comment Share on other sites More sharing options...
Igor Posted January 29, 2019 Share Posted January 29, 2019 44 minutes ago, ning said: do you mean fix/update legacy kernel is not a solution? No no, don't saying that. I didn't get there, too busy. I hope someone else will cover that up. Only pointing out that @Sergio solution wont help. Link to comment Share on other sites More sharing options...
scaevola Posted January 30, 2019 Share Posted January 30, 2019 The most practical solution (although not perfect, because it requires reversing the systemd version) was proposed by @thexman in this thread: https://forum.armbian.com/topic/8852-ssh-doesnt-work-on-orange-pi-zero/?page=3 Just clone his git script, run it and halt systemd updates. This will save your machine from problems with ssh and other services until this would be repaired in ubuntu. 1 Link to comment Share on other sites More sharing options...
thexman Posted January 30, 2019 Share Posted January 30, 2019 @scaevola Thanks for mentioning me. Here is my script on Github to automatically dowgrade systemd to latest functioning release, just in case that someone is interested or wants to try or use it ;). No need to dig in the original thread. The readme there also describes how to use it. Upgrading to mainline kernel also works, I have tried that the other day. But be aware that some applications especially crafted for legacy might break. 1 Link to comment Share on other sites More sharing options...
Sergio Posted January 30, 2019 Share Posted January 30, 2019 @Igor , will it help to avoid this in the future if I migrate to Debian GNU / Linux 9 (stretch) 4.19.17-sunxi? Link to comment Share on other sites More sharing options...
thexman Posted January 30, 2019 Share Posted January 30, 2019 @Sergio You don't inevitably need to switch to Stretch to avoid the issue. Simply switching to mainline on Xenial will also work. As already said legacy kernel features will not work anymore but that would be the same for any mainline. Switching can be done through the armbian configuration tool. 1 Link to comment Share on other sites More sharing options...
Sergio Posted January 30, 2019 Share Posted January 30, 2019 6 minutes ago, thexman said: @Sergio You don't inevitably need to switch to Stretch to avoid the issue. Simply switching to mainline on Xenial will also work. As already said legacy kernel features will not work anymore but that would be the same for any mainline. Switching can be done through the armbian configuration tool. Ok, thank you. I'll try. This is the first such case in my practice. On raspberry this has never happened before. 1 Link to comment Share on other sites More sharing options...
thexman Posted January 30, 2019 Share Posted January 30, 2019 @Sergio The problem with SBCs (except Raspberry) is that there is a huge variety of boards and SOCs (with limited native linux manufacturer support) which have to be integrated. This also made a legacy kernel necessary in the first place. In case of the RPI you actually have one type of SOC with different revisions which is much better supported and easier to integrate. Also the legacy kernel is quite old which might in this case also be the actual reason. I assume that the issue not absolutely has something to do with the SOC or board but rather with the kernel version. Because also other virtualized providers with specific kernels suffered the issue and this has actually nothing to do with a specific SBC or SOC. Link to comment Share on other sites More sharing options...
Sergio Posted January 30, 2019 Share Posted January 30, 2019 9 minutes ago, thexman said: @Sergio The problem with SBCs (except Raspberry) is that there is a huge variety of boards and SOCs (with limited native linux manufacturer support) which have to be integrated. This also made a legacy kernel necessary in the first place. In case of the RPI you actually have one type of SOC with different revisions which is much easier to support. Also the legacy kernel is quite old which might in this case also be the actual reason. I assume that the issue not absolutely has something to do with the SOC or board but rather with the kernel version. Because also other virtualized providers with specific kernels suffered the issue and this is has nothing to do with a SBC. And how many end users were affected?! ( It is a matter of trust and choice. For me prior to this moment, the orange had some advantage over the raspberry, but now it is gone. Link to comment Share on other sites More sharing options...
thexman Posted January 30, 2019 Share Posted January 30, 2019 @Sergio I can't tell how many end users were affected but also servers suffered. So this is definitely no OPI issue and I wouldn't untrust the platform. Take a look in this thread for further reading because I attached a handful of bug reports referencing to Ubuntu Launchpad concerning this issue. The origin is a systemd security update done by Ubuntu. If you would have run the same distro and kernel revision on a RPI it probably would have suffered the same issue because also x86_x64 platforms were affected. It is a annoying bug I totally agree and this shouldn't happen at all. Link to comment Share on other sites More sharing options...
ning Posted January 31, 2019 Author Share Posted January 31, 2019 12 hours ago, Sergio said: @Igor , will it help to avoid this in the future if I migrate to Debian GNU / Linux 9 (stretch) 4.19.17-sunxi? this is only happens on legacy kernel, due to kernel is too old and has bug. 1 Link to comment Share on other sites More sharing options...
Igor Posted January 31, 2019 Share Posted January 31, 2019 Patch is applied and it seems to work, updates pushed up:https://github.com/armbian/documentation/commit/417821896fedd9623bfc6d7dac13cc5392729c80 https://github.com/armbian/upload/commit/3ea43843df0ccfa7e749b41cffa27f38c3ba53e8 ... but it seems not all problems went away. I am experiencing network troubles. Lan cable was not connected to the switch 3 Link to comment Share on other sites More sharing options...
thexman Posted January 31, 2019 Share Posted January 31, 2019 @Igor Quote ... but it seems not all problems went away. I am experiencing network troubles. Lan cable was not connected to the switch haha, well that actually can happen quite easily Link to comment Share on other sites More sharing options...
thexman Posted January 31, 2019 Share Posted January 31, 2019 @Igor Can confirm that the updated fixed the issue. Updated to latest kernel, armbian + systemd. Thanks for the fix 👌 1 Link to comment Share on other sites More sharing options...
Recommended Posts