3 3
ning

SSH not working after upgrade (Orange pi Lite Armbian_Ubuntu_xenial)

Recommended Posts

due to systemd updates to a new version, maybe it's not compact with old kernel.

after system upgrade, it fails to start sshd.

 

after read code, insystemd-tmpfile chase_symlinks() return -EBADF  from kernel,

I guess legacy kernel not compact with newer systemd.

Share this post


Link to post
Share on other sites

Hello,

After upgrading the Linux - SSH stopped working.

I tried reinstalling the OS and and it works fine before running apt upgrade.

 

Board Orange pi Lite

Armbian_5.65_Orangepilite_Ubuntu_xenial_default_3.4.113_desktop

Spoiler

Boot error messages:


[FAILED] Failed to start Create Static Device Nodes in /dev.
See 'systemctl status systemd-tmpfiles-setup.service' for details.

[FAILED] Failed to start OpenBSD Secure Shell server.

See 'systemctl status ssh.service' for details.

 

$ systemctl status systemd-tmpfiles-setup.service
● systemd-tmpfiles-setup.service - Create Volatile Files and Directories
   Loaded: loaded (/lib/systemd/system/systemd-tmpfiles-setup.service; static; v
   Active: failed (Result: exit-code) since Sat 2019-01-12 17:15:12 UTC; 36min a
     Docs: man:tmpfiles.d(5)
           man:systemd-tmpfiles(8)
  Process: 539 ExecStart=/bin/systemd-tmpfiles --create --remove --boot --exclud
 Main PID: 539 (code=exited, status=1/FAILURE)

 

$ systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enab
   Active: failed (Result: start-limit-hit) since Sat 2019-01-12 17:15:23 UTC; 2
  Process: 1218 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=255)


My knowledge of Linux is enough only to reinstall and not to update the OS  ((
 

Share this post


Link to post
Share on other sites

Being hit by this issue today (though I'm not sure since other network services are starting normally and I don't have currently access to a console), I wonder why systemd is being offered as an upgrade if it's going to cause this problem.

Share this post


Link to post
Share on other sites
On 1/12/2019 at 9:02 PM, Sergey_zh said:

Hello,

After upgrading the Linux - SSH stopped working.

I tried reinstalling the OS and and it works fine before running apt upgrade.

 

Board Orange pi Lite

Armbian_5.65_Orangepilite_Ubuntu_xenial_default_3.4.113_desktop

  Reveal hidden contents

Boot error messages:


[FAILED] Failed to start Create Static Device Nodes in /dev.
See 'systemctl status systemd-tmpfiles-setup.service' for details.

[FAILED] Failed to start OpenBSD Secure Shell server.

See 'systemctl status ssh.service' for details.

 

$ systemctl status systemd-tmpfiles-setup.service
● systemd-tmpfiles-setup.service - Create Volatile Files and Directories
   Loaded: loaded (/lib/systemd/system/systemd-tmpfiles-setup.service; static; v
   Active: failed (Result: exit-code) since Sat 2019-01-12 17:15:12 UTC; 36min a
     Docs: man:tmpfiles.d(5)
           man:systemd-tmpfiles(8)
  Process: 539 ExecStart=/bin/systemd-tmpfiles --create --remove --boot --exclud
 Main PID: 539 (code=exited, status=1/FAILURE)

 

$ systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enab
   Active: failed (Result: start-limit-hit) since Sat 2019-01-12 17:15:23 UTC; 2
  Process: 1218 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=255)


My knowledge of Linux is enough only to reinstall and not to update the OS  ((
 

 

Hi! Use the "UsePrivilegeSeparation no" parameter in the sshd_config file or create the required directory "/var/run/sshd" before starting the service by systemd script.

Share this post


Link to post
Share on other sites

in kernel 3.4.113 file_table.c: fget/fget_light will check

 

if (!(file->f_mode & FMODE_PATH))

else

return NULL;

 

need to patch this part to enable fstat on PATH fds.

 

 

 

Share this post


Link to post
Share on other sites

while mainline kernel doesn't check FMODE_PATH, in fstat call path. other path still check FMODE_PATH. 

 

I have an idea to fix this by patch sunxi8i kernel

 

split fget/fget_light to __fget(fd, mode)/__fget_light(fd, mode)

and old fget == __fget(fd, FMODE_PATH), fget_light == __fget_light(fd, FMODE_PATH)

 

on fstat call path, use __fget_light(fd,0)

 

but I don't have build env, so past my idea here.

Share this post


Link to post
Share on other sites

How to disable all automatic updates in ARMBIAN 5.38 stable Ubuntu 16.04.4 LTS 3.4.113-sun8i on Orange PC Plus?
First, ssh stopped working, then lighttpd and xrdp !!!

Share this post


Link to post
Share on other sites
2 hours ago, Sergio said:

How to disable all automatic updates in ARMBIAN 5.38 stable Ubuntu 16.04.4 LTS 3.4.113-sun8i on Orange PC Plus?
First, ssh stopped working, then lighttpd and xrdp !!!


He asked, he answered)

1. sudo nano /etc/apt/apt.conf.d/02periodic

// Enable the update/upgrade script (0=disable)
APT::Periodic::Enable "0";

// Do "apt-get update" automatically every n-days (0=disable)
APT::Periodic::Update-Package-Lists "0";

// Do "apt-get upgrade --download-only" every n-days (0=disable)
APT::Periodic::Download-Upgradeable-Packages "0";

// Run the "unattended-upgrade" security upgrade script
// every n-days (0=disabled)
// Requires the package "unattended-upgrades" and will write
// a log in /var/log/unattended-upgrades
APT::Periodic::Unattended-Upgrade "0";

// Do "apt-get autoclean" every n-days (0=disable)
APT::Periodic::AutocleanInterval "0";


2. sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

 

// Automatically upgrade packages from these (origin:archive) pairs
Unattended-Upgrade::Allowed-Origins {
//      "${distro_id}:${distro_codename}";
//      "${distro_id}:${distro_codename}-security";
        // Extended Security Maintenance; doesn't necessarily exist for
        // every release and this system may not have it installed, but if
        // available, the policy for updates is such that unattended-upgrades
        // should also install from here by default.
//      "${distro_id}ESM:${distro_codename}";
//      "${distro_id}:${distro_codename}-updates";
//      "${distro_id}:${distro_codename}-proposed";
//      "${distro_id}:${distro_codename}-backports";
};

 

Share this post


Link to post
Share on other sites

This not exactly a solution. Next manual apt update and upgrade will break it again due to the bug in systemd. In a package that is outside our control. We could only provide a hack via our repository.

Share this post


Link to post
Share on other sites
46 minutes ago, Igor said:

This not exactly a solution. Next manual apt update and upgrade will break it again due to the bug in systemd. In a package that is outside our control. We could only provide a hack via our repository.



 

do you mean fix/update legacy kernel is not a solution?

Share this post


Link to post
Share on other sites
55 minutes ago, Igor said:

This not exactly a solution. Next manual apt update and upgrade will break it again due to the bug in systemd. In a package that is outside our control. We could only provide a hack via our repository. 

What kind of hack are we talking about?

Share this post


Link to post
Share on other sites
44 minutes ago, ning said:

do you mean fix/update legacy kernel is not a solution?

 
No no, don't saying that. I didn't get there, too busy. I hope someone else will cover that up. 

Only pointing out that @Sergio solution wont help.

Share this post


Link to post
Share on other sites

The most practical solution (although not perfect, because it requires reversing the systemd version) was proposed by @thexman in this thread:

https://forum.armbian.com/topic/8852-ssh-doesnt-work-on-orange-pi-zero/?page=3

Just clone his git script, run it and halt systemd updates. This will save your machine from problems with ssh and other services until this would be repaired in ubuntu.

Share this post


Link to post
Share on other sites

@scaevola

Thanks for mentioning me. Here is my script on Github to automatically dowgrade systemd to latest functioning release, just in case that someone is interested or wants to try or use it ;). No need to dig in the original thread. The readme there also describes how to use it. Upgrading to mainline kernel also works, I have tried that the other day. But be aware that some applications especially crafted for legacy might break.

Share this post


Link to post
Share on other sites

@Sergio

You don't inevitably need to switch to Stretch to avoid the issue. Simply switching to mainline on Xenial will also work. As already said legacy kernel features will not work anymore but that would be the same for any mainline. Switching can be done through the armbian configuration tool.

Share this post


Link to post
Share on other sites
6 minutes ago, thexman said:

@Sergio

You don't inevitably need to switch to Stretch to avoid the issue. Simply switching to mainline on Xenial will also work. As already said legacy kernel features will not work anymore but that would be the same for any mainline. Switching can be done through the armbian configuration tool. 

Ok, thank you.  I'll try.
This is the first such case in my practice. On raspberry this has never happened before.

Share this post


Link to post
Share on other sites

@Sergio

The problem with SBCs (except Raspberry) is that there is a huge variety of boards and SOCs (with limited native linux manufacturer support) which have to be integrated. This also made a legacy kernel necessary in the first place. In case of the RPI you actually have one type of SOC with different revisions which is much better supported and easier to integrate. Also the legacy kernel is quite old which might in this case also be the actual reason. I assume that the issue not absolutely has something to do with the SOC or board but rather with the kernel version. Because also other virtualized providers with specific kernels suffered the issue and this has actually nothing to do with a specific SBC or SOC.

Share this post


Link to post
Share on other sites
9 minutes ago, thexman said:

@Sergio

The problem with SBCs (except Raspberry) is that there is a huge variety of boards and SOCs (with limited native linux manufacturer support) which have to be integrated. This also made a legacy kernel necessary in the first place. In case of the RPI you actually have one type of SOC with different revisions which is much easier to support. Also the legacy kernel is quite old which might in this case also be the actual reason. I assume that the issue not absolutely has something to do with the SOC or board but rather with the kernel version. Because also other virtualized providers with specific kernels suffered the issue and this is has nothing to do with a SBC. 

And how many end users were affected?! (
It is a matter of trust and choice. For me prior to this moment, the orange had some advantage over the raspberry, but now it is gone.

Share this post


Link to post
Share on other sites

@Sergio

I can't tell how many end users were affected but also servers suffered. So this is definitely no OPI issue and I wouldn't untrust the platform. Take a look in this thread for further reading because I attached a handful of bug reports referencing to Ubuntu Launchpad concerning this issue. The origin is a systemd security update done by Ubuntu. If you would have run the same distro and kernel revision on a RPI it probably would have suffered the same issue because also x86_x64 platforms were affected. It is a annoying bug I totally agree and this shouldn't happen at all.

Share this post


Link to post
Share on other sites
12 hours ago, Sergio said:

@Igor , will it help to avoid this in the future if I migrate to Debian GNU / Linux 9 (stretch) 4.19.17-sunxi?

this is only happens on legacy kernel, due to kernel is too old and has bug.

Share this post


Link to post
Share on other sites

@Igor

 

Quote

... but it seems not all problems went away. I am experiencing network troubles. 

Lan cable was not connected to the switch 

 

haha, well that actually can happen quite easily :D

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
3 3