[N2] resolvconf-pull-resolved bug -> Connection timed out


Recommended Posts


Odroid-N2 using Armbian_20.08.1_Odroidn2_focal_current_5.8.5

I've recently got a "Connection timed out" issue. For instance, I can open files with nano, use htop, armbian-config, update and upgrade packages, but I can't reboot (everything over ssh). I'm having a feeling that the issue started when I tried to figure out if it's possible to get HDMI to work with my 4K TV when the HDMI cable is not plugged in at boot. In that case, I booted several times and chaged boot.ini with setenv display_autodetect "false", setenv hdmimode "1080p60hz" and  setenv disablehpd "true" without success. armbianmonitor -u information is attached at the end in my post where this line might be of interest: "[    0.000000] [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!".

Conection timed out message:

Spoiler

$ sudo systemctl reboot
Failed to set wall message, ignoring: Connection timed out
Failed to reboot system via logind: Connection timed out
Failed to start reboot.target: Connection timed out
See system logs and 'systemctl status reboot.target' for details.
It is possible to perform action directly, see discussion of --force --force in man:systemctl(1).
$ sudo systemctl reboot --force
Failed to execute operation: Connection timed out
It is possible to perform action directly, see discussion of --force --force in man:systemctl(1).
$ sudo systemctl status reboot.target
Failed to get properties: Connection timed out

 


Running  armbianmonitor -v:

Spoiler

$ sudo armbianmonitor -v
Starting package integrity check. This might take some time. Be patient please...

It appears you may have corrupt packages.

This is usually a symptom of filesystem corruption caused by SD cards or eMMC
dying or burning the OS image to the installation media went wrong.

The following changes from packaged state files were detected:

/usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf

 

Does armbianmonitor -v give any useful information? It detects corrupted pacakges, but which? The 10-globally-managed-devices.conf does not seem to exist. At least I can't read it with cat or nano.

sysctl --system

Spoiler

$ sudo sysctl --system
* Applying /etc/sysctl.d/10-console-messages.conf ...
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
* Applying /etc/sysctl.d/10-kernel-hardening.conf ...
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-link-restrictions.conf ...
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/10-magic-sysrq.conf ...
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
* Applying /etc/sysctl.d/10-ptrace.conf ...
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf ...
vm.mmap_min_addr = 32768
* Applying /usr/lib/sysctl.d/50-default.conf ...
net.ipv4.conf.default.promote_secondaries = 1
sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
fs.protected_regular = 1
fs.protected_fifos = 1
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/99-sysctl.conf ...
kernel.printk = 3 4 1 3
vm.swappiness = 30
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.icmp_echo_ignore_all = 1
net.ipv6.icmp.echo_ignore_all = 1
kernel.randomize_va_space = 2
fs.suid_dumpable = 0
kernel.dmesg_restrict = 1
kernel.yama.ptrace_scope = 1
kernel.unprivileged_bpf_disabled = 1
net.core.bpf_jit_enable = 1
net.core.bpf_jit_harden = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.ip_forward = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.default.forwarding = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.default.forwarding = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_rfc1337 = 1
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.route.flush = 1
net.ipv6.route.flush = 1
* Applying /usr/lib/sysctl.d/protect-links.conf ...
fs.protected_fifos = 1
fs.protected_hardlinks = 1
fs.protected_regular = 2
fs.protected_symlinks = 1
* Applying /etc/sysctl.conf ...
kernel.printk = 3 4 1 3
vm.swappiness = 30
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.icmp_echo_ignore_all = 1
net.ipv6.icmp.echo_ignore_all = 1
kernel.randomize_va_space = 2
fs.suid_dumpable = 0
kernel.dmesg_restrict = 1
kernel.yama.ptrace_scope = 1
kernel.unprivileged_bpf_disabled = 1
net.core.bpf_jit_enable = 1
net.core.bpf_jit_harden = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.ip_forward = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.default.forwarding = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.default.forwarding = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_rfc1337 = 1
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.route.flush = 1
net.ipv6.route.flush = 1

 


armbianmonitor -u

Spoiler

Full log: https://pastebin.com/MeJdigT8
Snippets:


[    0.000000] [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!
...
[    1.977141] thermal thermal_zone0: binding zone cpu-thermal with cdev thermal-cpufreq-0 failed:-17
[    1.977817] cpufreq: cpufreq_online: CPU2: Running at unlisted freq: 999999 KHz
[    1.977878] cpufreq: cpufreq_online: CPU2: Unlisted initial frequency changed to: 1000000 KHz
[    1.978854] thermal thermal_zone0: binding zone cpu-thermal with cdev thermal-cpufreq-1 failed:-17
...

 


I guess the only way to reboot is to pull the plug? Or could it be done without pulling the plug if I connect to the N2 with UART? I'm thinking about doing a new fresh install, but before that, is it worth trying to use the legacy kernel (4.9)? Thanks for the awesome work with Armbian! It has truly been a blessing since I first discovered it!

Link to post
Share on other sites
Armbian is a community driven open source project. Do you like to contribute your code?

Thanks for letting me know! I'll do a new clean install using Armbian_20.08.1_Odroidn2_focal_current_5.8.5 and change kernel to use legacy in armbian-config forwards. I'm looking for a much more stable experience running a home server than what I've encountered the last few months running mainline. 

Link to post
Share on other sites
13 minutes ago, Z11ntal33r said:

So if you were going to setup N2 as a home NAS, you would rather use 5.7 for some time before jumping to a higher stable kernel (>= 5.9) rather than setup 4.9 and staying there?


Long story short. 4.9.y is legacy kernel, a kernel where initial support for this hardware was developed. Its old lack modern features, but board specific functions work well / better. In modern kernel basic functions also works well, but 5.8.y is ATM in general fragile ... we had decent stability with 5.6.y / 5.7.y.

5.8.y is next LTS kernel and things will get better soon. If you need a stable operations now, use my recommendation.


And. This board is not recommended for a NAS. Expect same problems as on Raspberry Pi. Regardless of kernel.

Link to post
Share on other sites

Thanks again for a great reply! I'll stay put to mainline and use 5.7 until 5.8 has matured. 

I do not have high expectations or hope for a extremely stable experience with N2, but if it can somewhat give me a decent experience that will be more than enough. I bet I've changed my arm device in a year or two. Out of couriosity, do you have any similar boards you would recommend for NAS usage in the $100-150 price range?

Link to post
Share on other sites

I think I found the root of the issue I had. It seems to be related to resolvconf and particularly resolvconf-pull-resolved where I had systemd-resolved installed and handling resolv.conf with a symlink. Removing the resolvconf package did the trick. I'll let the server run for some days, yet it seems fine some far. I'm going to make sure that I remove resolvconf in my installation script to prevent this mess from happening again... This is something to be aware of as others might be a victim of the issue as well. It's a quite serious one too.

 

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968015

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=967906

 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...