Network Manager woes

[reverend_t]

Ubuntu Server edition doesn't use the fundamentally limited Network Manager for a reason. For any kind of semi-complex networking configuration, (say VLANs, bridges, per-network firewall rules,  any kind of policy based routing and so on and so on). Also, conversely on little, largely headless servers with static network configurations, Network Manager is a big, bloaty and intrusive process that adds boot time and complications.

 

Just recently I have struggled hugely with difficult to replicate networking issues. I thought (and Network Manager reported) that NM didn't touch the interfaces specified in e/n/i. Turns out not to be the case.

 

Is there any rationale behind Armbian's decision to offer "Ubuntu Server" with added NM? Especially given that it's primary use case is for desktop/laptop scenarios?

[zador.blood.stained]

8 minutes ago, reverend_t said:

Is there any rationale behind Armbian's decision to offer "Ubuntu Server" with added NM? Especially given that it's primary use case is for desktop/laptop scenarios?

It's good as a default choice for inexperienced users, especially for configuring wireless devices and it helps in getting less questions like "I added some random crap that I found on the Internet to /etc/network/interfaces but my wireless still doesn't work". Unfortunately since we added support for cheap and popular devices (read: H3/H5 Orange Pi boards) we have enough people with no prior Linux experience trying to use our images so there NM is a reasonable default choice.

Any person with descent Linux knowledge can remove NM and use any other alternative for configuring network related stuff.

[reverend_t]

Hmmm, I've never thought that e/n/i is particularly complicated. Like most of GNU/Linux it's a powerful tool, possible to get wrong but it's also a solid part of the Linux learning experience. Isn't that also what Armbian is at least partially about? Even Netplan, which is slated to replace ifupdown in 18.04, is largely based on e/n/i (state your desired outcome) principles rather than NM (coddle you from any chance of mishap) principles.

 

And the fact that boards like the Orange Pi Zeroes, etc, open up the world of Linux to more people is something to be celebrated, surely?

 

However, if this is the way things ust be then there's a couple of issues/suggestions:

 

1. The "standard" Ubuntu/Debian way of disabling Network Manager appears to be broken (at least in H3 5.35 Dev):

$ sudo systemctl stop NetworkManager.service
$ sudo systemctl disable NetworkManager.service

Following these (man page) commands good ol' Network Manager is somehow re-enabled on the next boot. I had to take slightly more drastic step of:

mv /lib/systemd/system/NetworkManager.service /lib/systemd/system/NetworkManager.service.BAK

after the previous steps to actually slay the vampire. I don't know whether this is due to Armbian's integration of NM into its basic tools or what.

 

 

2. The description in /e/n/i needs to be more balanced

 

It sounds like it was written by someone fuming at the ears. It should, of course, emphasise the benefits of NM but should also point out that for any vaguely interesting routing set up (say you want to play with LXD and create neat bridges, VLANs or whatever) then it should contain clear and working instructions on how to disable NM (since the standard Debian way doesn't seem to work). Maybe even better would be to add into armbianconfig. I'll investigate if I have time!  

[zador.blood.stained]

10 minutes ago, reverend_t said:

I've never thought that e/n/i is particularly complicated.

It's not complicated if you know what you are doing, if you know where to get info (i.e. reading man pages over who knows how old tutorials) and know what to expect from it, especially if you have 2 interfaces (i.e. wired+wireless) connected to the same network.

 

12 minutes ago, reverend_t said:

Isn't that also what Armbian is at least partially about?

Obviously there is a noticeable disagreement regarding what Armbian is about.

 

14 minutes ago, reverend_t said:

And the fact that boards like the Orange Pi Zeroes, etc, open up the world of Linux to more people is something to be celebrated, surely?

No, the original Zero is not a good board for this, IMO it's better to start with a RPi which has a recent enough supported kernel, better onboard wireless (where it exists) and has some HW+FW based undervoltage and overcurrent protection.

 

17 minutes ago, reverend_t said:

1. The "standard" Ubuntu/Debian way of disabling Network Manager appears to be broken

AFAIK NM has several services and several udev rules (in addition to config overrides for some packages like dnsmasq), it would be easier to remove it completely.

 

 

[raschid]

Just for the record:

18 hours ago, reverend_t said:

I've never thought that e/n/i is particularly complicated. Like most of GNU/Linux it's a powerful tool, possible to get wrong but it's also a solid part of the Linux learning experience.

Yep, agree. I don't use NM on any of my boards.

 

18 hours ago, reverend_t said:

 Isn't that also what Armbian is at least partially about?

For me it is.

18 hours ago, reverend_t said:

And the fact that boards like the Orange Pi Zeroes, etc, open up the world of Linux to more people is something to be celebrated, surely?

For me it certainly was.

18 hours ago, reverend_t said:

The description in /e/n/i needs to be more balanced

I would second that.

 

18 hours ago, zador.blood.stained said:

Obviously there is a noticeable disagreement regarding what Armbian is about.

Nothing wrong with that.

 

Peace!

 

[tkaiser]

On 5.1.2018 at 6:14 PM, reverend_t said:

I thought (and Network Manager reported) that NM didn't touch the interfaces specified in e/n/i. Turns out not to be the case.

 

Sounds like you were running into the bug (or settings mismatch) referenced at the end of https://askubuntu.com/questions/842773/ubuntu-server-and-networkmanager (where it's also described how to let NM take care of wired interfaces in Ubuntu Server).

 

Great resource for people struggling with NM concepts is https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/ch-introduction_to_rhel_networking

 

And for experts who prefer anachronistic Debian configuration attempts (fiddling around with text files instead of profiles) the obvious solution is an 'apt purge network-manager && apt autoremove'.

[tkaiser]

On 5.1.2018 at 7:05 PM, reverend_t said:

Hmmm, I've never thought that e/n/i is particularly complicated.

 

Why not applying then for a job as full time working but unpaid member of the support staff dealing with something like this every other day: 

 

 

Using nmtui-connect would work instantly but since there are all those crappy tutorials out there telling people to fiddle around with /etc/network/interfaces and other text files so many users are wasting their time again and again.

 

[Tido]

Or add in the documentation: for Experts, how to remove the Networkmanager

because the comment in here will get lost anyway after some time

[Tido]

https://docs.armbian.com/User-Guide_Fine-Tuning/#how-to-change-network-configuration

[ab1jx]

Good to know.  I'm coming at Armbian from the perspective that I'd rather be using straight Debian, but the Arm machines have some peculiar hardware which require specialized knowledge, kernels, drivers, etc.  I want it to be as much like normal Debian as possible, not keep stumbling over hidden "helpful" features.  For that matter I'd probably be running OpenBSD but by last account they're woefully far behind and don't even have HDMI working yet, you can't install without a serial console.  As far as I'm concerned everything not an Arm machine is a dinosaur.  I still have a couple 10+ year old i386/amd laptops but most of my efforts go into setting up my Arm machines (and using them).  I'd like to get back to the Gphoto program I'm working on and some SDR stuff, not keep messing with operating systems.  But until Arms are as mainstream as i386 only Armbian, Raspbian, etc. can deal with them.

 

With a relatively stock Pi all I have to do is set /etc/hostname so it's unique.  It boots up, DHCP works because it needs NTP.  I can get on the internet from it, also I can boot one headless and connect to that hostname from another machine.  I grudgingly gave up static IPs once I realized that hostnames from DHCP get fed to DNS in my Android phone hotspot/AP.  That's not quite right in my Rock64 yet.  On a Pi, even under Stapleberg's amd64 Buster it seems to not matter that much whether I use my interfaces file or not.  Sometimes I forget to make one or copy one in, it still works.

[Igor]

1 hour ago, ab1jx said:

not keep messing with operating systems.

How do you define more Debian? What is missing, what is too much, what you can not do?

 

BTW. I use Debian Stretch with Gnome3 on my Notebook, but my server runs Ubuntu Bionic with XFCE. Both with reasons.
 

1 hour ago, ab1jx said:

For that matter I'd probably be running OpenBSD but by last account they're woefully far behind and don't even have HDMI working yet

The same or similar situation is with most mainstream Linux distribution. You can grab "clean" Debian or whatever, but almost nothing will work as you already figured out. You literally jump back few years. Not worth to have supposable cleaner Debian with no support, which will changed only a little since it's a generic distro. 

 

1 hour ago, ab1jx said:

With a relatively stock Pi

Raspbian is Debian Stretch based with possible Wheezy compatibility hacks. I don't use it. I never did and don't plan to.

 

Armbian is as clean as possible Jessie, Stretch and Stallmanised Ubuntu Xenial and Bionic.

[chwe]

I think it's due to the fact that NM is not part of server Debian by default.. But I don't really see a problem here.. you can tell NM to ignore *random interface* and in bionic you can modify which renderer you wan't to use by default. People who know what they're doing won't have much problems to adjust networking to their needs and for the others (including me ) NM gives a smooth way to configure network fast and it's the same whatever 'sort' of armbian I use (xenial, bionic, wheezy or stretch).. 

You could argue that it should every time be similar to the distribution/versions defaults but on the other side I think armbian should be 'predictable' for some basic features. From a support point of view it's a way easier when 'we' can assume that network configuration is done by NM. There might be some exception from this rule (e.g. the espressoBin and for the R2, I switched to networkd - it just didn't like NM on firstboot ). 

 

For those who really don't like NM, http://lmgtfy.com/?q=network+manager+ignore+interface shouldn't be that hard.  But having a documentation which says:

• for wheezy you should configure it via /network/interfaces
• for ubuntu xenial it depends if you use desktop or server due to this is solved differently 
• for ubuntu bionic you've to configure which renderer you want and then do it either this or this way
• etc.

That's not gonna work well due to *to much documentation to read*.  I see armbian as a own distribution (where you can decide if it's in debian or ubuntu flavor) and as a own distribution it has things people like and things (some) people don't like. If everyone would like a linux distribution the map shouldn't look like the following: 

 

[ab1jx]

1 hour ago, Igor said:

How do you define more Debian? What is missing, what is too much, what you can not do?

 

BTW. I use Debian Stretch with Gnome3 on my Notebook, but my server runs Ubuntu Bionic with XFCE. Both with reasons.
 

The same or similar situation is with most mainstream Linux distribution. You can grab "clean" Debian or whatever, but almost nothing will work as you already figured out. You literally jump back few years. Not worth to have supposable cleaner Debian with no support, which will changed only a little since it's a generic distro. 

 

Raspbian is Debian Stretch based with possible Wheezy compatibility hacks. I don't use it. I never did and don't plan to.

 

Armbian is as clean as possible Jessie, Stretch and Stallmanised Ubuntu Xenial and Bionic.

 

I imagine that someday you'll be able to go to https://www.debian.org/distrib/netinst and download an installer that will work on any Arm machine.  It will contain something that probes the various boot scenarios, find one that works, try network methods until one works or it gives up with a message about what to do next.  That would be an install kernel, which would download a real one and enough of the OS so you could pick what you wanted to install.  Something that works like menuconfig would be good, with defaults if you don't want to bother selecting, then it would do the downloading.  Wouldn't need X to do that, and maybe it wouldn't know about the full 57k of stuff Synaptic can access. On next boot you'd have a full kernel with the programs you selected, X and/or Weston or something else.  I think the differences between Pi, Rock, Banana, etc. could be handled by loading modules.

 

At my last job we used to set up Windows machines and maintain them for an office of 30 people.  We used Sysprep and Ghost, we could reload a machine in half an hour.  Everybody got everything installed, whether they knew how to use it or not.  People could log out, go somewhere else and log in for an afternoon, everything worked the same.  Workstations were expendable, everything got done with files on a server, which used a RAID, had nightly backups to tape, all that.  I'm pretty sure we never lost anything in the 9 years I was there.  Viruses, hardware failures, all just part of the job.

 

Raspbian was Wheezy when I started, now it's Stretch.  I'm building a tablet with a Pi and the 7 inch touchscreen they sell.  No physical keyboard, so I wanted to autologin as root.  Took a couple days to find the PAM rule preventing that, it was new with Stretch.  Now whenever something doesn't work I first suspect PAM, Selinux, Apparmor, all of which are in use.  I've been curious lately about http://www.linuxfromscratch.org/ but again I doubt it works out of the box on a particular Arm machine.  With OpenBSD there was source for everything, patches used patch(1) to apply text changes and you recompiled.  Kernel, apps, X, everything.  You can study the source as working examples by grepping for something in there.  With Linux you can get source but somehow it's not the same.

[Igor]

5 minutes ago, ab1jx said:

I imagine that someday you'll be able to go to https://www.debian.org/distrib/netinst and download an installer that will work on any Arm machine. 

General ARM kernel exists in some upstream distros but is mostly useless here and supports only basic features. Also, we could merge certain kernels today ... actually, we do merge them all the time, but most people don't notice since merging takes years to complete. Most of those cheap boards have no BIOS and you need to init hardware with u-boot or something similar. You need loader per board. For this reason, we need a different approach. Since most of the boards have one media for boot and system, there is no much need for classical Debian installer, but can and will probably be implemented once.

 

One of the project's aims is to get a grip over ARM diversity and so far there are no known shortcuts and that's why so many great mainstream distribution struggle or rather not deal with anything else than RPi.

 

In general, there is a big diversity among ARM hardware and it is much more complex than unified, standardized and boring x86 world. You can add Raspberry Pi to this group since its design is simple, outdated and already comes with fairly good support. You "only" need to play with userspace, while Linux ... is the kernel. Ask Torvalds which distro he prefers?  

 

Years to go.

 

24 minutes ago, ab1jx said:

Raspbian was Wheezy when I started, now it's Stretch.  I'm building a tablet with a Pi and the 7 inch touchscreen they sell.  No physical keyboard, so I wanted to autologin as root.  Took a couple days to find the PAM rule preventing that, it was new with Stretch

1

But Raspbian is also not Debian and will never be. A Debian which needs a FAT boot partition?

BTW. Armbian desktop boots into the desktop without a password by default. You can switch to the regular display manager if you like and you can switch desktop from user mode to root with editing one config file. We use nodm display manager. https://packages.debian.org/jessie/misc/nodm

 

25 minutes ago, ab1jx said:

but again I doubt it works out of the box on a particular Arm machine.

ARM.bian philosophy: get the best possible kernel (usually and unfortunately that one doesn't come from the mainstream) and take most matured userspace. Don't complicate.

 

When you start to ask yourself which distribution or which desktop (features) you want or like on your ARM hardware ... Stop and go back to the kernel. Only in x86 you can forget about the kernel since most of the distributions have the same, the one and only (kernel.org + perhaps minor distro related hacks). Compiling and mixing is easy.

 

On our ARM boards, nothing from this is simple and easy. That's is another reason why our project exists. To bring complexity down to the earth.

[ab1jx]

Hmm, nodm.  I just grabbed a copy onto my arm64 Pi to look at.  Is it possible to boot to a real command line, see the normal bootup messages, watch for errors, etc.?  It's mostly too fast to read but on my Pis I can tell if the wifi is having trouble getting a lease, that sort of thing.  What I see on the Rock with Armbian is a black screen until there's a login prompt.  If something's broken it just stays black.  I can put the SD in a reader and try to look at logs on another machine but it's a guessing game.  I set my default target to the multi user target but that didn't actually accomplish much.

[chwe]

1 hour ago, ab1jx said:

What I see on the Rock with Armbian is a black screen until there's a login prompt. 

ever connected a serial console to your rock? Debug stuff happens during boot without is often hard or impossible. 

 

3 hours ago, ab1jx said:

I imagine that someday you'll be able to go to https://www.debian.org/distrib/netinst and download an installer that will work on any Arm machine.

 

3 hours ago, Igor said:

Most of those cheap boards have no BIOS and you need to init hardware with u-boot or something similar.

 

@tkaiser posted this once.. Don't know where and in which topic.. but that might go in this direction.  https://fosdem.org/2018/schedule/event/hwenablement_simplifying_soc_enablement_in_linux/

 

[Igor]

 

2 hours ago, ab1jx said:

Is it possible to boot to a real command line, see the normal bootup messages, watch for errors, etc.?

You mean such way?
https://www.youtube.com/watch?v=6K9zJULoFpU

On some kernels, we have to choose either serial either HDMI verbose output. Sometimes it can be done on both, sometimes HDMI is enabled too late and can't be changed or we have no idea how ... Remember hardware diversity. Even UART speed is not the same, but it's usually on "standard" RPi pins.

 

2 hours ago, ab1jx said:

What I see on the Rock with Armbian is a black screen until there's a login prompt.  If something's broken it just stays black. 

If something is broken on a DEV board, you need at least a serial console. Other options are too limited.

[ab1jx]

Well, I guess I'll have to rig up a serial console.  I've got a couple of those serial-USB adapters and I think both my old laptops have "real" serial ports.  Not sure about flexibility on baud rates, stop bits, etc.  Except this is probably 3 or 5 volts, not +/- 12, there are chips that are level converters. I think I've got a couple.  OK, "They’re 3V3 TTL"  https://www.raspberrypi.org/blog/pinout-for-gpio-connectors/ 

Always seemed like you could take an old 40 pin hard drive cable and only strip the wires you needed, might as well do 5 volts and ground at the same time.  My Pis and Zeroes have normal bootup messages.  Like what dmesg shows or /var/log/messages, or journalctl.  klogctl maybe.

 

But you mean if some essential file gets trashed or there's a kernel panic I'd have to have a serial connection hooked up or I'd miss it?  Or wait forever staring at a blank screen.  Jeez.  Something I've seen logs all the boot messages into a file that gets overwritten next boot.  Maybe one of the BSDs.

[chwe]

12 hours ago, ab1jx said:

But you mean if some essential file gets trashed or there's a kernel panic I'd have to have a serial connection hooked up or I'd miss it?  Or wait forever staring at a blank screen.  Jeez.  Something I've seen logs all the boot messages into a file that gets overwritten next boot.  Maybe one of the BSDs.

For debug bootissues on SBCs serial UART is essential.. For reading a logs after a failed boot, you need a properly initialized filesystem (kernel not u-boot). Whereas UART can be forced to give you some boot-information even without a properly loaded devicetree (e.g. CONFIG_DEBUG_LL_UART_8250=y, CONFIG_DEBUG_UART_PHYS & CONFIG_DEBUG_UART_VIRT,  CONFIG_EARLY_PRINTK=y and earlprintk as bootarg is needed + phys and virt address of the UART you use for debug).