EspressoBIN - Bionic feedback

[Jens Bauer]

I've set up my local EspressoBIN with 18.04 Bionic (Armbian_5.44_Espressobin_Ubuntu_bionic_next_4.14.40.7z).

 

My impression is that it's very much like setting up Xenial and the same issues seem to be present.

I mention them in the order of importance.

1: #reboot sometimes hang, so does #poweroff. It would be great to get this fixed, because it would be a disaster if a device hangs on rebooting from remote (eg. the device is 200km away).

2: sudo apt-get update && sudo apt-get upgrade destroys the network setup. No network access is possible after doing this. I think a fix is important, because people would likely issue those commands.

3: Network seem to be configured slightly different than on Xenial; it seems I cannot configure a static IP on the WAN port (see my other post specifically on static IP). For now, I'll be using a static IP address on the br0, this works, so it's not the highest priority.

 

I definitely also need to write some good stuff.

 

1: 3Gbit JMB321 Port Multipliers work well with Xenial (and likely also Bionic); just remember to connect the boot device to SATA0 and there should be no problems.

2: 6Gbit JMB575 (StarTech) works great with Bionic (and likely also Xenial); again connect the boot device to SATA0.

 

 

For those, who want to boot "directly" from SATA, this is what I do on both boards; it's fairly easy and there are plenty of ways you can do it.

The first time, I just made a 'dd' from my SD-card's partition to my harddisk's partition. This works fine and the partition will be expanded when Armbian boots.

My preferred way, though, is to make a tarball of the unmodified SD-image's partition, boot my EspressoBIN from the SD-card and extract the files onto my desired partition. This gives me a different UUID from the SD-card, thus inserting the Armbian SD card won't confuse my boot-up sequence.

 

To create a tarball, this can be used (mounted as /mnt/sd) ...

cd /mnt/sd && sudo tar -cpzf /data/armbian-bionic-rootfs.tar.gz .

 

To extract onto your partition (mounted as /mnt/rootfs), this can be used ...

cd /mnt/rootfs && sudo tar -xpzf /data/armbian-bionic-rootfs.tar.gz -C .

 

Those will preserve permissions, timestamps and other attributes as necessary, but this will only happen if you're root, otherwise permissions will be messed up and you'll get a half-working system (quite useless).

 

I use a custom UUID on my own setup, so that I only need to set it once in u-boot. I highly recommend using UUID for identifying the boot device, especially is you use a port multiplier, because you never know if a USB-drive suddenly get /dev/sda instead of the drive you expect.

 

In your startup boot sequence, I recommend a 3-stage check ...

1: Check for SD-card (SD/MMC slot)

2: Check for USB-boot

3: Use a 'scsi scan; scsi dev $bootdev:$bootpart;' then check for your SATA-device here.

 

Personally I only have step 1 and 3, since I often mount my SD-card as a USB-drive when I do not want to boot from it.

[Jens Bauer]

This is how I've currently set up my u-boot:

setenv bootcmd 'run boot_armbian'
setenv boot_armbian 'run get_images; run set_bootargs; run load_script2; booti $kernel_addr $ramfs_addr $fdt_addr'
setenv get_images 'tftpboot $kernel_addr $image_name; tftpboot $ftd_addr $ftd_name; run get_ramfs'
setenv get_ramfs 'if test "${ramfs_name}" != "-"; then setenv ramfs_addr 0x8000000; tftpboot $ramfs_addr $ramfs_name; else setenv ramfs_addr -; fi'
setenv set_bootargs 'setenv bootargs $console $root ip=$ipaddr:$serverip:$gatewayip:$netmask:$hostname:$netdev:none nfsroot=$serverip:$rootpath $extra_params'
setenv load_script2 'if test -e mmc 0:1 boot/boot.scr; then echo "... booting from SD-card"; setenv boot_interface mmc; else echo "booting from SATA"; scsi scan; setenv boot_interface scsi; scsi dev 0:1; fi; if test -e $boot_interface 0:1 boot/boot.scr; then ext4load $boot_interface 0:1 0x00800000 boot/boot.scr; source; fi'

# A suggested, but untested SATA-boot (try MMC first, then USB, then SATA):
setenv load_script3 'if test -e mmc 0:1 boot/boot.scr; then echo "... booting from SD-card"; setenv boot_interface mmc; else usb start; if test -e usb 0:1 boot/boot.scr; then echo "... booting from USB"; setenv boot_interface usb; else echo "booting from SATA"; setenv boot_interface scsi; scsi scan; scsi dev 0:1; fi; fi; if test -e $boot_interface 0:1 boot/boot.scr; then ext4load $boot_interface 0:1 0x00800000 boot/boot.scr; source; fi'

There might be typos, because I don't have any copy-and-paste from my serial terminal and in addition, load_script3 has not been tested, so it may contain errors as well.

-But you should get the idea; it's in fact just a tiny addition to armbian's default setup.

[Jens Bauer]

I've gotten a little further with finding out what makes "apt upgrade" stop working on the Espressobin (this likely applies to both Bionic and Xenial).

 

As soon as I boot Bionic, I see that 2 upgrades are available, before I issue 'apt update'.

I can safely issue 'apt upgrade' without first issuing 'apt update'!

After rebooting, network is still online.

If I then do an 'apt update' followed by 'apt upgrade apt' (eg. only the first upgradable item in the list), I lose network when setting up systemd ...

Spoiler

Unpacking initramfs-tools-bin (0.130ubuntu3.1) over (0.130ubuntu3) ...

Setting up systemd (237-3ubuntu10.3) ...

Progress: [ 19%] [##################................................................................................]
(lost network connectiong when at 19%, but it seems that the installation continued and actually finished anyway)

 

-So far, so good; I'll try and see how many packages can be upgraded without losing network; so for now I'll try avoiding upgrading systemd.

 

I've also noticed that both 'poweroff' and 'reboot' hangs here ...

Spoiler

Unmounted /{partition}.
Reached target Unmount All Filesystems.
Stopped target Local File Systems (Pre).
Stopped Remount Root and Kernel File Systems.
Stopped Create Static Device Nodes in /dev.
Reached target Shutdown.
Reached target Final Step.
         Starting Reboot...
reboot: Restarting system
(hangs here)

 

I've begun using these instead of 'reboot' and 'poweroff'; I have not experienced that they hang yet:

• sudo systemctl poweroff
• sudo systemctl reboot

I'm successfully running the following on both Bionic and Xenial:

• Redis (3 instances)
• Git+gitolite+gitweb (11 different locations, 11 x gitweb too) with login on port 9418
• NFS
• Netatalk-3.1.11
• Nginx-1.14.0 with the horrible fcgicrap
• GeoIPLite2 with CPAN modules
• Transmission (for Armbian torrents)
• Archivemount, unar, unzip
• Unbound
• Disabled RAM logging (since I run on SATA)
• Disabled unattended upgrades (since I'm running the boards as servers)
• BTRFS on all partitions except from the boot and root partitions
• 200+ web sites with CGI-scripts (ported from Apache in less than a week) - much more responsive than Apache2 on my Mac Mini

 

Note: If running Redis-server 4.0.11, the CPAN module will not build, due to that some strings have been changed in Redis. The easy fix is to install redis-server-4.0.10 and then install the CPAN module. This also means that you should not upgrade Redis after that.

 

On Xenial, I was able to build and run my own version of cfgiwrap, but on Bionic, I've made several attempts and can not get it to work - even if I manually just replace the binary and nothing else. I've tried downloading different versions of fcgiwrap (even from Ubuntu/Bionic), but no dice so far.

 

It's likely I'll be building and installing qmail next - I had to upgrade the TLS code, since Bionic comes with OpenSSL-1.1.0, which means the currently available code and patches fail.

 

[Jens Bauer]

Today I tried to install the latest bionic image; the Armbian_5.59_Espressobin_Ubuntu_bionic_next_4.18.6.

 

The first thing I tried was making a tarball, then writing this tarball to a freshly formatted partition and modifying it.

This almost booted with a few tweaks; I was dropped into an emergency shell.

I managed to get it running, however, there is no network connection at all; the LED on the network interface stays off.

(I did expect that would happen at some point, since that happened with earlier versions after 'apt upgrade').

Not being able to figure out what caused the emergency shell to be invoked, I thought that I'd try installing the tarball on a freshly formatted partition without modifying anything after the installation.

That resulted in a never-ending loop of requesting root to change password from 1234 to something else.

So I finally gave in and wrote the image to a brand new Sandisk microSD card.

I attempted to boot that from USB, but since that didn't work, I moved the card to my microSD card slot.

That didn't go well either. I actually forgot what the problem was by now, but it didn't get very far.

 

... Has the Armbian_5.59_Espressobin_Ubuntu_bionic_next_4.18.6 image been tested on an Espressobin board ?

-Did anyone succeed in booting from it yet ? -Anyone got network connection on the board with this new image ?

[Jens Bauer]

I just tried installing Armbian_5.68_Espressobin_Ubuntu_bionic_next_4.19.12.7z but ...

 

Now there's absolutely no network at all.

 

Earlier I had to do an 'apt update; apt upgrade' in order to make it impossible to connect to anything.

Now, if I install Bionic, I have no network right from the start; I do not need to do anything.

 

Have anyone succeeded in getting network on Bionic (after doing an 'apt upgrade') the past 3 months ?

[Jens Bauer]

I had another go at Armbian_5.68_Espressobin_Ubuntu_bionic_next_4.19.12.7z and had some progress:

 

This is exactly what happens on first boot, right after a clean installation.

 

Let's try and resolve a domain name

$ systemd-resolve google.com
google.com: resolve call failed: Failed to activate service 'org.freedesktop.resolve1': timed out (service_start_timeout=25000ms)

So it didn't work; Let's get some status information on systemd-resolve

$ systemd-resolve --status
Failed to get global data: Connection timed out

Seems it's not paying attention, now let's see which version is installed

$ apt policy systemd
  Installed: 237-3ubuntu10.9
  Candidate: 237-3ubuntu10.9
  Version table:
 *** 237-3ubuntu10.9 500
        500 http://ports.ubuntu.com bionic-security/main arm64 Packages
        500 http://ports.ubuntu.com bionic-updates/main arm64 Packages
        100 /var/lib/dpkg/status
     237-3ubuntu10 500
        500 http://ports.ubuntu.com bionic/main arm64 Packages

Try and re-activate it

$ sudo systemctl restart systemd-resolved.service

Now let's try and resolve a domain name

$ systemd-resolve google.com
google.com: resolve call failed: No appropriate name servers or networks for name found

The error-message has changed, we probably had some progress, let's see if we have a DNS address

$ grep -v '#' /etc/systemd/resolved.conf

[Resolve]
DNS=1.1.1.1

Not exactly what I wished for, so back up resolved.conf and replace by OpenDNS

# cd /etc/systemd
# cp -n resolved.conf resolved.conf.orig
# echo -e "[Resolve]\nDNS=208.67.222.222" >resolved.conf
# systemctl restart systemd-resolved.service

Check if the changes took place

$ systemd-resolve --status | grep 'DNS Server'
         DNS Servers: 208.67.222.222

Let's see if we can ping our gateway

$ ping 10.0.0.1
connect: Network is unreachable

Looking at the LED on the WAN port's 8P8C, only one LED is turned on

$ ip link | grep wan
5: wan@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

Try turning on the LED on the WAN port

# ip link set wan up

(could also use: # ifconfig wan up)

both 8P8C LED lights are now on, let's see the changes

$ ip link | grep wan
5: wan@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

OK, now see if we can look up a domain name

$ systemd-resolve google.com
google.com: resolve call failed: No appropriate name servers or networks for name found

Still no luck, try and see if we can ping the gateway

$ ping 10.0.0.1
connect: Network is unreachable

OK, let's add an ip-address to the WAN interface and see if that helps

# ip addr add 10.0.1.2/8 dev wan

$ ping 10.0.0.1

Now pings get through, let's try and see if we can make a domain name lookup

$ systemd-resolve google.com
google.com: resolve call failed: All attempts to contact name servers or networks failed

No dice, but I can SSH into the EspressoBIN board from another computer, so I have network, but the resolver is not working.

 

Issues:

• Network is not brought up by default.
• Resolving does not work out-of-the-box.
• IP address does not survive a reboot, thus no servers can start up at boot.
• Resolver still does not work, even after getting the network up and running, thus I can not install software via 'apt'.

I do not know how to get any further from here, but hopefully this is helpful to someone.

 

[Igor]

11 hours ago, Jens Bauer said:

I do not know how to get any further from here, but hopefully this is helpful to someone.

Did you check Debian version? Perhaps there is an issue with systemd at Bionic only?

[Jens Bauer]

57 minutes ago, Igor said:

Did you check Debian version? Perhaps there is an issue with systemd at Bionic only?

I'm not really keen on running Debian. Partly because I'm already running a complete Xenial setup on my "production server" and wouldn't like to switch.

Besides, things look kinda more promising now than a few months ago, since I've actually been able to get some network availability from 'nothing'.

I'm pretty sure that someone who knows more about Linux networking than me, would know how to easily fix the 4 mentioned issues.

 

I've been practicing a little with nmcli (not at all experienced with this tool), and have gotten slightly further; none of the issues have been fixed, though.

Using nmcli, I've been able to add an IP-address/range, a gateway and two ipv4.dns addresses to 'bond0'. I still have to do all this by hand every time, so at the moment I've switched back to an earlier version of Bionic, since I'm using the board as a file server too.

[Jens Bauer]

For anyone coming across this thread due to having the same problems as I had, I'd like to say that I am not entirely sure where the problems were.

This works for me:

1: Write Armbian_5.75_Espressobin_Ubuntu_bionic_next_4.19.20.img onto a micro-SD card.

2: Boot from the card

3: Make sudo apt update and sudo apt upgrade

4: Reboot the EspressoBIN

The ethernet interface now comes up and works as expected.

You can install 4.19.20 on a harddisk partition of your choice. I am using ext4 for my rootfs and also have a 'thin' boot partition which only holds boot.scr and armbianEnv.txt; nothing more; this points to the actual boot partition (I wish I could use UUIDs for 'finding' the kernel, but haven't figured that one out yet).

If you also have a boot partition, I recommend not loading the kernel and ramdisk from that partition, otherwise you will have to mount the boot partition each time you do a sudo apt upgrade - and if you forget, you're in trouble - this could have been the problem that I had, I am not sure.

 

Reboot and poweroff seem to be very reliable now. I don't think I've had any hangs like I used to with Xenial.

Thus as of writing this I'm in the (long) process of moving my server from Xenial to Bionic.

 

I've also gotten myself a couple of extra USB-to-Ethernet adapters. One is a Ugreen USB3-to-GbE with a 3-port USB3-hub. I don't think this worked on Xenial, but it's detected on Bionic. I haven't yet gotten a link up on it, but it seems like it will work - I will have a look at this later.

[Jens Bauer]

After running Bionic for quite a while, I decided to move on to Focal.

Focal's RESET (reboot command) seem to be very robust, it has not failed me at all.

So I bet the problem no longer exists with Focal and would like to recommend upgrading.