Wireguard on Armbian & TinkerBoard

[root]

Trying to set Wireguard (www.wireguard.io).

I can compile and load the module (which is an issue on most distributions as they lack the necessary kernel headers), but bringing up the interface fails:

 

root@tinkerboard:/etc/wireguard# wg-quick up mullvad 

[#] ip link add mullvad type wireguard

[#] wg setconf mullvad /dev/fd/63

[#] ip address add 10.99.0.23 dev mullvad

[#] ip link set mtu 1420 dev mullvad

[#] ip link set mullvad up

[#] wg set mullvad fwmark 51820

[#] ip -6 route add ::/0 dev mullvad table 51820

[#] ip -6 rule add not fwmark 51820 table 51820

RTNETLINK answers: Address family not supported by protocol

[#] ip link delete dev mullvad

root@tinkerboard:/etc/wireguard#

 

No idea what could be wrong there. The same configuration works fine on a VM running Ubuntu Desktop and on a Raspberry Pi with Raspbian.

 

What am I missing here?

[zx2c4]

Hey there,

 

WireGuard author here. Sorry to hear you're having problems. This certainly seems like strange behavior. Could you send me the output of:

cat /proc/version
cat /lib/modules/$(uname -r)/build/.config || zcat /proc/config.gz || cat /boot/config-$(uname -r)
ip -d addr
ip -d route
sysctl net

Then I can get to the core of the issue.

 

However, in looking for a workaround, you might try mitigating this issue by removing "::/0" from the AllowedIPs= line of your config file, so that it only reads "AllowedIPs=0.0.0.0/0". I'd be interested to learn whether this "fixes" the problem.

 

Also, the best place to get help in real time is on the WireGuard IRC channel -- it's #wireguard on Freenode.

 

-Jason

[Myy]

Could you provide the output of the following command too :

 

zcat /proc/config.gz | grep -i fwmark

 

Could be a missing kernel option.

[root]

Removing ::/0 from the config file works like a charm, thanks a ton! This kind of support is really something!

 

root@tinkerboard:~# wg-quick up mullvad
[#] ip link add mullvad type wireguard
[#] wg setconf mullvad /dev/fd/63
[#] ip address add 10.99.0.23 dev mullvad
[#] ip link set mtu 1420 dev mullvad
[#] ip link set mullvad up
[#] wg set mullvad fwmark 51820
[#] ip -4 route add 0.0.0.0/0 dev mullvad table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0
[#] echo "nameserver 10.99.0.1" > /etc/resolv.conf
root@tinkerboard:~# 

 

For further debugging, command output attached (separated by equal signs, if you need to jump quickly).

 

output.gz

[root]

23 minutes ago, Myy said:

Could you provide the output of the following command too :

 

zcat /proc/config.gz | grep -i fwmark

 

Could be a missing kernel option.

No output to this one, @Myy.

root@tinkerboard:~# zcat /proc/config.gz | grep -i fwmark
root@tinkerboard:~#
 

[Myy]

Is it really resolved ?

 

On the first commands it looked like it was setting up an ipv6 node, but on the successful one, it looks like it set up an ipv4 node. Is that all right ?

[root]

Yep, that's fine for me. I'm a bit dated, hence shying away from all this ipv6 thing.

 

[root]

Just did a few speed tests using the TinkerBoard, OpenVPN and Wireguard. I'm on a ~200 Mbps down / 20 Mbps up Internet connection.

 

LAN performance (no encryption, Gigabit Ethernet): ~600 Mbps

Direct connection performance (average): 230 Mbps down / 24 Mbps up.

OpenVPN performance (average, using Mullvad.net as a provider):  ~25 Mbps down / 20 Mbps up.

Wireguard performance (average, using Mullvad.net test servers): ~220 Mbps down / 22 Mbps up (hence maxing up my connection, negligible loss).

 

If there's anyone on a faster connection here (300-500 Mbps) and has a Tinkerboard, I'd be very interested to see if / when the CPU maxes out on the Tinkerboard while using Wireguard.

During my tests with the Wireguard, the individual kworker processes hit occasionally 30% in 'top' output, but were usually in the 15-20% range.

While doing a bit more intensive testing (a wget running in a loop with Wireguard connection active), the CPU temperature reached 72C (LE: same values with or without heatsink, slightly better with it on).

 

With my current infrastructure, if the CPU doesn't get too hot, the TinkerBoard is perfectly viable as an Internet gateway to route all traffic through a Wireguard VPN.  

I just need to find a USB Ethernet adapter which can do 250 real Mbps and is supported by Armbian. Any recommendations/experience?

[Igor]

19 hours ago, root said:

I just need to find a USB Ethernet adapter which can do 250 real Mbps and is supported by Armbian. Any recommendations/experience?

 

AX88197 should work with asix module if enabled, perhaps this one: http://www.ebay.de/itm/172659324500 There is also one Realtek r8152 based, but can't find it.

 

 

Edited August 6, 2017 by Igor
Added Realtek variant

[root]

Until getting a second adapter, I set off to see if there's anything else that can be done with my current hardware. And... there is. 

I managed to make the TinkerBoard my VPN router using Wireguard. It would be similar with OpenVPN, but (see figures above) speeds are not that exciting.

 

If anyone else is interested, here are the steps. I assume you are using a "standard" setup with your end equipment (laptops, PCs, TVs etc) connected to a router (I have a Netgear R7000), which in turn is connected to a cable modem / fiber modem / any other sort of uplink.

What you need: TinkerBoard (TB for short) running Armbian, power supply, a network cable, case (optional, using the RPI 3 standard one), a WireGuard or OpenVPN connection to a remote endpoint.

 

1. Install Armbian on the TB.

2. Connect your TB to the router using a wired connection. You will get an IP address, say 192.168.1.5. The router is 192.168.1.1 and your PC is 192.168.1.10.

3. Perform a full update (apt update && apt dist-upgrade) and reboot (you will likely need to power cycle the box, mine doesn't reboot properly).

--> Note: I believe the OpenVPN package in the Armbian is a bit old (I think I saw 2.3.10, current is 2.4); this may give you trouble with some options.

4. Set up your WireGuard or OpenVPN connection on the TB. Note Jason's hint above about removing the ipv6 route. Install the iptables package (apt-get install iptables).

5. Bring up your WireGuard or OpenVPN tunnel. You will get a new interface (called as you wish - we'll use wg0 for the Wireguard one and tun0 for the OpenVPN).

So far so good, all standard steps. Now comes the slightly trickier part.

6. On your router, in the DHCP options (this typically sits in the LAN chapter, YMMV), set the default gateway (the one which gets sent out to clients) to 192.168.1.5 (the IP of the TB).

7. On the TB, issue the following commands to enable packet forwarding:

echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/route add default gw 192.168.1.1 dev eth0
/sbin/iptables -A FORWARD -i wg0 -o eth0 -j ACCEPT
/sbin/iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE

Above - eth0 is the router's interface which carries the IP address 192.168.1.1. wg0 is the Wireguard interface on the TB (tun0 would be VPN one).

If you want to make the changes permanent, edit /etc/sysctl.conf for ip_forward and add the iptables and route lines somewhere (I'm lazy, so I've put them in rc.local)

 

8. Reconnect your computer to the LAN (replug the cable, ipconfig /release and /renew and so on). Your gateway should now be 192.168.1.5 (instead of the old 192.168.1.1

9. Try pinging a remote host, should work. Try a traceroute, the first hop should be 192.168.1.5, the second 192.168.1.1, the third your Wireguard/OpenVPN remote end. 

10. If you have any special routes declared on the old router, you will need to carry them over to the Tinkerboard. For instance, add the necessary commands to rc.local (yes, lazy).

11. If everything works, don't forget to make your tunnel start automatically, prior to any iptables / route invocations. OpenVPN starts as a service, for Wireguard I've set a manual call in (no surprise...) rc.local.

 

The result: I'm maxing out my Internet connection (200+ Mbps down) while using Wireguard. In order to do that with OpenVPN, I would have needed an i3 (at least, if not an i5) processor.  I'm happy.

Thanks to the Armbian team, to Jason for Wireguard, to Mullvad.net for providing the testing infrastructure.

[zx2c4]

Hi @root,

 

Thanks for your debugging gzip and testing that the workaround was successful.

 

It looks like the core of the issue is that the Armbian kernel you're using doesn't enable CONFIG_IPV6_MULTIPLE_TABLE (even though it does enable CONFIG_IP_MULTIPLE_TABLES, the IPv4 equivalent). While you don't use IPv6, so this doesn't really matter for you, others do and the ability to do policy routing with multiple routing tables is pretty much an "expected feature" of Linux these days. So it's a bit odd that it's there for v4 but not for v6. I'm guessing this is just a simple error in the default kernel config, so I've submitted this pull request for fixing it: https://github.com/armbian/build/pull/727

 

In any case, from your perspective, I think the workaround suggestion of my first reply is the correct way. Since you're not using v6, and not specifying a v6 interface address, it doesn't make much sense to have the v6 default gateway (::/0) in AllowedIPs or in your routes. So, with the change you made, things are now coherent and fine.

 

We have, in the process though, unearthed a potential problem for other Armbianers down the road, so thanks for the detailed report.

 

Jason

[root]

Thank you Jason (and Igor and Myy).

Next hurdle from me: booting from a USB stick. All the tutorials I see reference Linaro-based distributions, which have a configuration file in /boot. I assume it's slightly more difficult with Armbian... 

[root]

Wondering if Jason (@zx2c4) is still hanging around here...

Question: is there a description / manual for the Wireguard config file syntax (i.e. all possible parameters)?

 

Background: I have two Wireguard tunnels set up.  The first (call it wg0) is my "main" interface (all traffic is routed through it); the second (wg1) is only a site-to-site connection to a different location. So when going from my location to the remote location, the wg1 traffic goes through wg0 until it hits wg0's endpoint and over the public Internet afterwards.

Problem: I cannot set up the second interface (wg1) using wg-quick - or, more accurately, I can bring it up using wg-quick, but I cannot bring it down. When I do "wg-quick down wg1", the entire connectivity goes MIA. Obviously, I am not too keen to do that on the remote site, where I don't have "hands and eyes" to reboot the box, so I ended up building my own (ugly) script to start and stop the tunnel.

 

The exact moment where everything dies is (I have added a "set -x" in the del_if() function in wg-quick):

 

+ cmd ip -4 rule delete table main suppress_prefixlength 0
+ echo '[#] ip -4 rule delete table main suppress_prefixlength 0'
[#] ip -4 rule delete table main suppress_prefixlength 0
+ ip -4 rule delete table main suppress_prefixlength 0

 

My limited understanding says wg-quick is taking down the main (routing?) table, which causes all connectivity to collapse.

I am therefore wondering if there is a way to tell wg-quick  that this is not a "main" tunnel, i.e. it is not supposed to use it for redirecting / un-redirecting all traffic (similar to OpenVPN's "redirect-gateway def1" option). I googled a fair bit, but couldn't find anything.

[Technicavolous]

On 8/5/2017 at 8:57 AM, Igor said:

AX88197 should work with asix module if enabled, perhaps this one: http://www.ebay.de/itm/172659324500 

In the graphic on this listing the chipset part number is noted as AX 88179, which brings up a lot of items on Amazon, etc. I also want one of these ;]

 

This one has the bonus of bringing a USB3 hub outside the Cloudshell case ... off topic but thanks for bringing this to my attention.

[tkaiser]

2 hours ago, Technicavolous said:

AX 88179, which brings up a lot of items on Amazon, etc. I also want one of these

 

Me not, I prefer the better alternative: RTL8153. Better driver support, slightly better performance, slightly lower CPU utilization.

[Technicavolous]

4 hours ago, tkaiser said:

RTL8153

Again good data from the expert thanks ;]

[lampra]

Hi @root, could you share your experience on compiling the module?

I am trying to compile wireguard on cubietruck (A20), Armbian 5.31 kernel 4.11 but did not succeed so far.

[root]

Hi @lampra.

As far as I remember, I couldn't get it compiled with 4.11, as I couldn't find any suitable kernel headers version.

My solution was to switch to the stretch branch - I could find everything I needed for that.

Here's how my apt sources.list file looks like (note the last line):

root@tinkerboard:~# grep -v ^# /etc/apt/sources.list
deb http://ports.ubuntu.com/ xenial main restricted universe multiverse
deb http://ports.ubuntu.com/ xenial-security main restricted universe multiverse
deb http://ports.ubuntu.com/ xenial-updates main restricted universe multiverse
deb http://ports.ubuntu.com/ xenial-backports main restricted universe multiverse
deb http://beta.armbian.com/ stretch main restricted universe multiverse utils xenial-desktop
root@tinkerboard:~#

To switch to the latest kernel - note I run a Tinkerboard (RK3288) rather than a Cubieboard (A20). Your image name will differ -- most likely you need linux-image-dev-sunxi and linux-headers-dev-sunxi.

Quote

    apt update

    apt install linux-headers-dev-rockchip

    apt install linux-image-dev-rockchip

 

After that (and a reboot) - do a full update (apt update && apt upgrade) and then follow the tutorial on wireguard.io:

add-apt-repository ppa:wireguard/wireguard
apt update
apt install wireguard-dkms wireguard-tools

 

As a side note, I was running the dev since my posts above (all the 4.14-rcX series, now just upgrading to 4.15.0-rc2) with zero stability issues. 

[TonyMac32]

I would recommend sticking with Next if you're using the device as a daily driver, especially since Dev on Tinker Board is almost always an RC candidate kernel (that comes with it's own bugs separate from anything this project may do.)

 

I would double check, but I think Next image is now built with 4.14.y LTS kernel

[lampra]

Thank you for the instructions.

I started with debian stretch next and the module compiles and works fine on cubietruck and kernel 4.14.4-sunxi