Cutting edge, several networking problems

[ManoftheSea]

Armbianmonitor:

http://ix.io/1RY7

I recently, yesterday, updated u-boot to 800-800.  I did a dist-upgrade to buster.  I've got kernel 4.19.56.  I'm using espressobin as a firewall for my home network, connected to comcast (where I used to have ipv6 addresses).  systemd reports version 241.  This is part status-report, documentation for others, and questions.

 

Used to have ipv6 addresses - So before the dist-upgrade and kernel upgrade, I was on .20 in Jessie.  And systemd-networkd was giving me prefix delegation such that my home net had public IPv6.  Now, systemd is unable to complete the transaction.  It just repeats this block:  [edit] I had it misconfigured.

 

Network startup - When the board boots, it seems the wan interface is slow.  The first time systemd-networkd tries to configure the wan interface, it gives an error, saying "Network is down".  This is easy enough to fix, I just log into the box after any boot and send a `ip link set wan up`, and it works from there. 

Again with networkd, the bridge interface seems to try to configure too early.  See below, "could not bring up interface: invalid argument".  This one, however, resolves itself when the lan interface gets set up, as it detects the carrier and goes back to set up the bridge again.  So this one's not so bad.

orkd[299]: No virtualization found in /proc/device-tree/*
orkd[299]: UML virtualization not found in /proc/cpuinfo.
orkd[299]: This platform does not support /proc/sysinfo
orkd[299]: Found VM virtualization none
orkd[299]: lo: Unmanaged
orkd[299]: wan: Could not bring up interface: Network is down
orkd[299]: lan0: Joined netdev
orkd[299]: lan0: Bringing link up
orkd[299]: lan1: Joined netdev
orkd[299]: lan1: Bringing link up
orkd[299]: eth0: Flags change: +UP +LOWER_UP +RUNNING
orkd[299]: LLDP: Started LLDP client
orkd[299]: eth0: Started LLDP.
orkd[299]: eth0: Gained carrier
orkd[299]: eth0: Configured
...
orkd[299]: br0: udev initialized link
orkd[299]: br0: Link state is up-to-date
orkd[299]: br0: found matching network '/etc/systemd/network/10-br0.network'
orkd[299]: br0: Setting address genmode for link
orkd[299]: br0: Bringing link up
orkd[299]: br0: Flags change: +UP
orkd[299]: br0: Could not bring up interface: Invalid argument
orkd[299]: Got inotify event on bus bus-api-network.
orkd[299]: Added inotify watch for /run on bus bus-api-network: 2
orkd[299]: Added inotify watch for /run/dbus on bus bus-api-network: -1
orkd[299]: lan0: Flags change: +LOWER_UP +RUNNING
orkd[299]: lan0: Gained carrier

I found a systemd bug report 12784 that describes similar behavior to what I'm seeing, with Debian sid, kernel 5.2, and systemd 241.  It's very recent, so maybe it's already fixed and just needs to propagate?  

 

NFTables and network devices - I have an nftable rule that makes use of "iif" (input interface), which supposedly uses the network interface index rather than its name.  Which means, if "wan" doesn't exist when nftables.service gets started, then the ruleset fails.  That's what I'm seeing.  I corrected to "iifname", which does a string compare per packet.  That seems wrong.  Does anyone have a suggestion for how to order nftables rules for after a given interface is created, but before it's brought up?  Maybe a oneshot service that's "after" udev but "before" systemd-networkd?

 

And while I'm on nftables, my ruleset has a "counter".  But "nft list counters" doesn't show it, nor getting more specific into "nft list counters table inet filter".  When I attempted to name the counter in the ruleset, nft complained about a missing file/directory.  I vaguely guess that there's a module missing, but I don't know how to debug that further, and it's not important enough that I'll spend the time, unless someone else would benefit too.

 

But my final state, after a manual process each post-reboot, is that I have a capable, functional linux box as my network firewall.  And that's pretty cool.

Edited September 5, 2019 by ManoftheSea
Retracting bug report

[ManoftheSea]

This past weekend, I thought I'd further tune my home network.  My goal was to get suricata running and protecting my network.

 

Reminder: Armbian Buster, Kernel 4.19.56

 

Suricata was installed by package, along with oinkmaster-suricata.  The "emerging" ruleset was installed according to that package, just a run of "suricata-oinkmaster-updater".  The Suricata default installation does NOT appear to work on the espressobin.  This default uses AF_PACKET to, as I understand it, listen to packets coming in on eth0 and going out on "any other interface".

 

Well, we only have eth0 on EspressoBIN.

 

So I added a rule on the firewall using nft.  Note that my firewall table is named "filter", and the added chain is named "IPS".  The system's firewall is in a chain called "firewall" at priority 0, not shown here, so suricata should only process packets that would otherwise be allowed by static rules.

nft add chain inet filter IPS { type filter hook forward priority 10\; policy accept\;}
nft add rule inet filter IPS queue num 2 - 3 fanout,bypass

This rule sends packets to queues 2 and 3, which should allow tuning suricata to run on both processors and share the work.  "bypass", at the end, means that if suricata fails, it fails open, such that the network still works but without protection.

 

The default install of suricata selects the af-packet mode of operation in the suricata.service file.  That is edited with "systemctl edit suricata.service", which will end up putting a /etc/systemd/system/suricata.service.d/override.conf file in place:

[Service]
ExecStart=
ExecStart=/usr/bin/suricata -D -q 2 -q 3 -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid

The first line clears the ExecStart from the base file, and the second has "-q 2 -q 3" to tell suricata to listen to the NFQUEUEs as numbered in the nft rule.

 

Test the setup by watching the /var/log/suricata/fast.log and running `curl -A "BlackSun" example.com` - there should be an ALERT as defined in the emerging user agents rules.  Beware that trying against google will go over HTTPS, and therefore the IPS can't detect the content of the traffic.

 

Hopefully this is of use to someone.

[ManoftheSea]

Let me retract what I thought was a bug, but is actually only a misconfiguration.  In /etc/systemd/network/10-wan.network, I had "IPv6AcceptRA=no".  To get IPv6 addresses from upstream, I needed yes.  After I fixed that, I'm getting prefix-delegation again.  I already had the needed /etc/systemd/network/10-br0.network entry "IPv6PrefixDelegation=dhcpv6".

 

So, just that one complaint about wan being slow to come up to figure out.