Espressobin support development efforts

[danglin]

5 minutes ago, danglin said:

The delay won't fix the mmc read error problem.

I should mention that there is a difference between the linux and u-boot device trees for the espressobin.  There is a voltage

regulator that switches between 1.8V and 3.3V to power SD card.  The linux tree has an entry for this regulator that the u-boot

tree lacks.  Possibly, this affects hot reboots.

 

        vcc_sd_reg1: regulator {
                compatible = "regulator-gpio";
                regulator-name = "vcc_sd1";
                regulator-min-microvolt = <1800000>;
                regulator-max-microvolt = <3300000>;
                regulator-boot-on;

                gpios = <&gpionb 4 GPIO_ACTIVE_HIGH>;
                gpios-states = <0>;
                states = <1800000 0x1
                          3300000 0x0>;
                enable-active-high;
        };

 

[JDL]

GlobalScale has posted a bunch of resources for the V7 board on their FTP site:

ftp://downloads.globalscaletechnologies.com/Downloads/Espressobin/

 

The "ESPRESSObin V7" seems to have just gone live in the past couple hours. I am downloading and hopefully this will help with support for the v7 boards. 

[ManoftheSea]

Hello, short time reader and first time poster here.  Love what you're doing.  Got an Espressobin v7 on ARMBIAN 5.70 stable Debian GNU/Linux 9 (stretch) 4.19.14-mvebu64.

 

First thing I discovered, my "wan" port is in the 2-port block with lan0, and lan1 is off by itself.  Could someone direct me to where the driver assigns the names?  It doesn't seem to be a udev rule.

 

Second, I can't get systemd-networkd to get a dhcp-assigned address on "wan".  Running dhclient works.  Maybe that's not an Armbian specific issue.  systemd says it's version 232.

 

Last (for now), I tried to do ipv4 masquerading with nft.  My guess is that some modules are missing from the build.  iptables allowed me to apply the masquerade rule alright.  I'll look into how to contribute a patch to build, but it may take me some time.  I suggest I'm missing "nft_chain_nat_ipv4", but again if someone can tell me how to discover what modules are failing to load when I try to add the nft rule, I'll provide the feedback.

 

Thank you in advance for any help

[martinayotte]

10 minutes ago, ManoftheSea said:

I suggest I'm missing "nft_chain_nat_ipv4"

First thing is to check with "grep NFT_CHAIN_NAT_IPV4 /boot/config-*" .

 

[ManoftheSea]

Nothing found by that name

 

From running the commands on a system that has a more complete build, the full list appears to be:

nft_masq_ipv4          16384  1
nft_masq               16384  1 nft_masq_ipv4
nft_chain_nat_ipv4     16384  2
nf_nat_ipv4            16384  2 nft_chain_nat_ipv4,nft_masq_ipv4
nf_nat                 36864  1 nf_nat_ipv4
nf_conntrack          163840  4 nf_nat,nf_nat_ipv4,nft_masq,nft_masq_ipv4
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  2 nf_conntrack,nf_nat

Of these, my espressobin system has nf_nat_ipv4, but neither nft_{masq,chain_nat}_ipv4.  The rest all load on espressobin "4.19.14-mvebu64 #5.70"

[martinayotte]

16 minutes ago, ManoftheSea said:

From running the commands on a system that has a more complete build

this is from "lsmod", but can you grep "NFT_" /boot/config-* instead on the working board.

Then, do the same on this ExpressoBin to compare both ...

Do you have enough skills to build you own kernel using Armbian scripts ?

If Yes, you can create a patch against this file : https://github.com/armbian/build/blob/master/config/kernel/linux-mvebu64-next.config

Then, do the build, if you get it working, you can submit a PR for your patch to be merged ...

[ManoftheSea]

The "working board" is a debian stretch amd64 - my personal laptop, on 4.19.  From comparing the configs, I have submitted pull request 1234 - However, the build system doesn't let Debian reach the kconfig menu to do it *right*.  I will see if I can use the Docker instructions.

[martinayotte]

7 minutes ago, ManoftheSea said:

I have submitted pull request 1234

I see that it is for config/kernel/linux-mvebu-dev.config, but be careful : for ExpressoBin, it should be 7 config/kernel/linux-mvebu64-dev.config

[ManoftheSea]

If I'm understanding things correctly, the names of the ethernet ports on the espressobin come from the device tree - Specifically:

                        port@1 {
                                reg = <1>;
                                label = "wan";
                                phy-handle = <&switch0phy0>;
                        };

                        port@2 {
                                reg = <2>;
                                label = "lan0";
                                phy-handle = <&switch0phy1>;
                        };

                        port@3 {
                                reg = <3>;
                                label = "lan1";
                                phy-handle = <&switch0phy2>;
                        };

But the layout on the board actually has {[wan],[lan0]} {lan1}.  A correction would relabel the ports that, I assume, others are using.  Armbian's 5.70 image clumps them all together in br0, so new users will not be affected, but upgrades to the DTS would change the naming of ports, potentially catastrophically for users with an Espressobin firewall or in a remote location.

 

Which is to say, I know how to solve the problem for me, but not how to do so for everyone.

[cpwrunner]

Hi Everyone,

 

Presently I am running a number of espressobins as wireless access point using openwrt. Similar to how armbian provision's the espressobin's network interfaces openwrt seems to disable the topaz switch of the espressobin and then create three individual network adapters wan, lan0 and lan1.  I have run into a problem where clients can join the wireless networks which the espressobin's provide but when I roam, that is disconnect a wireless client from one and then connect to another the wireless client will not be able to obtain a DHCP address for 5 minutes. I have noticed that once the message "bridge topology change detected: propogating tcn bdpu" in the log messages. Once this message appears the wireless clients appear to be able to obtain a DHCP address. Presumably since openwrt bridges the wireless adapter (wlan0) with the wan adapter which is the uplink port (wan interface) for each on each espressobin wireless access point, the bridge (br-wan) may be somehow allowing the DHCPOFFER packets to traverse the bridge once the arp entries have timedout.  The problem is that there is no indication from the arp tables of the espressobin's that this is indeed what is happening. According to the arptables for br-wan (wlan0 and wan bridged together) as a wireless client roams the arp entries seem to changed well before the bridge topology log message appears and according to these tables the bridge appears to behave as expected.

 

It appears that armbian does the same thing that openwrt does, turns off the topaz switch and creates three separate interfaces. My question is, is it possible that the switch behavior has not been completely turned off and is somehow causing the type of behavior where DHCP and arp traffic is dropped until arp entries timeout as I have described and has been described on the espressobin wiki here: http://espressobin.net/forums/topic/are-lan0-and-lan1-bridged-in-hardware/ ?

 

Thanks

 

SOLVED/WORKAROUND: I expermimented with setting the ageing parameter of the bridge between the wlan0 and wan to a much lower value 1000 instead of 30000, this allowed for fast roaming between the APs or at least a much lower delay because the arp entries for the clients will timeout more quickly. This parameter is set in /sys/class/net/br-wan/bridge/ageing or something like that.  

 

[skandalfo]

Modern kernels have likely switched the driver of the topaz switch to a switchdev based driver.

 

The idea is that you set up a bridge interface (br0) over the individual port interfaces and the kernel infrastructure takes care of setting up the offloading of the switching to the hardware.

 

My EspressoBIN is not on right now, but IIRC the default setup was for br0 to be automatically set up on boot (not really sure about that).

[ManoftheSea]

On 1/18/2019 at 5:07 PM, martinayotte said:

I see that it is for config/kernel/linux-mvebu-dev.config, but be careful : for ExpressoBin, it should be 7 config/kernel/linux-mvebu64-dev.config

It turned out I had edited the "dev" config rather than the "next" config, also.  Since then, I added #1252 which was accepted.  However, it seems nothing's built since then.  Is that my fault (some error in some log?)  Is there somewhere I should look, to assist in fixing the problem?  Is there some other reason builds aren't "daily" but are perhaps backlogged?

[martinayotte]

15 minutes ago, ManoftheSea said:

However, it seems nothing's built since then.

What kind of error you're facing  ?

[Igor]

44 minutes ago, ManoftheSea said:

However, it seems nothing's built since then. 

Nightly builds are made if nothing breaks. If it breaks ... I need to fix it, since there is no backup for this role. Sometimes I don't have time for that.

 

46 minutes ago, ManoftheSea said:

Is there somewhere I should look, to assist in fixing the problem?

Short term - if you already sent a PR - no. Long term: https://forum.armbian.com/staffapplications/ or https://www.armbian.com/get-involved Those who work on the project barely sustain current levels.

[FlashBurn]

I just got my EspressoBin V7 and did some tests with it.

 

I flashed the current uboot as told on the download page. Then I measured the performance of the board with sbc-bench (from tkaiser) and found that something is wrong with the cpu frequency.

 

This is the console output from the "flash-image-DDR4-1g-1cs-1200_750.bin":

TIM-1.0
WTMI-devel-18.12.0-a0a1cb8
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.260V
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v1.5(release):1f8ca7e (Marvell-devel-18.12.2)
NOTICE:  BL1: Built : 13:50:24, Dec 26 2018
NOTICE:  BL1: Booting BL2
NOTICE:  BL2: v1.5(release):1f8ca7e (Marvell-devel-18.12.2)
NOTICE:  BL2: Built : 13:50:25, Dec 26 2018
NOTICE:  BL1: Booting BL31
NOTICE:  BL31: v1.5(release):1f8ca7e (Marvell-devel-18.12.2)
NOTICE:  BL31: Built : 13

U-Boot 2018.03-devel-18.12.3-gc9aa92c-armbian (Dec 26 2018 - 13:45:06 +0100)

Model: Marvell Armada 3720 Community Board ESPRESSOBin
       CPU     1200 [MHz]
       L2      800 [MHz]
       TClock  200 [MHz]
       DDR     750 [MHz]
DRAM:  1 GiB
Comphy chip #0:
Comphy-0: USB3          5 Gbps
Comphy-1: PEX0          2.5 Gbps
Comphy-2: SATA0         6 Gbps
Target spinup took 0 ms.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs
PCIE-0: Link down
MMC:   sdhci@d0000: 0
Loading Environment from SPI Flash... SF: Detected mx25u3235f with page size 256 Bytes, erase size 64 KiB, total 4 MiB
OK
Model: Marvell Armada 3720 Community Board ESPRESSOBin
Net:   eth0: neta@30000 [PRIME]
Hit any key to stop autoboot:  0

And this is the link to the log output of the sbc-bench.sh run:

 

sbc-bench.sh log output

 

As one can see the cpu frequency is under 800MHz although I used the uboot for 1200MHz. Even the performance suggest that the board is not even running with 800MHz.

 

This is the console output from the "flash-image-DDR4-1g-1cs-1000_800.bin":

TIM-1.0
WTMI-devel-18.12.0-a0a1cb8
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.132V
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v1.5(release):1f8ca7e (Marvell-devel-18.12.2)
NOTICE:  BL1: Built : 13:50:08, Dec 26 2018
NOTICE:  BL1: Booting BL2
NOTICE:  BL2: v1.5(release):1f8ca7e (Marvell-devel-18.12.2)
NOTICE:  BL2: Built : 13:50:09, Dec 26 2018
NOTICE:  BL1: Booting BL31
NOTICE:  BL31: v1.5(release):1f8ca7e (Marvell-devel-18.12.2)
NOTICE:  BL31: Built : 13:5

U-Boot 2018.03-devel-18.12.3-gc9aa92c-armbian (Dec 26 2018 - 13:45:06 +0100)

Model: Marvell Armada 3720 Community Board ESPRESSOBin
       CPU     1000 [MHz]
       L2      800 [MHz]
       TClock  200 [MHz]
       DDR     800 [MHz]
DRAM:  1 GiB
Comphy chip #0:
Comphy-0: USB3          5 Gbps
Comphy-1: PEX0          2.5 Gbps
Comphy-2: SATA0         6 Gbps
Target spinup took 0 ms.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs
PCIE-0: Link down
MMC:   sdhci@d0000: 0
Loading Environment from SPI Flash... SF: Detected mx25u3235f with page size 256 Bytes, erase size 64 KiB, total 4 MiB
OK
Model: Marvell Armada 3720 Community Board ESPRESSOBin
Net:   eth0: neta@30000 [PRIME]
Hit any key to stop autoboot:  0

And this is the link to the log output of the sbc-bench.sh run:

 

sbc-bench.sh log output

 

As one can see the cpu frequency is circa 800MHz although I used the uboot for 1000MHz. Even the performance suggest that the board is running with 800MHz.

 

If someone is putting me in the right direction I can try to compile another uboot/kernel and test again.

[ManoftheSea]

My numbers look very similar to yours.  I'm using the 1000-1000 u-boot, and it looks like I'm doing about 6% better

http://ix.io/1Ars

 

[FlashBurn]

If I trust the log file and the measurements behind it, your board is also just running with 800MHz. Now it would be good if someone if a V5 board could run the benchmark and post their result. So we could find out if the problem with the cpu frequency is just a problem with the new revision or if it is a general problem.

[ebin-dev]

@FlashBurn, @ManoftheSea Did you adapt the settings in /etc/default/cpufrequtils ? The default ondemand governor settings are for a 1000Mhz CPU max clock speed and have to be adapted to other 'available frequency steps' in case you use 1200MHz (see the output of 'cpufreq-info' for a 1000_800 bootloader):

 

cat /etc/default/cpufrequtils

# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=250000
MAX_SPEED=1000000
GOVERNOR=ondemand

cpufreq-info

cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1
  CPUs which need to have their frequency coordinated by software: 0 1
  maximum transition latency: 0.97 ms.
  hardware limits: 200 MHz - 1000 MHz
  available frequency steps: 200 MHz, 250 MHz, 500 MHz, 1000 MHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance, schedutil
  current policy: frequency should be within 250 MHz and 1000 MHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 250 MHz (asserted by call to hardware).
  cpufreq stats: 200 MHz:0.00%, 250 MHz:80.32%, 500 MHz:0.60%, 1000 MHz:19.08%  (3243)
analyzing CPU 1:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1
  CPUs which need to have their frequency coordinated by software: 0 1
  maximum transition latency: 0.97 ms.
  hardware limits: 200 MHz - 1000 MHz
  available frequency steps: 200 MHz, 250 MHz, 500 MHz, 1000 MHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance, schedutil
  current policy: frequency should be within 250 MHz and 1000 MHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 250 MHz (asserted by call to hardware).
  cpufreq stats: 200 MHz:0.00%, 250 MHz:80.32%, 500 MHz:0.60%, 1000 MHz:19.08%  (3243)

 

Edit: sbc-bench results for a V5_0_1 board (1000_800) are here. The 'measured' CPU frequency is about 800MHz. The OpenSSL benchmark is also down by about 20% compared to values achieved in August 2017 (different bootloader). I think that this decline in performance was introduced some time ago by changes in the bootloader in order to get rid of the stability issues observed by some users.

[FlashBurn]

@ebin-dev

 

No I did not know about this, but this does not explain the results for 1000MHz.

 

I had a quick look at your benchmark result and also there it seems the cpu is just running at 800 MHz. Both the measured frequence and the performance indicate that the cpu is not running with 1000 MHz.

[FlashBurn]

I´m still trying to figure out what is wrong with the performance/cpu frequency.

 

I also did try the current armbian uboot with arch linux and this is my result from 7zip:

[alarm@alarm ~]$ uname -a
Linux alarm 4.20.7-1-ARCH #1 SMP PREEMPT Wed Feb 6 18:58:45 MST 2019 aarch64 GNU/Linux
[alarm@alarm ~]$ 7z b

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C,Utf16=off,HugeFiles=on,64 bits,2 CPUs LE)

LE
CPU Freq: 64000000 - - - - - - - -

RAM size:     993 MB,  # CPU hardware threads:   2
RAM usage:    441 MB,  # Benchmark threads:      2

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       1040   166    610   1012  |      25746   200   1102   2198
23:       1017   172    604   1037  |      25251   200   1095   2186
24:       1002   176    612   1078  |      24677   199   1086   2166
25:        990   183    619   1130  |      24088   200   1074   2144
----------------------------------  | ------------------------------
Avr:             174    611   1064  |              200   1089   2174
Tot:             187    850   1619

This is a result as one would expect from 7zip running with 1200MHz (according to the results from tkaiser).

 

I got sbc-bench.sh running under arch, but now I´m not able anymore to reproduce this result

 

So if anyone got a tip what I could try.

 

Is it possible that the cpu throttles because of temperature?

[FlashBurn]

Ok, this is as far as I got today:

 

I can reproduce the expected results in arch linux with 1200MHz. This is a log from sbc-bench.sh:

 

sbc-bench.sh log output

 

The problem seems to be the cpu frequency scaling. If I don´t change the cpu governor in arch, then it seems to performance and this one seems not to change the cpu frequency and it stays as put in by uboot.

I modified sbc-bench as to not change the cpu frequency scaling and voila I get the expected results.

 

If the cpu frequency scaling ever worked, something changed somewhere (uboot? kernel?) and this is the problem.

 

I will try to look further if I can find the problem.

 

Edit::

Just finished reading some pages back of this thread and there you already came to the same conclusion that something is wrong with the cpu scaling

 

Has anyone tried to investigate further what the problem is and/or where one has to look?

Edited February 9, 2019 by FlashBurn

[Igor]

 

16 hours ago, FlashBurn said:

Is it possible that the cpu throttles because of temperature?

That is the idea, yes. If it throttles abnormally, perhaps thermal points or measurements are wrong?
 

15 hours ago, FlashBurn said:

you already came to the same conclusion that something is wrong with the cpu scaling

You are using ArchLinux kernel which can and it is different. Use only Armbian (kernel) if you want help on Armbian forum - there is already enough complexity and Espressobin V7 seems to need some adjustments below (atf/u-boot) for proper functioning.

[FlashBurn]

19 minutes ago, Igor said:

You are using ArchLinux kernel which can and it is different. Use only Armbian (kernel) if you want help on Armbian forum - there is already enough complexity and Espressobin V7 seems to need some adjustments below (atf/u-boot) for proper functioning.

 

I just used arch linux as another test to find out where the problem could be and I also was able to reproduce the good benchmark values on armbian too. With the armbian kernel it is the same, as long as the cpu frequency scaling is not used, the performance is as expected. And I assume this is right for all board revisions.

 

I was able to compile my own kernel and uboot and will now check if I can find something out. But this will take some time as there is no documentation for the cpu available.

[FlashBurn]

I think I found the problem, but at the moment I can´t get my board to run longer than a minute and so can´t test it.

 

The mainline kernel is missing a patch from marvell´s kernel tree:

 

https://github.com/MarvellEmbeddedProcessors/linux-marvell/commit/80d4cec4cef8282e5ac3aaf98ce3e68fb299a134

 

The problem seems to be that the code detects a wrong XTAL with 40MHz, but the frequency of the XTAL is just 25MHz. With this wrong assumption the PLL values are calculated wrong.

 

So maybe someone could give this patch a try and tell back if it fixes the problem with the wrong cpu frequency.

[FlashBurn]

I tested a kernel with the applied patch for correctly recognize the XTAL, but this did not help with setting the right cpu frequency

 

I will try to have a look what uboot does and maybe I find something what is different to what the kernel does.

 

Can someone tell an uboot and kernel version which worked?

[ebin-dev]

With kernel 4.12.1 and stock firmware 17.02 openssl performance was fine - with kernel 4.14.27 and firmware 17.10 (both versions compiled 15 March 2018 and 4 October 2017) openssl performance was degraded (see here).

[FlashBurn]

I tried to find something which seems off in the bootcode, but everythinh I looked at seems to be correct. I had a look at version 17.06 and 18.12 and the clocks are initialised the same. This is backed up by the fact that the cpu frequency seems right as long as the kernel does not try to change the frequency.

 

If someone could test the old firmware with a current kernel and tell the result it would maybe help.

 

My next guess is that the wrong clock source is selected in the kernel. There is one function which seems to select a clock source for the cpu and there it could happen that the wrong clock got selected. The file is drivers/clk/mvebu/armada-37xx-periph.c and the function clk_pm_cpu_set_parent(). I will try if I can find out who calls this function, but till now I had no luck in finding that out.

 

I come to this conclusion, because it also seems the cpu is running with the selected (from the firmware image one uses) ram frequency.

 

I will try to try this out with running all 4 firmware images (1200, 1000, 800 and 600) and when the performance is saying that the cpu is running with the ram frequency it is one step to solving this problem.

 

As background information, one can select 4 sources for the clock of the peripherals (tbg-a-s, tbg-a-p, tbg-b-s and tbg-b-p) and with a divider of 1 and the wrong tbg the cpu runs with the wrong frequency.

[FlashBurn]

I had a look at marvell's linux kernel source and I'm pretty sure that cpu frequency scaling worked with version 4.4.52 for 600, 800 and 1000MHz, but not with 1200MHz because the wrong clock source was selected for 1200MHz.

 

So I assume if this kernel set the clock source that the current kernel also sets the clock source and that there is something wrong.

 

I just have to find out where and who sets this ;-)

[FlashBurn]

So the good thing is I found the problem, the bad thing is I don´t know how to fix it

 

The problem is, like I assumed, that the wrong clock source is selected or better to say, no clock source is selected and because of that it is the wrong one.

 

In drivers/cpufreq/armada-37xx-cpufreq.c the dvfs is initialized. The right clock source should be selected with the 2 lines at the end of the function armada37xx_cpufreq_dvfs_setup(). But this does not call the needed function clk_pm_cpu_set_parent() in driver/clk/armada-37xx-periph.c.

 

I fixed it for me with hard coding the right value in the registers.

 

So all that is needed for fixing the problem is that the function clk_pm_cpu_get_parent() needs to be called, to get the right clock source and then clk_pm_cpu_set() needs to be called with that source. I don´t know enough about the clk code to know what needs to be done the right way.

 

The question is now, whom to tell this, so that this could be fixed?

 

This is a sbc-bench.sh run with my hard coded fix and 1200MHz: http://ix.io/1BCD

[FlashBurn]

As I had a deeper look into the clk source code I can say that I know what the programer wanted to do with the last two lines of armada37xx_cpufreq_dvfs_setup(). This is the place where the right TBG clock should be set, but this could never have worked. As in the clk.c file one can see that nothing is done when the parent clock to be set is the same as the one already set.

 

I will try what happens if I first set NULL as parent and then the old parent. Maybe this will work.