How to make ESPRESSObin v7 stable?

[Anders 4]

Armbianmonitor:

http://ix.io/1Jsj

Hi All!

 

I am having a lot of trouble making my ESPRESSObin v7 1GB DDR4 run without crashes. I have tried both Bionic (5.75 4.19.20) and Stretch (5.75 4.19.20) images, and neither are stable. I have tried with a lot of different PSUs (all switch-mode though) and booting from both microSD and USB sticks. My U-Boot is "flash-image-ddr4-1g-1cs-1000_800.bin".

 

It seems like there is about 50% chance that it will boot without a crash.

 

The type of crashes I get seems random. Ex

http://ix.io/1JGY

 

And then it just freezes.

 

Or this one which includes a proper kernel Oops:

http://ix.io/1JGZ

 

While running, I get errors like this one:

[   71.136812] BUG: Bad page state in process khugepaged  pfn:31d99
[   71.140206] page:ffffffbf00c76640 count:0 mapcount:0 mapping:0000000000002000 index:0x0
[   71.148428] flags: 0x0()
[   71.151049] raw: 0000000000000000 0000000000000000 ffffffbf00c76648 0000000000002000
[   71.159015] raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
[   71.166975] page dumped because: non-NULL mapping
[   71.172329] BUG: Bad page state in process khugepaged  pfn:3261c
[   71.178001] page:ffffffbf00c98700 count:0 mapcount:0 mapping:0000000000002000 index:0x0
[   71.186236] flags: 0x0()
[   71.188853] raw: 0000000000000000 0000000000000000 ffffffbf00c98708 0000000000002000
[   71.196825] raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
[   71.204782] page dumped because: non-NULL mapping

 

 

Once successfully booted, setting the CPUfreq governor to "performance" seems to make it more stable.

 

Am I the only one having problems getting a stable boot on V7?

 

Has anyone of you any tips on how to make it stable? I am willing to try compiling a new kernel if some experimental patches exists.

Edited May 21, 2019 by Anders

[Anders 4]

Here is an update: I just tried replacing the kernel with 4.4.8-armada-17.02-espressobin downloaded from http://espressobin.net/tech-spec/

I put the "Image" file at "/boot/Image", and the "armada-3720-community.dtb" at "/boot/dtb/marvell/armada-3720-community.dtb".

 

Now everything seems fine. I haven't seen any crashes at all so far, and rebooting works fine.

 

root@espressobin:~# uname -a
Linux espressobin 4.4.8-armada-17.02.2-g8148be9 #8 SMP PREEMPT Wed Jul 4 11:59:53 UTC 2018 aarch64 GNU/Linux
root@espressobin:~# 7za b

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs LE)

LE
CPU Freq:   997   997   998   998   997   998   995   996

RAM size:     930 MB,  # CPU hardware threads:   2
RAM usage:    441 MB,  # Benchmark threads:      2

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:        868   152    555    844  |      22465   199    962   1918
23:        859   153    574    876  |      22076   199    958   1911
24:        853   153    599    918  |      21731   200    955   1908
25:        848   153    632    968  |      21340   199    953   1899
----------------------------------  | ------------------------------
Avr:             153    590    902  |              199    957   1909

 

This is probably a stupid question, but why don't we just apply the patches from https://github.com/MarvellEmbeddedProcessors/linux-marvell/tree/linux-4.4.8-armada-17.02-espressobin to the 4.19.20 kernel used by Armbian?

 

What is the current state of that work? Can I help out?

[Anders 4]

I just tried building and booting Linux 5.1.0-g72cf0b074 directly from torvalds/linux.git. It boots fine, but after some time, it crashes with the same kind of errors as the Armbian kernel. Here is one of them:

http://ix.io/1JH1

Edited May 21, 2019 by Anders

[Anders 4]

The situation is the same with the kernel from archlinuxarm, even though it only runs at 800 MHz:

http://ix.io/1JH2

Edited May 21, 2019 by Anders

[chrisf 25]

I had kernel panic issues with my Espressobin, when it was functional.

It worked best with the 1200/750 firmware. It's probably something to do with memory timing/board layout.

[Igor 2451]

14 hours ago, Anders said:

This is probably a stupid question, but why don't we just apply the patches from https://github.com/MarvellEmbeddedProcessors/linux-marvell/tree/linux-4.4.8-armada-17.02-espressobin to the 4.19.20 kernel used by Armbian?

This is exactly what is behind, but keeping it updated with resources that works on Armbian is close to impossible.
 

14 hours ago, Anders said:

What is the current state of that work? Can I help out?

It is summed/documented/resulted in:

• patches on top of mainline: https://github.com/armbian/build/tree/master/patch/kernel/mvebu64-next
• kernel config: https://github.com/armbian/build/blob/master/config/kernel/linux-mvebu64-next.config
• u-boot / ATF building process: https://github.com/armbian/build/blob/master/config/sources/mvebu64.conf (if it's a kernel problem, this can be fully skipped)

... and forum will also provide some clues about that state.

[ebin-dev 65]

13 hours ago, Anders said:

 

I just tried building and booting Linux 5.1.0-g72cf0b074 directly from torvalds/linux.git. It boots fine, but after some time, it crashes with the same kind of errors as the Armbian kernel. Here is one of them:

 

 

Could you flash your bootloader to 600_600 and try again ? If no errors occur in that case we could assume that the bootloader needs to be adapted at least for V7 boards ...

[Anders 4]

@Igor Thanks!

 

 

8 hours ago, ebin-dev said:

Could you flash your bootloader to 600_600 and try again ? If no errors occur in that case we could assume that the bootloader needs to be adapted at least for V7 boards ... 

That seems to work fine. I only ran it for about 30 minutes though.

 

root@espressobin:~# uname -a
Linux espressobin 5.1.0-g72cf0b074 #1 SMP PREEMPT Sun May 19 18:21:04 CEST 2019 aarch64 GNU/Linux
root@espressobin:~# 7za b

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs LE)

LE
CPU Freq:   439   450   368   589   591   505   585   487

RAM size:     984 MB,  # CPU hardware threads:   2
RAM usage:    441 MB,  # Benchmark threads:      2

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:        438   122    349    427  |      13482   198    581   1151
23:        524   146    365    534  |      13276   198    580   1149
24:        520   147    381    560  |      12839   195    577   1127
25:        517   147    402    591  |      12862   198    580   1145
----------------------------------  | ------------------------------
Avr:             141    374    528  |              197    580   1143
Tot:             169    477    835

I'm confused. I do not understand what this has to do with the U-Boot bootloader? 1000_800 worked fine with 4.4.8-armada-17.02-espressobin kernel.

 

My board came with "U-Boot 2017.03-armada-17.10.2-g14aeedc" preinstalled, which on boot reported:

CPU    @ 1000 [MHz]
L2     @ 800 [MHz]
TClock @ 200 [MHz]
DDR    @ 800 [MHz]

 

Which is why I choose to flash "flash-image-ddr4-1g-1cs-1000_800.bin".

Espressobin v7 should run at 1.2 GHz, so I just tried flashing "flash-image-ddr4-1g-1cs-1200_750.bin", but now my board does not boot:

Marvell>> bubt flash-image-ddr4-1g-1cs-1200_750.bin spi usb
Burning U-BOOT image "flash-image-ddr4-1g-1cs-1200_750.bin" from "usb" to "spi"
USB0:   Register 2000104 NbrPorts 2
Starting the controller
USB XHCI 1.00
USB1:   USB EHCI 1.00
scanning bus 0 for devices... 1 USB Device(s) found
scanning bus 1 for devices... 2 USB Device(s) found
Image checksum...OK!
SF: Detected mx25u3235f with page size 256 Bytes, erase size 64 KiB, total 4 MiB
Erasing 917504 bytes (14 blocks) at offset 0 ...Done!
Writing 889028 bytes from 0x8000000 to offset 0 ...Done!
Marvell>> 
Marvell>> 
Marvell>> reset
resetting ...
TIM-1.0
WTMI-devel-18.12.0-a0a1cb8
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 0.898V

It never gets any further than "SVC REV: 5, CPU VDD voltage: 0.898V" - any ideas?

[ebin-dev 65]

18 hours ago, Anders said:

That seems to work fine. I only ran it for about 30 minutes though.

 

We are desperately waiting for the next release of u-boot / atf / A3700-utils by Marvell (enhanced DDR tuning for all EspressoBins is expected). Together with the pending patch performance may increase by about 20 % and stability of V7 EspressoBins will hopefully be established too ...

[kostap 2]

I have updated the 18.12 branch with patches from current development stream. Please check if it improves the stability.

https://github.com/MarvellEmbeddedProcessors/A3700-utils-marvell/commits/A3700_utils-armada-18.12

[ebin-dev 65]

@kostap Thank you very much !

@Anders Please find the updated bootloader here (u-boot is unchanged; WTMI is updated to 18.12.1) - it works fine on my V5_0_1 EspressoBin (https://pastebin.com/xJCtLXsH). It should solve the stability issues ...

 

Edit: the recovery images are also updated: sata-images and uart-images

Edit: links changed ; the images are now available through Armbian servers

[barish 8]

I got my EspressoBin-V7 yesterday and spent most of today to get it "stable" with Armbian. I haven't succeeded so far, although with your patch @ebin-dev, I got an uptime of almost 10 mins, which is an absolute record for now... I tried 1000-800 and 800-800-Mhz so far. I am running the Armbian provided Debian kernel on it (4.19.20). I tried what @Anders describes above and replace the kernel and dts file, but for me, it doesn't work: The kernel gets stuck at the very beginning

[    0.000000] bootconsole [uart0] enabled

was the last message of boot process.

 

Update: I just tried the flash-image-ddr4-1g-1cs-1200_750.bin and this broke u-boot completely. I will have to do a rescue with WtpDownload_linux tomorrow...

[ManoftheSea 6]

@barish, I was recently trying to debug a similar thing.  Do you have "verbosity=1" in /boot/armbianEnv.txt?  If so, you're telling the kernel not to log.  I set mine to 7 based on a very poor understanding of things I read, and it gives me more log.  If you already knew this, then maybe my message will help someone else sometime.

[ebin-dev 65]

@barish Please try the new 600_600 flash image.

[barish 8]

Thanks for your replies. I spent many hours today to upload a recovery image onto the EspressoBin, but no success. Usually, after a few seconds, the upload stops (no error message), longest was about 30s till stop. I am not sure if this has to do with the fact, that I am using a Virtual Box Debian guest on a MacOS host, so that the serial connection might not be as perfectly timed as necessary? I have never had any problems with this and a serial connection at 115200 baud is not rocket science really (I tried lower baud rates too...).

 

If I can't find a solution for this, the EspressoBin is going to take its way back to GlobalScale.

 

Update:

Ok VirtualBox is one part of the problem. I installed the Prolific drivers for MacOS, I compiled WtpDownload for MacOS (which went really smoothly), and *most* of the time, I can now accomplish a recovery upload. The 600_600 from 18.12v2 got me back to the Marvell>> prompt, which I really welcomed when I saw it. So I "bubt" the 600_600 u-boot and started setting the env parameters, but in the middle of that the board crashed and rebooted...!

 

I am about to give it up. I will try once more setting the env and booting from SD, but that's the last chance I am willing to give to the V7 board...

Edited May 22, 2019 by barish
update

[barish 8]

Ok, that's it. Twice more the board crashed when I tried to set the env parameters.

 

BTW I tried three different 12V power supplies, all have operated reliably in the past, one delivers about 11,5V, one 12,1V, one 12,5V, all 2A or more.

 

The board is sitting on the bottom part of the original housing, which includes a large heat sink, so it has never become hot (I'd say below 30°C). I am not sure if I should try another specimen or get a refund... or try a V5, they are supposed to be more reliable, are they.

[ebin-dev 65]

On 5/22/2019 at 10:27 PM, barish said:

I am not sure if I should try another specimen or get a refund... or try a V5, they are supposed to be more reliable, are they.

 

You could ask for a replacement V7 board AND for a V5 board as some kind of compensation for all the trouble you had so far.

My V5_0_1 boards are running 24/7 (with Armbian Stretch, mainline-stable, kernel 4.19.20) ... 

[Anders 4]

@barish I managed to fix mine by following those two guides: https://dl.armbian.com/espressobin/u-boot/rescue/uart-rescue-espressobin.pdf and http://wiki.espressobin.net/tiki-index.php?page=Bootloader+recovery+via+UART+-+v7&highlight=recovery

I used the files from https://dl.armbian.com/espressobin/u-boot/rescue/uart-images-18.12-v2/uart-images-ddr4-1g-1cs-1000_800.tgz

 

@ebin-dev Using flash-image-ddr4-1g-1cs-600_600.bin from 2019-05-21 seems fine so far:

root@espressobin:~# uname -a
Linux espressobin 4.19.20-mvebu64 #5.75 SMP PREEMPT Fri Feb 8 09:54:18 CET 2019 aarch64 aarch64 aarch64 GNU/Linux
root@espressobin:~# 7za b

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs LE)

LE
CPU Freq:   296   296   296   296   296   296   294

RAM size:     990 MB,  # CPU hardware threads:   2
RAM usage:    441 MB,  # Benchmark threads:      2

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:        289   143    197    282  |       6691   199    287    571
23:        286   142    205    292  |       6616   199    287    573
24:        284   143    214    306  |       6553   199    289    575
25:        283   143    227    324  |       6476   199    290    576
----------------------------------  | ------------------------------
Avr:             143    211    301  |              199    288    574
Tot:             171    249    437

root@espressobin:~# uname -a
Linux espressobin 5.1.0-g72cf0b074 #1 SMP PREEMPT Sun May 19 18:21:04 CEST 2019 aarch64 GNU/Linux
root@espressobin:~# 7za b

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs LE)

LE
CPU Freq:   594   594   594   594   595   595   594   595

RAM size:     984 MB,  # CPU hardware threads:   2
RAM usage:    441 MB,  # Benchmark threads:      2

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:        532   147    353    518  |      13527   198    582   1155
23:        525   147    365    536  |      13303   198    581   1152
24:        521   147    381    561  |      13050   198    579   1146
25:        518   147    402    593  |      12934   198    580   1151
----------------------------------  | ------------------------------
Avr:             147    375    552  |              198    580   1151
Tot:             173    478    851

 

[ebin-dev 65]

22 minutes ago, Anders said:

Using flash-image-ddr4-1g-1cs-600_600.bin from 2019-05-21 seems fine so far:

 

Thank you for your feedback. Do you realise that the CPU clock speed is different with kernel 4.19.20-mvebu64 (about 300MHz) and with 5.1.0-g72cf0b074 (about 600MHz) ? (see my comment in the other thread)

 

[Anders 4]

@ebin-dev Yes. I do not know why though.

 

Using flash-image-ddr4-1g-1cs-800_800-2019-05-21.bin also seems to work fine:

root@espressobin:~# uname -a 
Linux espressobin 4.19.20-mvebu64 #5.75 SMP PREEMPT Fri Feb 8 09:54:18 CET 2019 aarch64 aarch64 aarch64 GNU/Linux
root@espressobin:~# 7za b

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs LE)

LE
CPU Freq:   793   793   794   794   794   794   794   794

RAM size:     990 MB,  # CPU hardware threads:   2
RAM usage:    441 MB,  # Benchmark threads:      2

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:        726   149    473    706  |      17714   200    758   1512
23:        721   150    488    735  |      17480   200    757   1513
24:        715   151    510    770  |      17197   200    755   1510
25:        711   151    538    812  |      16950   200    755   1509
----------------------------------  | ------------------------------
Avr:             151    502    756  |              200    756   1511
Tot:             175    629   1133

root@espressobin:~# uname -a
Linux espressobin 5.1.0-g72cf0b074 #1 SMP PREEMPT Sun May 19 18:21:04 CEST 2019 aarch64 GNU/Linux
root@espressobin:~# 7za b

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs LE)

LE
CPU Freq:   647   794   794   794   794   794   794   794

RAM size:     984 MB,  # CPU hardware threads:   2
RAM usage:    441 MB,  # Benchmark threads:      2

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:        715   147    474    696  |      17894   196    778   1528
23:        707   147    490    721  |      17845   198    778   1545
24:        702   148    512    755  |      17546   198    776   1540
25:        698   148    541    798  |      17261   198    774   1536
----------------------------------  | ------------------------------
Avr:             147    504    742  |              198    777   1537
Tot:             173    640   1140

 

 

flash-image-ddr4-1g-1cs-1000_800-2019-05-21.bin also works fine so far, but only runs at 800 mhz?:

root@espressobin:~# uname -a
Linux espressobin 4.19.20-mvebu64 #5.75 SMP PREEMPT Fri Feb 8 09:54:18 CET 2019 aarch64 aarch64 aarch64 GNU/Linux
root@espressobin:~# 7za b

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs LE)

LE
CPU Freq:   793   794   794   794   794   794   794   794

RAM size:     990 MB,  # CPU hardware threads:   2
RAM usage:    441 MB,  # Benchmark threads:      2

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:        730   150    472    711  |      17719   200    758   1513
23:        720   150    488    734  |      17447   199    757   1510
24:        716   151    510    770  |      17170   199    756   1507
25:        711   151    538    813  |      16964   200    756   1510
----------------------------------  | ------------------------------
Avr:             151    502    757  |              199    757   1510
Tot:             175    630   1133

root@espressobin:~# uname -a
Linux espressobin 5.1.0-g72cf0b074 #1 SMP PREEMPT Sun May 19 18:21:04 CEST 2019 aarch64 GNU/Linux
root@espressobin:~# 7za b

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs LE)

LE
CPU Freq:   794   794   793   794   794   794   794   794

RAM size:     984 MB,  # CPU hardware threads:   2
RAM usage:    441 MB,  # Benchmark threads:      2

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:        714   147    473    695  |      18084   198    778   1544
23:        707   147    490    720  |      17813   198    777   1542
24:        702   148    512    756  |      17451   198    775   1532
25:        699   148    541    798  |      17261   198    775   1536
----------------------------------  | ------------------------------
Avr:             147    504    742  |              198    776   1539
Tot:             173    640   1140

 

 

 

 

 

But sometimes it freezes very early in the boot process. This happened with all versions:

Marvell>> reset
resetting ...
TIM-1.0
WTMI-devel-18.12.1-e6bb176
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.073V
(no more output)

 

flash-image-ddr4-1g-1cs-1200_750-2019-05-21.bin killed my board again. Same as above, but last line is: "SVC REV: 5, CPU VDD voltage: 0.898V".

 

Also, I haven't tested with the patches from https://github.com/MarvellEmbeddedProcessors/linux-marvell/pull/19 yet. Will do that once I recover my board.

Edited May 28, 2019 by Anders
Add tests for other images.

[ebin-dev 65]

1 hour ago, Anders said:

Also, I haven't tested with the patches from https://github.com/MarvellEmbeddedProcessors/linux-marvell/pull/19 yet.

 

@Anders The reason for the different CPU clock speeds with kernel 4.19.20 and 5.1.0 is that the above patches already made it into the kernel (since 4.19.4x). Therefore with newer kernels the CPU is clocked as expected. In kernel 4.19.20 (and with earlier kernels) the CPU was clocked with lower speeds (800MHz instead of 1000MHz, 300MHz instead of 600MHz) as you have seen yourself.

 

Now the new boot loader has to (ddr) tune the system such that it is stable under these new circumstances.

 

1 hour ago, Anders said:

flash-image-ddr4-1g-1cs-1000_800-2019-05-21.bin also works fine so far, but only runs at 800 mhz?:

 

Would you please test the 1000_800 image again (from here) ? I am surprised that the CPU only runs at 800 MHz in this case. Please press the reset button once you have flashed the 1000_800 image ...

 

(You need to adapt the values in /etc/default/cpufrequtils to available values shown by cpufreq-info and to reboot)

[barish 8]

@Anders I am happy for you that the u-boot and kernel work for you. I did exactly what you did with all different u-boots and never got a stable board, not even a stable pre-kernel prompt (marvell>>). The only thing that was (mostly) stable was the pre-u-boot prompt (>)... but even then I had crashes during upload with WtpDownload, which may also be due to the fact that Prolific PL2303 drivers for MacOS (and for Windows) are quite buggy and lead to system crashes of the host (I had twice a detached process of MacOS with 100% cpu on one core, completely un-killable, I had to reboot the machine). I can't really understand why the EspressoBin uses this very cheap USB-serial-chip.

OT: For me, it's not a good sign that the very only and first GlobalScale-board I ordered is so completely unreliable (I was thinking of using it for a series, this idea is more than dead now).

[Anders 4]

On 5/28/2019 at 10:17 AM, ebin-dev said:

 

@Anders The reason for the different CPU clock speeds with kernel 4.19.20 and 5.1.0 is that the above patches already made it into the kernel (since 4.19.4x). Therefore with newer kernels the CPU is clocked as expected. In kernel 4.19.20 (and with earlier kernels) the CPU was clocked with lower speeds (800MHz instead of 1000MHz, 300MHz instead of 600MHz) as you have seen yourself.

 

Now the new boot loader has to (ddr) tune the system such that it is stable under these new circumstances.

 

 

Would you please test the 1000_800 image again (from here) ? I am surprised that the CPU only runs at 800 MHz in this case. Please press the reset button once you have flashed the 1000_800 image ...

 

(You need to adapt the values in /etc/default/cpufrequtils to available values shown by cpufreq-info and to reboot)

 

@ebin-dev Using flash-image-ddr4-1g-1cs-1000_800-2019-05-21.bin (md5: 1b0cc3d743be6e5881d0f9010df79f4f), and mainline Linux with the config from https://github.com/armbian/build/blob/master/config/kernel/linux-mvebu64-next.config this is what I get:

Marvell>> version   
U-Boot 2018.03-devel-18.12.3-gc9aa92c-armbian (Feb 20 2019 - 09:45:04 +0100)

aarch64-linux-gnu-gcc (Linaro GCC 7.4-2019.02) 7.4.1 20181213 [linaro-7.4-2019.02 revision 56ec6f6b99cc167ff0c2f8e1a2eed33b1edc85d4]
GNU ld (Linaro_Binutils-2019.02) 2.28.2.20170706


root@espressobin:~# uname -a
Linux espressobin 5.2.0-rc2+ #3 SMP PREEMPT Fri May 31 18:28:10 CEST 2019 aarch64 GNU/Linux
root@espressobin:~# cat /etc/default/cpufrequtils
# WARNING: this file will be replaced on board support package (linux-root-...) upgrade
ENABLE=true
MIN_SPEED=200000
MAX_SPEED=1300000
GOVERNOR=ondemand
root@espressobin:~# cpufreq-info
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1
  CPUs which need to have their frequency coordinated by software: 0 1
  maximum transition latency: 0.97 ms.
  hardware limits: 200 MHz - 1000 MHz
  available frequency steps: 200 MHz, 250 MHz, 500 MHz, 1000 MHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance, schedutil
  current policy: frequency should be within 200 MHz and 1000 MHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 200 MHz (asserted by call to hardware).
  cpufreq stats: 200 MHz:81.44%, 250 MHz:1.75%, 500 MHz:0.79%, 1000 MHz:16.02%  (193)
analyzing CPU 1:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1
  CPUs which need to have their frequency coordinated by software: 0 1
  maximum transition latency: 0.97 ms.
  hardware limits: 200 MHz - 1000 MHz
  available frequency steps: 200 MHz, 250 MHz, 500 MHz, 1000 MHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance, schedutil
  current policy: frequency should be within 200 MHz and 1000 MHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 200 MHz (asserted by call to hardware).
  cpufreq stats: 200 MHz:81.44%, 250 MHz:1.75%, 500 MHz:0.79%, 1000 MHz:16.02%  (193)
root@espressobin:~# 7za b

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs LE)

LE
CPU Freq:   500   794   794   794   794   794   794   794

RAM size:     987 MB,  # CPU hardware threads:   2
RAM usage:    441 MB,  # Benchmark threads:      2

                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:        717   149    469    698  |      18046   200    772   1541
23:        708   149    485    722  |      17712   199    770   1533
24:        703   149    506    757  |      17468   200    769   1534
25:        699   150    534    799  |      17211   199    769   1532
----------------------------------  | ------------------------------
Avr:             149    498    744  |              199    770   1535
Tot:             174    634   1139

If I do "cpufreq-set -g performance", "cpufreq-info" reports: "current CPU frequency is 1000 MHz (asserted by call to hardware).", though 7zip continues to report 794.

 

No immediate crashes so far. Got a crash on boot:

starting version 232
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
Begin: Waiting for root file system ... Begin: Running /scripts/local-block ... done.
[    6.189319] Internal error: Oops: 86000005 [#1] PREEMPT SMP
[    6.192204] Modules linked in:
[    6.195346] CPU: 0 PID: 211 Comm: spi0 Not tainted 5.2.0-rc2+ #3
[    6.201522] Hardware name: Globalscale Marvell ESPRESSOBin Board (DT)
[    6.208154] pstate: 80000005 (Nzcv daif -PAN -UAO)
[    6.213085] pc : 0x100000
[    6.215778] lr : spi_res_release+0x5c/0x98
[    6.219978] sp : ffffff801173bd00
[    6.223382] x29: ffffff801173bd00 x28: 0000000000000000 
[    6.228848] x27: ffffff8011121b30 x26: ffffff8011029000 
[    6.234314] x25: ffffffc03fb5bba0 x24: dead000000000100 
[    6.239780] x23: dead000000000200 x22: ffffff801005b818 
[    6.245245] x21: ffffffc03fb5b800 x20: ffffff801005b7c0 
[    6.250710] x19: ffffffc03fbed040 x18: 0000000000000000 
[    6.256176] x17: 0000000000000000 x16: 0000000000000000 
[    6.261642] x15: 0000000000000000 x14: 0000000000000000 
[    6.267107] x13: 0000000000000001 x12: 0000000000000000 
[    6.272573] x11: 0000000000000001 x10: 0000000000000910 
[    6.278038] x9 : ffffff801173bbe0 x8 : ffffffc03c86e630 
[    6.283504] x7 : ffffffc03c86dd80 x6 : 0000000018f873c1 
[    6.288970] x5 : 0000000000000323 x4 : 0000000000000000 
[    6.294435] x3 : 0000000000100000 x2 : ffffffc03fbed058 
[    6.299901] x1 : ffffff801005b7c0 x0 : ffffffc03fb5b800 
[    6.305368] Call trace:
[    6.307877]  0x100000
[    6.310210]  spi_transfer_one_message+0x25c/0x3c8
[    6.315047]  __spi_pump_messages+0x32c/0x520
[    6.319439]  spi_pump_messages+0x14/0x20
[    6.323474]  kthread_worker_fn+0xac/0x198
[    6.327591]  kthread+0x124/0x128
[    6.330908]  ret_from_fork+0x10/0x1c
[    6.334584] Code: bad PC value
[    6.337717] ---[ end trace c792078f48e6ceda ]---

 

Edited May 31, 2019 by Anders
cpufreq-set -g performance, crash

[ebin-dev 65]

1 hour ago, Anders said:

Got a crash on boot:

 

If you want to learn something you need to test with kernel 4.19.46 (please select next-nightly in armbian-config)

with the new boot loader 1000_800 and 800_800.  /etc/default/cpufrequtils should be adapted to:

cat /etc/default/cpufrequtils

#1000_800 
ENABLE=true
MIN_SPEED=250000
MAX_SPEED=1000000
GOVERNOR=ondemand

Otherwise It is difficult to say if the crashes on boot are caused by kernel 5.2 or by the new boot loader.

[Anders 4]

3 hours ago, ebin-dev said:

If you want to learn something you need to test with kernel 4.19.46 (please select next-nightly in armbian-config)

Done. Result: http://ix.io/1KAv 

No crashes yet in Linux yet, but about 5% of the time U-boot comes no further than:

TIM-1.0
WTMI-devel-18.12.1-e6bb176
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.155V

 

 

On 5/28/2019 at 10:17 AM, ebin-dev said:

The reason for the different CPU clock speeds with kernel 4.19.20 and 5.1.0 is that the above patches already made it into the kernel (since 4.19.4x).

No. The patches from https://github.com/MarvellEmbeddedProcessors/linux-marvell/pull/19 are neither in mainline nor in linux-next.

Update: I tried applying the patches, and now it always fails (Same as the one reported here: http://lists.infradead.org/pipermail/linux-arm-kernel/2019-March/638623.html):

Loading, please wait...
starting version 232
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
Begin: Waiting for root file system ... [    5.771744] Internal error: synchronous parity or ECC error: 86000018 [#1] PREEMPT SMP
[    5.777053] Modules linked in:
[    5.780191] CPU: 1 PID: 23 Comm: kworker/1:1 Not tainted 5.2.0-rc2+ #4
[    5.786906] Hardware name: Globalscale Marvell ESPRESSOBin Board (DT)
[    5.793550] Workqueue:  0x0 (events)
[    5.797211] pstate: 40000085 (nZcv daIf -PAN -UAO)
[    5.802147] pc : dequeue_entity+0x48/0x578
[    5.806351] lr : dequeue_entity+0x24/0x578
[    5.810559] sp : ffffff801139bc80
[    5.813965] x29: ffffff801139bc80 x28: 0000000000000000
[    5.819430] x27: 0000000000000000 x26: ffffffc03fb012c0
[    5.824895] x25: ffffff8010bdf400 x24: ffffffc03fde1bc0
[    5.830360] x23: ffffff8011009000 x22: 0000000000000009
[    5.835827] x21: 0000000157a1bc9e x20: ffffffc03fb00e00
[    5.841292] x19: ffffffc03fde1c40 x18: 0000000000000000
[    5.846758] x17: 0000000000000000 x16: 0000000000000000
[    5.852224] x15: 0000000000000000 x14: 0000000000000000
[    5.857688] x13: 0000000000000001 x12: 0000000000000000
[    5.863154] x11: 0000000000000026 x10: 0101010101010101
[    5.868620] x9 : 0000000000000000 x8 : 7f7f7f7f7f7f7f7f
[    5.874086] x7 : 0000000000000000 x6 : 0000000018f0cb72
[    5.879551] x5 : 00ffffffffffffff x4 : 0000000000000008
[    5.885017] x3 : 000000003b4485a0 x2 : 0000000000000000
[    5.890483] x1 : 0000000000000000 x0 : 00000001579f2800
[    5.895949] Call trace:
[    5.898460]  dequeue_entity+0x48/0x578
[    5.902313]  dequeue_task_fair+0x54/0x590
[    5.906437]  deactivate_task+0x70/0xc8
[    5.910290]  __schedule+0x2a4/0x4c0
[    5.913870]  schedule+0x38/0xc8
[    5.917099]  worker_thread+0xc8/0x460
[    5.920862]  kthread+0x124/0x128
[    5.924177]  ret_from_fork+0x10/0x1c
[    5.927851] Code: f944e800 cb0002b5 cb0102b5 f9406280 (b5001da0)
[    5.934121] ---[ end trace 21831cae888e94ae ]---

 

Edited May 31, 2019 by Anders
Add result

[ebin-dev 65]

10 hours ago, Anders said:

No. The patches from https://github.com/MarvellEmbeddedProcessors/linux-marvell/pull/19 are neither in mainline nor in linux-next.

 

If you have a look at armada-37xx-cpufreq.c in the source tree of kernel 4.19.46 and 5.1.6 you will see that i.e. lines 449, 450 have changed to 

unsigned long u_volt = avs_map[dvfs->avs[load_lvl]] * 1000;
freq = base_frequency / dvfs->divider[load_lvl];

This corresponds to new lines 442, 443 of the patch file. So both kernels were patched (I just took the corresponding tar balls from kernel.org).

 

Thank you for testing kernel 4.19.46 on your V7 EspressoBin. It is not stable with the 1000_800 bootloader - not even at the u-boot prompt, the CPU is clocked with 800MHz instead of 1000MHz and the 1200_750 bootloader even bricks your EspressoBin. I think that the vendor must be made aware of this.

 

If you now switch to the other bootloader 800_800 - is your V7 Espresso stable now with kernel 4.19.46 ?

cat /etc/default/cpufrequtils

#800_800 
ENABLE=true
MIN_SPEED=200000
MAX_SPEED=800000
GOVERNOR=ondemand

 

[Anders 4]

@ebin-dev

> If you have a look at armada-37xx-cpufreq.c in the source tree of kernel 4.19.46 and 5.1.6 you will see that i.e. lines 449, 450 have changed to [...]

You are right, I didn't catch that. But that is only one of the 3 changes (L172, L423) from https://github.com/MarvellEmbeddedProcessors/linux-marvell/pull/19/commits/8fe9a4c4a024a6353e810a1dbb5e4bc78bff60a8

 

> Thank you for testing kernel 4.19.46 on your V7 EspressoBin. It is not stable with the 1000_800 bootloader - not even at the u-boot prompt, the CPU is clocked with 800MHz instead of 1000MHz

With 4.19.46 I haven't seen any Linux crashes yet (been running for about 8 hours), so in that sense it is stable. But yes, 7zip only reports 800 MHz, and u-boot sometimes fails to boot.

 

> 1200_750 bootloader even bricks your EspressoBin. I think that the vendor must be made aware of this.

Alright, anything else that you would like me to test? And what exactly should I report to them?

 

> If you now switch to the other bootloader 800_800 - is your V7 Espresso stable now with kernel 4.19.46 ? 

Using flash-image-ddr4-1g-1cs-800_800-2019-05-21.bin (md5 09a1b1ecf658ed74802bc65806fa92af), u-boot still fails to boot sometimes, with the last line being "SVC REV: 5, CPU VDD voltage: 1.073V".

When the OS is running, everything seems fine so far: http://ix.io/1KCy

 

Also, did you see these two fairly new patches that mentions Espressobin?

http://lists.infradead.org/pipermail/linux-arm-kernel/2019-May/655136.html

http://lists.infradead.org/pipermail/linux-arm-kernel/2019-May/655147.html

 

Thanks for your help by the way

[ebin-dev 65]

1 hour ago, Anders said:

With 4.19.46 I haven't seen any Linux crashes yet (been running for about 8 hours), so in that sense it is stable. But yes, 7zip only reports 800 MHz, and u-boot sometimes fails to boot.

 

That are good news for V7 owners.

 

But it also means that the boot loader needs some finishing touches for V7 (boot loader crashes, 1200_750 bricks the V7, CPU runs with 800MHz instead of 1000MHz) and V3-5 EspressoBins too (1000_800 causes kernel panics on boot, CPU runs with 800MHz instead of 1000MHz). These issues occur at least with kernel 4.19.4x and above (cpufreq patch applied).

 

I don't think that we need to report anything else directly to GST/Marvell.

 

Regarding the other patches in the pipeline:  There were many undocumented/unannounced changes by GlobalScle to the hardware of V5 and V7 EspressoBins. Most of the issues caused by that were already corrected in the boot_loader by Marvell and in Armbian. I hope that @Igor will have a look at those patches some time. As far as I know the required hardware (V7 EspressoBin with soldered emmc) is not available for testing purposes ...

 

Edit: of course everybody is invited to test patches on their hardware and to post observations in the forum.

[Anders 4]

> the boot loader needs some finishing touches for V7 (boot loader crashes, 1200_750 bricks the V7, CPU runs with 800MHz instead of 1000MHz)

@ebin-dev We agree that the first two (boot loader crashes, 1200_750 bricks the V7) are bootloader problems.

But I'm not sure "CPU runs with 800MHz instead of 1000MHz" is a bootloader problem. I just tried flashing flash-image-ddr4-1g-1cs-1000_800-2019-05-21.bin, and booting up 4.4.52-armada-17.10.4-g719fc86-dirty (from http://espressobin.net/tech-spec/) and now 7zip also reports 1000 MHz: http://ix.io/1KCN

[ebin-dev 65]

2 minutes ago, Anders said:

I just tried flashing flash-image-ddr4-1g-1cs-1000_800-2019-05-21.bin, and booting up 4.4.52-armada-17.10.4-g719fc86-dirty (from http://espressobin.net/tech-spec/) and now 7zip also reports 1000 MHz: http://ix.io/1KCN

 

Very interesting ! Performance is up by 25% too. Is it stable ??

[Anders 4]

> Is it stable ??

No: http://ix.io/1KCP