Jump to content

Making EspressoBin V7 work in 2022 ...


fallen_aarch64

Recommended Posts

Beware! There are two different sets of DTB files. One set used internally only by U-Boot, bundled by U-Boot source code and statically linked into U-Boot binary. And then another set included in Linux kernel, used as external files and used for booting.

 

What you write about is I guess those DTB files supplied by Linux kernel and those are unrelated to U-Boot and firmware.

Link to comment
Share on other sites

@Pati!

Your suggestion was well founded and thank you for contributing to moving u-boot to mainline!

 

I've flashed the built image :

Marvell>> bubt u-boot/flash-image-ddr4-1g-1cs-1200_750.220209.bin spi usb
Burning U-BOOT image "u-boot/flash-image-ddr4-1g-1cs-1200_750.220209.bin" from "usb" to "spi"
USB0:   Register 2000104 NbrPorts 2
Starting the controller
USB XHCI 1.00
USB1:   USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
scanning bus 1 for devices... 1 USB Device(s) found
Image checksum...OK!
SF: Detected w25q32dw with page size 256 Bytes, erase size 4 KiB, total 4 MiB
Erasing 1146880 bytes (280 blocks) at offset 0 ...Done!
Writing 1146624 bytes from 0x8000000 to offset 0 ...Done!
Marvell>> 

 

Marvell>> reset

resetting ...
TIM-1.0
mv_ddr-devel-g7753d4b DDR4 16b 1GB 1CS 
WTMI-devel-18.12.1-4392eaf
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.225V
Setting clocks: CPU 1200 MHz, DDR 750 MHz

       #remark        --->  there is a several seconds delay after this line

       #remark        ---> booting continues as normal afterwards


CZ.NIC's Armada 3720 Secure Firmware v2021.09.07-27-g489262b (Feb  9 2022 18:50:42)
Running on ESPRESSObin
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.6(release):v2.6-365-ge0a6a512b
NOTICE:  BL1: Built : 18:52:58, Feb  9 2022
NOTICE:  BL1: Booting BL2
NOTICE:  BL2: v2.6(release):v2.6-365-ge0a6a512b
NOTICE:  BL2: Built : 18:53:02, Feb  9 2022
NOTICE:  BL1: Booting BL31
NOTICE:  BL31: v2.6(release):v2.6-365-ge0a6a512b
NOTICE:  BL31: Built : 18:53:12, Feb  9 2022


U-Boot 2022.04-rc1-00087-gb5c5b9a0be (Feb 09 2022 - 18:44:33 +0100)

DRAM:  1 GiB
Core:  37 devices, 21 uclasses, devicetree: separate
WDT:   Not starting watchdog-timer@8300
Comphy chip #0:
Comphy-0: USB3_HOST0    5 Gbps    
Comphy-1: PEX0          5 Gbps    
Comphy-2: SATA0         6 Gbps    
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs 
PCIe: Link down
MMC:   sdhci@d0000: 0, sdhci@d8000: 1
Loading Environment from SPIFlash... SF: Detected w25q32dw with page size 256 Bytes, erase size 4 KiB, total 4 MiB
OK
Model: Globalscale Marvell ESPRESSOBin Board
Net:   eth0: neta@30000 [PRIME]
Hit any key to stop autoboot:  0 

 

 

It was not necessary for my setup resetting the u-boot variables, including MAC addresses. They remained unchanged.

Are those per port MAC valid?

eth1addr=00:51:82:11:22:01
eth2addr=00:51:82:11:22:02
eth3addr=00:51:82:11:22:03

 

=> printenv
arch=arm
baudrate=115200
board=mvebu_armada-37xx
board_name=mvebu_armada-37xx
boot_a_script=ext4load ${boot_interface} ${devnum}:1 ${scriptaddr} ${prefix}boot.scr;source ${scriptaddr};
boot_prefixes=/boot/
boot_targets=usb sata mmc1 mmc0

...

 

I do not see big functional difference with the Armbian u-boot at this stage, but the VERY good result is that Armbian distribution boots from this u-boot without any adjustment :

...
Model: Globalscale Marvell ESPRESSOBin Board
Net:   eth0: neta@30000 [PRIME]
Hit any key to stop autoboot:  0 
starting USB...
Bus usb@58000: Register 2000104 NbrPorts 2
Starting the controller
USB XHCI 1.00
Bus usb@5e000: USB EHCI 1.00
scanning bus usb@58000 for devices... 2 USB Device(s) found
scanning bus usb@5e000 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
/boot/
1063 bytes read in 1 ms (1 MiB/s)
## Executing script at 06d00000
240 bytes read in 0 ms
21496320 bytes read in 58 ms (353.5 MiB/s)
8864567 bytes read in 25 ms (338.2 MiB/s)
Failed to load '/boot/dtb/marvell/armada-3720-community.dtb'
11127 bytes read in 1 ms (10.6 MiB/s)
## Loading init Ramdisk from Legacy Image at 01100000 ...
   Image Name:   uInitrd
   Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
   Data Size:    8864503 Bytes = 8.5 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 06000000
   Booting using the fdt blob at 0x6000000
   Loading Ramdisk to 3f285000, end 3faf92f7 ... OK
   Using Device Tree in place at 0000000006000000, end 0000000006005b76

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 5.10.60-mvebu64 (root@runner7) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #21.08.1 SMP PREEMPT Wed Aug 25 19:19:23 UTC 2021
[    0.000000] Machine model: Globalscale Marvell ESPRESSOBin Board
[    0.000000] earlycon: ar3700_uart0 at MMIO 0x00000000d0012000 (options '')
[    0.000000] printk: bootconsole [ar3700_uart0] enabled
Loading, please wait...
Starting version 247.3-6
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... d

....

 

It is yet to be seen how stable the board will remain with it or if it improves the stability.

I have another Espressobin v5, which is self-rebooting frequently. I shall build u-boot for it and retest the board stability with it. 

 

How could one take advantage of this mainline u-boot for booting regular Debian, Fedora etc distributions install media?

 

 

 

Link to comment
Share on other sites

7 minutes ago, y52 said:

I've flashed the built image :

Marvell>> bubt u-boot/flash-image-ddr4-1g-1cs-1200_750.220209.bin spi usb

 

Beware, 1.2 GHz CPU is broken and Marvell refused to fix it. Hence kernel maintainers decided to delete/deactive 1.2 GHz mode. So rather stick with 1 GHz mode.

 

9 minutes ago, y52 said:

Setting clocks: CPU 1200 MHz, DDR 750 MHz

       #remark        --->  there is a several seconds delay after this line

       #remark        ---> booting continues as normal afterwards

 

That is correct, at this stage is firmware initializing DDR and running DDR training algorithm.

 

10 minutes ago, y52 said:

Are those per port MAC valid?

eth1addr=00:51:82:11:22:01
eth2addr=00:51:82:11:22:02
eth3addr=00:51:82:11:22:03

 

No, they are invalid. You should clear them. That is why resetting of env variables is suggested.

 

Some espressobin models got assigned 3 MAC addresses, some models only one MAC address. And some models have MAC addresses printed on sticker. So it depends on which model have you bought.

 

13 minutes ago, y52 said:

[    0.000000] printk: bootconsole [ar3700_uart0] enabled
Loading, please wait...
Starting version 247.3-6
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... d

 

It looks like you have incorrect cmdline as it does not continue printing of dmesg output on terminal. Normally it should continue printing kernel dmesg output (lines prefixed with time in square brackets).

 

15 minutes ago, y52 said:

How could one take advantage of this mainline u-boot for booting regular Debian, Fedora etc distributions install media?

 

Modern distributions are using U-Boot distroboot feature or use UEFI. U-Boot distroboot is supported by this new U-Boot but you have to reset env to default value (distroboot is implemented as a u-boot script stored in more envs, which are part of default env). Debian uses U-Boot distroboot feature via its package "flash-kernel" (it has quite misleading name, on Espressobin it does not do any flashing, just prepares boot script compatible for distroboot). About Fedora, SUSE or UEFI based setup, it would be better to ask in Fedora or SUSE forum as I do not know these details.

Link to comment
Share on other sites

2 hours ago, Pali said:

U-Boot and wtmi_app.bin binaries are same for all variants. But if you want to clean them call:

make -C u-boot distclean
make -C mox-boot-builder clean

 

The last clean instruction doesn't work :

root@deb10:/src# make -C mox-boot-builder clean
make: Entering directory '/src/mox-boot-builder'
make -C u-boot clean
make[1]: Entering directory '/src/mox-boot-builder/u-boot'
make[1]: *** No rule to make target 'clean'.  Stop.
make[1]: Leaving directory '/src/mox-boot-builder/u-boot'
make: *** [Makefile:80: clean] Error 2
make: Leaving directory '/src/mox-boot-builder'

 

Is there any error?

Link to comment
Share on other sites

5 minutes ago, y52 said:

The last clean instruction doesn't work :

root@deb10:/src# make -C mox-boot-builder clean
make: Entering directory '/src/mox-boot-builder'
make -C u-boot clean
make[1]: Entering directory '/src/mox-boot-builder/u-boot'
make[1]: *** No rule to make target 'clean'.  Stop.
make[1]: Leaving directory '/src/mox-boot-builder/u-boot'
make: *** [Makefile:80: clean] Error 2
make: Leaving directory '/src/mox-boot-builder'

 

Is there any error?

 

That is a bug. Append additional make argument -i

Link to comment
Share on other sites

10 minutes ago, Pali said:

Beware, 1.2 GHz CPU is broken and Marvell refused to fix it. Hence kernel maintainers decided to delete/deactive 1.2 GHz mode. So rather stick with 1 GHz mode.

This is bad news. Do you mean building another u-boot for 1 Ghz.

While I was busy with reading documentation, my Espressobin @ 1.2 GHz disappeared or crashed. I need rebooting it now.

It frequency scaling seems to be in place :

 root@tiger:~# cpufreq-info  

hardware limits: 200 MHz - 1.20 GHz
  available frequency steps: 200 MHz, 300 MHz, 600 MHz, 1.20 GHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil
  current policy: frequency should be within 200 MHz and 1.20 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 200 MHz (asserted by call to hardware).
  cpufreq stats: 200 MHz:51.79%, 300 MHz:1.51%, 600 MHz:1.46%, 1.20 GHz:45.23%  (188)

..etc

Link to comment
Share on other sites

5 minutes ago, Pali said:

That is a bug. Append additional make argument -i



root@deb10:/src# make -i -C mox-boot-builder clean
make: Entering directory '/src/mox-boot-builder'
make -C u-boot clean
make[1]: Entering directory '/src/mox-boot-builder/u-boot'
make[1]: *** No rule to make target 'clean'.  Stop.
make[1]: Leaving directory '/src/mox-boot-builder/u-boot'
make: [Makefile:80: clean] Error 2 (ignored)
make -C arm-trusted-firmware realclean
make[1]: Entering directory '/src/mox-boot-builder/arm-trusted-firmware'
make[1]: *** No rule to make target 'realclean'.  Stop.
make[1]: Leaving directory '/src/mox-boot-builder/arm-trusted-firmware'
make: [Makefile:81: clean] Error 2 (ignored)
make -C wtmi clean
make[1]: Entering directory '/src/mox-boot-builder/wtmi'
  CLEAN
make[2]: Entering directory '/src/mox-boot-builder/wtmi/a53_helper'
  CLEAN
make[2]: Leaving directory '/src/mox-boot-builder/wtmi/a53_helper'
make[2]: Entering directory '/src/mox-boot-builder/wtmi/reload_helper'
  CLEAN
make[2]: Leaving directory '/src/mox-boot-builder/wtmi/reload_helper'
make[1]: Leaving directory '/src/mox-boot-builder/wtmi'
make -C wtmi/compressed clean
make[1]: Entering directory '/src/mox-boot-builder/wtmi/compressed'
  CLEAN
make[2]: Entering directory '/src/mox-boot-builder/wtmi'
  CLEAN
make[3]: Entering directory '/src/mox-boot-builder/wtmi/a53_helper'
  CLEAN
make[3]: Leaving directory '/src/mox-boot-builder/wtmi/a53_helper'
make[3]: Entering directory '/src/mox-boot-builder/wtmi/reload_helper'
  CLEAN
make[3]: Leaving directory '/src/mox-boot-builder/wtmi/reload_helper'
make[2]: Leaving directory '/src/mox-boot-builder/wtmi'
make[1]: Leaving directory '/src/mox-boot-builder/wtmi/compressed'
make -C mox-imager clean
make[1]: Entering directory '/src/mox-boot-builder/mox-imager'
make[1]: *** No rule to make target 'clean'.  Stop.
make[1]: Leaving directory '/src/mox-boot-builder/mox-imager'
make: [Makefile:84: clean] Error 2 (ignored)
rm -f wtmi_h.bin wtmi_app.bin a53-firmware.bin \
        untrusted-flash-image.bin untrusted-emmc-image.bin trusted-flash-image.bin trusted-uart-image.bin trusted-emmc-image.bin \
        trusted-secure-firmware.bin trusted-secure-firmware-uart.bin trusted-secure-firmware-emmc.bin \
        untrusted-secure-firmware.bin untrusted-secure-firmware-emmc.bin
make: Leaving directory '/src/mox-boot-builder'

Link to comment
Share on other sites

2 minutes ago, y52 said:

Do you mean building another u-boot for 1 Ghz.

 

Yes!

 

1 minute ago, y52 said:

It frequency scaling seems to be in place :

 root@tiger:~# cpufreq-info

 

This tool reports bogus information, do not trust it. The only way how to check CPU speed is to use some benchmark tool.

Link to comment
Share on other sites

21 minutes ago, Pali said:

It looks like you have incorrect cmdline as it does not continue printing of dmesg output on terminal. Normally it should continue printing kernel dmesg output (lines prefixed with time in square brackets).

My terminal output was an excerpt only. It continues as follows :

/boot/
1063 bytes read in 1 ms (1 MiB/s)
## Executing script at 06d00000
240 bytes read in 0 ms
21496320 bytes read in 58 ms (353.5 MiB/s)
8864567 bytes read in 25 ms (338.2 MiB/s)
Failed to load '/boot/dtb/marvell/armada-3720-community.dtb'
11127 bytes read in 1 ms (10.6 MiB/s)
## Loading init Ramdisk from Legacy Image at 01100000 ...
   Image Name:   uInitrd
   Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
   Data Size:    8864503 Bytes = 8.5 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 06000000
   Booting using the fdt blob at 0x6000000
   Loading Ramdisk to 3f285000, end 3faf92f7 ... OK
   Using Device Tree in place at 0000000006000000, end 0000000006005b76

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 5.10.60-mvebu64 (root@runner7) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #21.08.1 SMP PREEMPT Wed Aug 25 19:19:23 UTC 2021
[    0.000000] Machine model: Globalscale Marvell ESPRESSOBin Board
[    0.000000] earlycon: ar3700_uart0 at MMIO 0x00000000d0012000 (options '')
[    0.000000] printk: bootconsole [ar3700_uart0] enabled
Loading, please wait...
Starting version 247.3-6
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... Scanning for Btrfs filesystems
done.
Begin: Waiting for root file system ... Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
done.
Begin: Will now check root file system ... fsck from util-linux 2.36.1
[/sbin/fsck.ext4 (1) -- /dev/sda1] fsck.ext4 -a -C0 /dev/sda1 
/dev/sda1: clean, 42601/17711264 files, 1472968/61885440 blocks
done.
done.
Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... done.

Welcome to Debian GNU/Linux 11 (bullseye)!

[  OK  ] Created slice system-getty.slice.
[  OK  ] Created slice system-modprobe.slice.
[  OK  ] Created slice system-serial\x2dgetty.slice.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Forward Password R…uests to Wall Directory Watch.
[  OK  ] Set up automount Arbitrary…s File System Automount Point.
[  OK  ] Reached target Local Encrypted Volumes.

 

Is it correct one? Otherwise which cmdline should be used?

I don't believe it can print time stamp, as the board lacks the phisical clock HW.

Link to comment
Share on other sites

No there is missing kernel dmesg output. Lines which are prefixed with number in square brackets. It is timestamp since boot (not some real clock). First 5 lines are printed with timestamp zero as you can see, they are printed immediately at boot. You can call cat /proc/cmdline to see current cmdline.

Link to comment
Share on other sites

28 minutes ago, Pali said:
48 minutes ago, y52 said:

Are those per port MAC valid?

eth1addr=00:51:82:11:22:01
eth2addr=00:51:82:11:22:02
eth3addr=00:51:82:11:22:03

 

No, they are invalid. You should clear them. That is why resetting of env variables is suggested.

 

Some espressobin models got assigned 3 MAC addresses, some models only one MAC address. And some models have MAC addresses printed on sticker. So it depends on which model have you bought.

 

I've only one MAC printed on the sticker, which is 

ethaddr=F0:AD:4E:xx:xx:xx

Do you mean setting per port MAC to those ones or different? 

=> setenv eth1addr f0:ad:4b:aa:97:01

=> setenv eth2addr f0:ad:4b:aa:97:02

=> setenv eth3addr f0:ad:4b:aa:97:03

Link to comment
Share on other sites

Just now, y52 said:

I've only one MAC printed on the sticker, which is 

ethaddr=F0:AD:4E:xx

 

If you have only one MAC address then set only ethaddr env variable and delete all other eth*addr variables (delete = set to nothing, e.g. setenv set1addr).

Link to comment
Share on other sites

16 minutes ago, Pali said:

No there is missing kernel dmesg output. Lines which are prefixed with number in square brackets. It is timestamp since boot (not some real clock). First 5 lines are printed with timestamp zero as you can see, they are printed immediately at boot. You can call cat /proc/cmdline to see current cmdline.

root@opiplus:~# cat /proc/cmdline
root=UUID=434bc376-d90e-4b73-9c4c-52ee2544a428 rootwait rootfstype=ext4 console=ttyS0,115200 console=tty1 hdmi.audio=EDID:0 disp.screen0_output_mode=1920x1080p60 panic=10 consoleblank=0 loglevel=1 ubootpart= ubootsource=mmc usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u   sunxi_ve_mem_reserve=0 sunxi_g2d_mem_reserve=0 sunxi_fb_mem_reserve=16 cgroup_enable=memory swapaccount=1

 

 

How could  kernel dmesg output be fixed here?

Link to comment
Share on other sites

Just now, y52 said:

console=ttyS0,115200

 

This is wrong. It must be console=ttyMV0,115200.

 

How to fix it? This string is filled by U-Boot based on either some env variable (try to find it in U-Boot in printenv) or via boot script. So needs to change env variable or boot script.

Link to comment
Share on other sites

1 minute ago, y52 said:

If you manage to find a fix to the build clean procedure, I could rebuid the boot loader again. 

 

That you last attempt via make -i -C mox-boot-builder clean worked fine. It cleaned mox-boot-builder correctly.

Link to comment
Share on other sites

12 hours ago, Pali said:

That you last attempt via make -i -C mox-boot-builder clean worked fine. It cleaned mox-boot-builder correctly.

I am quite confused with the command logs, which say :

make[1]: *** No rule to make target 'clean'.  Stop.
make[1]: Leaving directory '/src/mox-boot-builder/mox-imager'
make: [Makefile:84: clean] Error 2 (ignored)

 

Contrary to 2 previous cleanup makes, which executed the "distclean" target, this one apparently raises an error about the  "clean" target.

 

It does call the binary removes afterwards, but previous errors raise doubt.

rm -f wtmi_h.bin wtmi_app.bin a53-firmware.bin \
        untrusted-flash-image.bin untrusted-emmc-image.bin trusted-flash-image.bin trusted-uart-image.bin trusted-emmc-image.bin \
        trusted-secure-firmware.bin trusted-secure-firmware-uart.bin trusted-secure-firmware-emmc.bin \
        untrusted-secure-firmware.bin untrusted-secure-firmware-emmc.bin
make: Leaving directory '/src/mox-boot-builder'

Link to comment
Share on other sites

13 hours ago, Pali said:

Modern distributions are using U-Boot distroboot feature or use UEFI. U-Boot distroboot is supported by this new U-Boot but you have to reset env to default value (distroboot is implemented as a u-boot script stored in more envs, which are part of default env). Debian uses U-Boot distroboot feature via its package "flash-kernel" (it has quite misleading name, on Espressobin it does not do any flashing, just prepares boot script compatible for distroboot). About Fedora, SUSE or UEFI based setup, it would be better to ask in Fedora or SUSE forum as I do not know these details.

Regular Debian distribution could be an interesting target, taken, that Armbian doesn't officially support Espressobin and Marvell's policy is not known for the future. Thus mainline distributions could be a valid solution running the boards, which support mainline u-boot, not Espressobin only. Could you prompt any doc, explaining how to set Debian with the mainline u-boot?

I've found a generic concept, but can not juge if this is the right way :

https://github.com/ARM-software/u-boot/blob/master/doc/README.distro

Link to comment
Share on other sites

Uff... I'm not sure if there is any Debian doc which explain all these things with U-Boot. But basically standard installation of Debian via debootstrap+qemu to SD card with installing of additional package "flash-kernel" for Espressobin should work fine.

 

That U-Boot document describe how distribution maintainers should prepare their systems and boot scripts to be compatible with U-Boot distroboot; document is not for end-users.

 

Maybe it would be a good idea to write some document / wiki page how to exactly install Debian on SD card for Espressobin...

Link to comment
Share on other sites

15 hours ago, Pali said:

This is wrong. It must be console=ttyMV0,115200.

 

How to fix it? This string is filled by U-Boot based on either some env variable (try to find it in U-Boot in printenv) or via boot script. So needs to change env variable or boot script.

My u-boot env variable shows like this :

=> printenv console
console=console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000

 

This record has been created by Marvell's setup. I found it after the board shipment.

It looks like a correct value, as

root@tiger:~# cat /proc/cmdline
console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 root=UUID=d4014581-f211-49a1-aa70-e3012adab7af rootfstype=ext4 rootwait loglevel=1 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u mtdparts=spi0.0:1536k(uboot),64k(uboot-environment),-(reserved) 

 

The system log also points to :

[  OK  ] Started /etc/rc.local Compatibility.
[  OK  ] Finished Permit User Sessions.
[  OK  ] Started Getty on tty1.
[  OK  ] Started Serial Getty on ttyMV0  <--

 

 

The output of yesterday from "root@opiplus:~# cat /proc/cmdline" was from a different board and should be ignored.

15 hours ago, y52 said:

No there is missing kernel dmesg output. Lines which are prefixed with number in square brackets. It is timestamp since boot (not some real clock). First 5 lines are printed with timestamp zero as you can see, they are printed immediately at boot. You can call cat /proc/cmdline to see current cmdline.

 

Is there a probable confusion between the serial console output, which I pasted here and the kernel dmesg output ?

The latter is like this :

root@tiger:~# dmesg 

[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2e2049cda, max_idle_ns: 440795202628 ns
[    0.000004] sched_clock: 56 bits at 12MHz, resolution 80ns, wraps every 4398046511080ns
[    0.000158] Console: colour dummy device 80x25
[    0.000244] Calibrating delay loop (skipped), value calculated using timer frequency.. 25.00 BogoMIPS (lpj=50000)
[    0.000257] pid_max: default: 32768 minimum: 301
[    0.000329] LSM: Security Framework initializing
[    0.000388] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.000403] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.002096] rcu: Hierarchical SRCU implementation.
[    0.002390] EFI services will not be available.
[    0.002547] smp: Bringing up secondary CPUs ...
[    0.003005] Detected VIPT I-cache on CPU1
[    0.003038] GICv3: CPU1: found redistributor 1 region 0:0x00000000d1d60000
[    0.003092] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.003205] smp: Brought up 1 node, 2 CPUs
[    0.003213] SMP: Total of 2 processors activated.
[    0.003219] CPU features: detected: 32-bit EL0 Support
[    0.003225] CPU features: detected: CRC32 instructions
[    0.003230] CPU features: detected: 32-bit EL1 Support
[    0.012209] CPU: All CPU(s) started at EL2
[    0.012251] alternatives: patching kernel code
[    0.013241] devtmpfs: initialized
[    0.016074] Registered cp15_barrier emulation handler
[    0.016092] Registered setend emulation handler
[    0.016312] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.016331] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.016992] pinctrl core: initialized pinctrl subsystem
[    0.017667] DMI not present or invalid.
[    0.018173] NET: Registered protocol family 16
[    0.019884] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.020003] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.020157] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations

..

[   16.424945] systemd-journald[883]: Received client request to flush runtime journal.
[   16.426921] systemd-journald[883]: File /var/log/journal/50bea5a2341c40588d32c8103dea6e71/system.journal corrupted or uncleanly shut down, renaming and replacing.
[   16.430273] systemd-journald[883]: Failed to open system journal: No space left on device  <<- how could it happen, that

                                                                                                                                                      <<- no space is left on a fresh

                                                                                                                                                      <<- system in /var/log?!!

root@tiger:~# df -h
Filesystem      Size  Used Avail Use% Mounted on

/dev/zram1       49M   47M     0 100% /var/log
 

armbian-truncate-logs was executed, but space hasn't been released.

Feb 10 15:45:01 tiger CRON[1777]: (root) CMD (/usr/lib/armbian/armbian-truncate-logs)

Link to comment
Share on other sites

6 minutes ago, y52 said:

Is there a probable confusion between the serial console output, which I pasted here and the kernel dmesg output ?

 

No. Into serial console output is automatically printed dmesg output.

 

9 minutes ago, y52 said:

This record has been created by Marvell's setup. I found it after the board shipment.

It looks like a correct value, as

root@tiger:~# cat /proc/cmdline
console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 root=UUID=d4014581-f211-49a1-aa70-e3012adab7af rootfstype=ext4 rootwait loglevel=1 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u mtdparts=spi0.0:1536k(uboot),64k(uboot-environment),-(reserved) 

 

Remove hardcoded mtdparts= option. U-Boot already supplied correct one via DT.

 

Missing logs are probably caused by loglevel=1 option. Remove it too.

Link to comment
Share on other sites

Strange, it created a lot of journal files :

root@tiger:/var/log# find /var/log -type f -size +100k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'
/var/log/dpkg.log: 256K
/var/log/lastlog: 290K
/var/log/bootstrap.log: 124K
/var/log/messages: 616K
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005d75e7fbcb946-5d79ae1c5d2a822b.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@41feaef8aaea4aeebdbfd85588bd6f50-0000000000000001-0005d79d74a0c313.journal: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system.journal: 600K
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005d75d4d03b614-562ddac3c029c870.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/user-1000@0005d75b7228bb10-b8023a23f5d777a2.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/user-1000.journal: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005d75e8825ae8b-5268820bb9603cf3.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005ca730c166775-bedeb0d5b1b229d8.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005d75e73375d81-15b39762db0b6f2e.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005d760798dab6d-4d0b0847ef35f164.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005d79d74a45e87-17f9b65b8b4f658d.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005d79e02397686-459a671912da5c7b.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005d75da4175f49-cf171d486f76577c.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005d75eab40e1d7-af36dcd5306d5dee.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@73ec10ac48374e71911b36d7eac0df92-0000000000000001-0005ca730c119e6f.journal: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@4b5289b2517f49b690d2cdd1ddef84fd-0000000000000001-0005ca7256aa7363.journal: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005d79c9dd13f78-44b1da9f9a974704.journal~: 2.5M
/var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005d75e6688df64-4a7f4bb6854d7583.journal~: 2.5M
/var/log/daemon.log: 395K
/var/log/syslog: 1.1M
root@tiger:/var/log# :> /var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@73ec10ac48374e71911b36d7eac0df92-0000000000000001-0005ca730c119e6f.journal
root@tiger:/var/log# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/zram1       49M   44M  1.3M  98% /var/log

Link to comment
Share on other sites

19 minutes ago, Pali said:

Remove hardcoded mtdparts= option. U-Boot already supplied correct one via DT.

 

Missing logs are probably caused by loglevel=1 option. Remove it too.

u-boot doesn't keep them:

=> printenv mtdparts
## Error: "mtdparts" not defined
=> printenv loglevel
## Error: "loglevel" not defined

 

Those values are retrieved from /boot/boot.cmd in bootargs :

 

#Remove hardcoded mtdparts= option. U-Boot already supplied correct one via DT.
#Missing logs are probably caused by loglevel=1 option. Remove it too.
#setenv bootargs "$console root=${rootdev} rootfstype=${rootfstype} rootwait loglevel=${verbosity} usb-storage.quirks=${usbstoragequirks} mtdparts=spi0.0:1536k(uboot),64k(uboot-environment),-(reserved) ${extraargs}"

setenv bootargs "$console root=${rootdev} rootfstype=${rootfstype} rootwait loglevel=${verbosity} usb-storage.quirks=${usbstoragequirks} mtdparts=spi0.0:1536k(uboot),64k(uboot-environment),-(reserved) ${extraargs}"

 

 

How should it be changed? Should the whole "mtdparts=spi0.0:1536k(uboot),64k(uboot-environment)" be removed or only partially? Is this the right way?

setenv verbosity "0"

setenv bootargs "$console root=${rootdev} rootfstype=${rootfstype} rootwait loglevel=${verbosity} usb-storage.quirks=${usbstoragequirks},-(reserved) ${extraargs}"

 

 

Link to comment
Share on other sites

I've modified the bootargs @ boot.cmd as follows :

setenv bootargs "$console root=${rootdev} rootfstype=${rootfstype} rootwait usb-storage.quirks=${usbstoragequirks},-(reserved) ${extraargs}"


root@tiger:/boot# mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr
Image Name:   
Created:      Thu Feb 10 16:29:24 2022
Image Type:   ARM Linux Script (uncompressed)
Data Size:    1293 Bytes = 1.26 KiB = 0.00 MiB
Load Address: 00000000
Entry Point:  00000000
Contents:
   Image 0: 1285 Bytes = 1.25 KiB = 0.00 MiB
root@tiger:/boot# ls -al boot.*
-rw-r--r-- 1 root root 38518 Aug 26 10:46 boot.bmp
-rw-r--r-- 1 root root  1285 Feb 10 16:29 boot.cmd
-rw-rw-r-- 1 root root  1357 Feb 10 16:29 boot.scr

 

 

Yes, the console output has changed (and a lot more of it, sometimes difficult to follow):

   Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
   Data Size:    8864503 Bytes = 8.5 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 06000000
   Booting using the fdt blob at 0x6000000
   Loading Ramdisk to 3f285000, end 3faf92f7 ... OK
   Using Device Tree in place at 0000000006000000, end 0000000006005b76

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 5.10.60-mvebu64 (root@runner7) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #21.08.1 SMP PREEMPT Wed Aug 25 19:19:23 UTC 2021
[    0.000000] Machine model: Globalscale Marvell ESPRESSOBin Board
[    0.000000] earlycon: ar3700_uart0 at MMIO 0x00000000d0012000 (options '')
[    0.000000] printk: bootconsole [ar3700_uart0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000003fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x3fdf2100-0x3fdf3fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000003fffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000]   node   0: [mem 0x0000000004000000-0x00000000041fffff]
[    0.000000]   node   0: [mem 0x0000000004200000-0x000000003fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000003fffffff]
[    0.000000] cma: Reserved 16 MiB at 0x000000003d000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 23 pages/cpu s55768 r8192 d30248 u94208
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: ARM erratum 845719
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 258048
[    0.000000] Policy zone: DMA
[    0.000000] Kernel command line: console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 root=UUID=d4014581-f211-49a1-aa70-e3012adab7af rootfstype=ext4 rootwait usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u,-(reserved) 
[    0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 979680K/1048576K available (13056K kernel code, 1664K rwdata, 4160K rodata, 1984K init, 556K bss, 52512K reserved, 16384K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=2.
[    0.000000]  Trampoline variant of Tasks RCU enabled.
[    0.000000]  Rude variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 192 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000d1d40000
[    0.000000] random: get_random_bytes called from start_kernel+0x31c/0x4dc with crng_init=0
[    0.000000] arch_timer: cp15 timer(s) running at 12.50MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2e2049cda, max_idle_ns: 440795202628 ns
[    0.000004] sched_clock: 56 bits at 12MHz, resolution 80ns, wraps every 4398046511080ns
[    0.008425] Console: colour dummy device 80x25
[    0.013001] Calibrating delay loop (skipped), value calculated using timer frequency.. 25.00 BogoMIPS (lpj=50000)
[    0.023507] pid_max: default: 32768 minimum: 301
[    0.028321] LSM: Security Framework initializing
[    0.033058] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.040627] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.050373] rcu: Hierarchical SRCU implementation.
[    0.055586] EFI services will not be available.
[    0.060378] smp: Bringing up secondary CPUs ...
[    0.065431] Detected VIPT I-cache on CPU1
[    0.065465] GICv3: CPU1: found redistributor 1 region 0:0x00000000d1d60000
[    0.065517] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.065628] smp: Brought up 1 node, 2 CPUs
[    0.087704] SMP: Total of 2 processors activated.
[    0.092533] CPU features: detected: 32-bit EL0 Support
[    0.097833] CPU features: detected: CRC32 instructions
[    0.103105] CPU features: detected: 32-bit EL1 Support
[    0.117234] CPU: All CPU(s) started at EL2

...

Welcome to Debian GNU/Linux 11 (bullseye)!
...

[    7.513677] systemd[1]: Created slice system-getty.slice.
[  OK  ] Created slice system-getty.slice.
[    7.538963] systemd[1]: Created slice system-modprobe.slice.
[  OK  ] Created slice system-modprobe.slice.
[    7.560848] systemd[1]: Created slice system-serial\x2dgetty.slice.
[  OK  ] Created slice system-serial\x2dgetty.slice.

...

[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.

Armbian 21.08.1 Bullseye ttyMV0 

 

Could the whole log be studied in some file after the boot?

 

This is a big progress since we have started working on it!

 

While I am thinking :

root@tiger:~# shutdown -h now

[   93.533252] systemd-shutdown[1]: Powering off.
[   93.537882] kvm: exiting hardware virtualization
[   93.559438] reboot: Power down
ERROR:   a3700_system_off needs to be implemented
PANIC at PC : 0x0000000004023848

 

Is that the board hardware issue or the lack of kernel function implementation?

 

Link to comment
Share on other sites

Now I spotted that you forgot to remove rest of mtdparts ,-(reserved).

 

10 minutes ago, y52 said:

Could the whole log be studied in some file after the boot?

 

Via journalctl. Log should be in some file in /var/log. Some of them are text, but soem (journald) are binary.

 

10 minutes ago, y52 said:

[   93.559438] reboot: Power down
ERROR:   a3700_system_off needs to be implemented
PANIC at PC : 0x0000000004023848

 

Is that the board hardware issue or the lack of kernel function implementation?

 

Espressobin does not have power switch like ATX PC boards, so software cannot power off Espressobin HW. For this reason that function a3700_system_off (in TF-A) is not implemented.

 

Kernel after stopping all servides, sent instruction to firmware (TF-A) to power down board and kernel itself stopped exection. Firmware (TF-A) received command and because it cannot do anything it just printed that error message and went to infinite loop (hang).

 

This state is like in old compuers... when they printed message you can now halt...

 

It is possible to implement a3700_system_off function in TF-A. But question is, what it should do?

Link to comment
Share on other sites

17 hours ago, Pali said:

If you have only one MAC address then set only ethaddr env variable and delete all other eth*addr variables (delete = set to nothing, e.g. setenv set1addr).

 

I've removed 3 records from u-boot environments  :

eth1addr=00:51:82:11:22:01
eth2addr=00:51:82:11:22:02
eth3addr=00:51:82:11:22:03

 

=> setenv eth1addr
=> setenv eth2addr
=> setenv eth3addr

=> printenv eth1addr
## Error: "eth1addr" not defined
=> printenv eth2addr
## Error: "eth2addr" not defined

 

I've only one left now

ethaddr=F0:AD:4E:XX:XX:XX

 

After saving-rebooting, this MAC is reproduced into the LAN port, which is actually connected to a cable :

eth0: f0:ad:4e:xx:xx:xx
lan0@eth0: fa:ad:4e:xx:xx:xx

 

on the other hand 2 other ports, which are not used for the moment, are left with some weird MACs:

lan1@eth0: 00:50:43:xx:xx:xx    <<-- looked up as Marvell Semiconductor
br0: 82:e3:7c:xx:xx:xx    <<--  Record Not Found in lookup.
wan@eth0: 06:01:c0:xx:xx:xx <<-- Record Not Found.

 

I wonder if my wan@eth0 port as well as  bridged port br0 (combined of lan0 and lan1) will be assigned fixed MACs, so that external DHCP servers, which lease IP's to them (WAN side and LAN side), assign fixed address to those interfaces, based on defined DHCPD rules. 

 

35 minutes ago, Pali said:

Now I spotted that you forgot to remove rest of mtdparts ,-(reserved).

I've modified bootargs as follows, leaving ${extraargs} :

setenv bootargs "$console root=${rootdev} rootfstype=${rootfstype} rootwait usb-storage.quirks=${usbstoragequirks}, ${extraargs}"

 

This is the 2nd new: 

root@tiger:~# cat /proc/cmdline
console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 root=UUID=d4014581-f211-49a1-aa70-e3012adab7af rootfstype=ext4 rootwait usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u, 

 

-(reserved) has been removed. Anything else to fix?

Link to comment
Share on other sites

I hope that after calling setenv you can also called saveenv so it was deleted also from env storage...

 

4 minutes ago, y52 said:

After saving-rebooting, this MAC is reproduced into the LAN port, which is actually connected to a cable :

eth0: f0:ad:4e:xx:xx:xx
lan0@eth0: fa:ad:4e:xx:xx:xx

 

on the other hand 2 other ports, which are not used for the moment are left with some weird MACs:

lan1@eth0: 00:50:43:xx:xx:xx    <<-- looked up as Marvell Semiconductor
br0: 82:e3:7c:xx:xx:xx    <<--  Record Not Found in lookup.
wan@eth0: 06:01:c0:xx:xx:xx <<-- Record Not Found.

 

... because this is not correct. You should have that your one MAC address on all lan0, lan1 and wan interfaces.

 

Could you check if boot script does not overwrite these mac addresses? Also could you check if you system does not have in some (probably autogenerated?) file stored those different MAC addresses?

Link to comment
Share on other sites

30 minutes ago, Pali said:

hope that after calling setenv you can also called saveenv so it was deleted also from env storage...

Certainly I did.

 

31 minutes ago, Pali said:

... because this is not correct. You should have that your one MAC address on all lan0, lan1 and wan interfaces.

 

Could you check if boot script does not overwrite these mac addresses? Also could you check if you system does not have in some (probably autogenerated?) file stored those different MAC addresses?

Armbian boot.cmd calls

load ${boot_interface} ${devnum}:1 ${scriptaddr} ${prefix}armbianEnv.txt

 

It appears, that it overrides u-boot MAC assignments. I've commented them out now :

root@tiger:/boot# vi /boot/armbianEnv.txt
verbosity=1
emmc_fix=off
spi_workaround=off
#eth1addr=06:01:C0:73:C5:71
#eth2addr=fa:ad:4e:84:25:2f
#eth3addr=00:50:43:0d:19:18

 

After rebooting, all 3 ports (eth0, wan@eth0, lan0@eth0 and lan1@eth0) are assigned the same MAC with Address PrefixF0:AD:4E (Globalscale Technologies), but not br0, which got the weird MAC br0: 82:e3:7c:xx:xx:xx. Anyway, it is the same as before latest adjustments.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines