Jump to content

Espressobin support development efforts

lanefu

By lanefu
in Marvell mvebu

 Share

Recommended Posts

Quintus23M

Quintus23M

Posted
Posted

I'm trying to use the latest nightly Armbian builds with my EspressoBin board (2GByte version).
- flashed this u-boot `flash-image-2g-1000_800_boot_sd_and_usb.bin`
- flashed SD card with this image `Armbian_5.34.171106_Espressobin_Ubuntu_xenial_default_4.4.96.img`

Armbian is booting, but after a minute or two it crashes. Same effect with older images, too.
Any help or ideas?

 

Ubuntu 16.04.3 LTS espressobin ttyMV0

espressobin login: [   64.314460] Unhandled fault: synchronous parity error (0x96000018) at 0xffffffc0009ff6d8
[   64.322836] Internal error: : 96000018 [#1] PREEMPT SMP
[   64.327965] Modules linked in: bridge stp llc zram zsmalloc lz4_compress mwifiex_pcie mwifiex cfg80211 rfkill
[   64.338316] CPU: 0 PID: 30 Comm: kworker/0:1 Not tainted 4.4.96-mvebu64 #14
[   64.345678] Hardware name: Marvell Armada 3720 Community Board (DT)
[   64.352159] task: ffffffc078507300 ti: ffffffc0786a4000 task.ti: ffffffc0786a4000
[   64.359361] PC is at __schedule+0x544/0x5d8
[   64.363847] LR is at __schedule+0xe4/0x5d8
[   64.368068] pc : [<ffffffc0009e3904>] lr : [<ffffffc0009e34a4>] pstate: 000001c5
[   64.375614] sp : ffffffc0786a7d50
[   64.379386] x29: ffffffc0786a7d50 x28: 0000000000000000
[   64.384783] x27: ffffffc000db2000 x26: 0000000000000000
[   64.390268] x25: 0000000000000000 x24: ffffffc078507770
[   64.395754] x23: ffffffc0009e39dc x22: ffffffc078507300
[   64.401240] x21: ffffffc000db2000 x20: 0000000000000000
[   64.406725] x19: ffffffc07efc2f00 x18: 0000000000000a03
[   64.412211] x17: 0000000000416140 x16: 0000000000000000
[   64.417696] x15: 0000000000000000 x14: ffffffc078507360
[   64.423182] x13: 0000000000000000 x12: 0000000000000000
[   64.428667] x11: 00000000afb50401 x10: afb504000afb5041
[   64.434152] x9 : 0000000000000000 x8 : 0000000000000400
[   64.439639] x7 : 0000000ef9708520 x6 : 000000001454999d
[   64.445124] x5 : 00ffffffffffffff x4 : 0000000000000015
[   64.450610] x3 : 0000000000000000 x2 : ffffffc0009ff000
[   64.456096] x1 : ffffffc078507300 x0 : ffffffc07efc2f00
[   64.461581]
[   64.463111] Process kworker/0:1 (pid: 30, stack limit = 0xffffffc0786a4020)
[   64.469942] Stack: (0xffffffc0786a7d50 to 0xffffffc0786a8000)
[   64.476319] 7d40:                                   ffffffc0786a7da0 ffffffc0009e39dc
[   64.484227] 7d60: ffffffc0786a4000 ffffffc07efc2700 ffffffc078689930 ffffffc07efc2700
[   64.492313] 7d80: ffffffc07efc2718 ffffffc0786a4000 ffffffc000dc51a0 ffffffc07efc2770
[   64.500489] 7da0: ffffffc0786a7dc0 ffffffc0000cd504 ffffffc078689900 ffffffc0000cd500
[   64.508576] 7dc0: ffffffc0786a7e20 ffffffc0000d3148 ffffffc078688c80 ffffffc000e44058
[   64.516572] 7de0: ffffffc000c29b40 ffffffc078689900 ffffffc0000cd358 0000000000000000
[   64.524568] 7e00: 0000000000000000 0000000000000000 0000000000000000 ffffffc078689900
[   64.532834] 7e20: 0000000000000000 ffffffc000085dd0 ffffffc0000d3050 ffffffc078688c80
[   64.541009] 7e40: 0000000000000000 0000000000000000 0000000000000000 ffffffc0000dc300
[   64.548917] 7e60: ffffffc0000d3050 0000000000000000 0000000000000000 ffffffc078689900
[   64.557003] 7e80: 0000000000000000 0000000000000000 ffffffc0786a7e90 ffffffc0786a7e90
[   64.565179] 7ea0: 0000000000000000 ffffffc000000000 ffffffc0786a7eb0 ffffffc0786a7eb0
[   64.573175] 7ec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.581261] 7ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.589436] 7f00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.597612] 7f20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.605520] 7f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.613695] 7f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.621692] 7f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.629778] 7fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.637954] 7fc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000
[   64.646041] 7fe0: 0000000000000000 0000000000000000 5555555555555555 5555555555555555
[   64.654124] Call trace:
[   64.656287] [<ffffffc0009e3904>] __schedule+0x544/0x5d8
[   64.662035] [<ffffffc0009e39dc>] schedule+0x44/0xb8
[   64.666712] [<ffffffc0000cd504>] worker_thread+0x1ac/0x4d0
[   64.672549] [<ffffffc0000d3148>] kthread+0xf8/0x110
[   64.677581] [<ffffffc000085dd0>] ret_from_fork+0x10/0x40
[   64.683150] Code: 17fffece 900000e2 aa1603e1 aa1303e0 (f9436c42)
[   64.463111] Process kworker/0:1 (pid: 30, stack limit = 0xffffffc0786a4020)
[   64.469942] Stack: (0xffffffc0786a7d50 to 0xffffffc0786a8000)
[   64.476319] 7d40:                                   ffffffc0786a7da0 ffffffc0009e39dc
[   64.484227] 7d60: ffffffc0786a4000 ffffffc07efc2700 ffffffc078689930 ffffffc07efc2700
[   64.492313] 7d80: ffffffc07efc2718 ffffffc0786a4000 ffffffc000dc51a0 ffffffc07efc2770
[   64.500489] 7da0: ffffffc0786a7dc0 ffffffc0000cd504 ffffffc078689900 ffffffc0000cd500
[   64.508576] 7dc0: ffffffc0786a7e20 ffffffc0000d3148 ffffffc078688c80 ffffffc000e44058
[   64.516572] 7de0: ffffffc000c29b40 ffffffc078689900 ffffffc0000cd358 0000000000000000
[   64.524568] 7e00: 0000000000000000 0000000000000000 0000000000000000 ffffffc078689900
[   64.532834] 7e20: 0000000000000000 ffffffc000085dd0 ffffffc0000d3050 ffffffc078688c80
[   64.541009] 7e40: 0000000000000000 0000000000000000 0000000000000000 ffffffc0000dc300
[   64.548917] 7e60: ffffffc0000d3050 0000000000000000 0000000000000000 ffffffc078689900
[   64.557003] 7e80: 0000000000000000 0000000000000000 ffffffc0786a7e90 ffffffc0786a7e90
[   64.565179] 7ea0: 0000000000000000 ffffffc000000000 ffffffc0786a7eb0 ffffffc0786a7eb0
[   64.573175] 7ec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.581261] 7ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.589436] 7f00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.597612] 7f20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.605520] 7f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.613695] 7f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.621692] 7f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.629778] 7fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.637954] 7fc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000
[   64.646041] 7fe0: 0000000000000000 0000000000000000 5555555555555555 5555555555555555
[   64.654124] Call trace:
[   64.656287] [<ffffffc0009e3904>] __schedule+0x544/0x5d8
[   64.662035] [<ffffffc0009e39dc>] schedule+0x44/0xb8
[   64.666712] [<ffffffc0000cd504>] worker_thread+0x1ac/0x4d0
[   64.672549] [<ffffffc0000d3148>] kthread+0xf8/0x110
[   64.677581] [<ffffffc000085dd0>] ret_from_fork+0x10/0x40
[   64.683150] Code: 17fffece 900000e2 aa1603e1 aa1303e0 (f9436c42)
[   64.375614] sp : ffffffc0786a7d50
[   64.379386] x29: ffffffc0786a7d50 x28: 0000000000000000
[   64.384783] x27: ffffffc000db2000 x26: 0000000000000000
[   64.390268] x25: 0000000000000000 x24: ffffffc078507770
[   64.395754] x23: ffffffc0009e39dc x22: ffffffc078507300
[   64.401240] x21: ffffffc000db2000 x20: 0000000000000000
[   64.406725] x19: ffffffc07efc2f00 x18: 0000000000000a03
[   64.412211] x17: 0000000000416140 x16: 0000000000000000
[   64.417696] x15: 0000000000000000 x14: ffffffc078507360
[   64.423182] x13: 0000000000000000 x12: 0000000000000000
[   64.428667] x11: 00000000afb50401 x10: afb504000afb5041
[   64.434152] x9 : 0000000000000000 x8 : 0000000000000400
[   64.439639] x7 : 0000000ef9708520 x6 : 000000001454999d
[   64.445124] x5 : 00ffffffffffffff x4 : 0000000000000015
[   64.450610] x3 : 0000000000000000 x2 : ffffffc0009ff000
[   64.456096] x1 : ffffffc078507300 x0 : ffffffc07efc2f00
[   64.461581]
[   64.463111] Process kworker/0:1 (pid: 30, stack limit = 0xffffffc0786a4020)
[   64.469942] Stack: (0xffffffc0786a7d50 to 0xffffffc0786a8000)
[   64.476319] 7d40:                                   ffffffc0786a7da0 ffffffc0009e39dc
[   64.484227] 7d60: ffffffc0786a4000 ffffffc07efc2700 ffffffc078689930 ffffffc07efc2700
[   64.492313] 7d80: ffffffc07efc2718 ffffffc0786a4000 ffffffc000dc51a0 ffffffc07efc2770
[   64.500489] 7da0: ffffffc0786a7dc0 ffffffc0000cd504 ffffffc078689900 ffffffc0000cd500
[   64.508576] 7dc0: ffffffc0786a7e20 ffffffc0000d3148 ffffffc078688c80 ffffffc000e44058
[   64.516572] 7de0: ffffffc000c29b40 ffffffc078689900 ffffffc0000cd358 0000000000000000
[   64.524568] 7e00: 0000000000000000 0000000000000000 0000000000000000 ffffffc078689900
[   64.532834] 7e20: 0000000000000000 ffffffc000085dd0 ffffffc0000d3050 ffffffc078688c80
[   64.541009] 7e40: 0000000000000000 0000000000000000 0000000000000000 ffffffc0000dc300
[   64.548917] 7e60: ffffffc0000d3050 0000000000000000 0000000000000000 ffffffc078689900
[   64.557003] 7e80: 0000000000000000 0000000000000000 ffffffc0786a7e90 ffffffc0786a7e90
[   64.565179] 7ea0: 0000000000000000 ffffffc000000000 ffffffc0786a7eb0 ffffffc0786a7eb0
[   64.573175] 7ec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.581261] 7ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.589436] 7f00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.597612] 7f20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.605520] 7f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.613695] 7f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.621692] 7f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.629778] 7fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.637954] 7fc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000
[   64.646041] 7fe0: 0000000000000000 0000000000000000 5555555555555555 5555555555555555
[   64.654124] Call trace:
[   64.656287] [<ffffffc0009e3904>] __schedule+0x544/0x5d8
[   64.662035] [<ffffffc0009e39dc>] schedule+0x44/0xb8
[   64.666712] [<ffffffc0000cd504>] worker_thread+0x1ac/0x4d0
[   64.672549] [<ffffffc0000d3148>] kthread+0xf8/0x110
[   64.677581] [<ffffffc000085dd0>] ret_from_fork+0x10/0x40
[   64.683150] Code: 17fffece 900000e2 aa1603e1 aa1303e0 (f9436c42)
[   64.368068] pc : [<ffffffc0009e3904>] lr : [<ffffffc0009e34a4>] pstate: 000001c5
[   64.375614] sp : ffffffc0786a7d50
[   64.379386] x29: ffffffc0786a7d50 x28: 0000000000000000
[   64.384783] x27: ffffffc000db2000 x26: 0000000000000000
[   64.390268] x25: 0000000000000000 x24: ffffffc078507770
[   64.395754] x23: ffffffc0009e39dc x22: ffffffc078507300
[   64.401240] x21: ffffffc000db2000 x20: 0000000000000000
[   64.406725] x19: ffffffc07efc2f00 x18: 0000000000000a03
[   64.412211] x17: 0000000000416140 x16: 0000000000000000
[   64.417696] x15: 0000000000000000 x14: ffffffc078507360
[   64.423182] x13: 0000000000000000 x12: 0000000000000000
[   64.428667] x11: 00000000afb50401 x10: afb504000afb5041
[   64.434152] x9 : 0000000000000000 x8 : 0000000000000400
[   64.439639] x7 : 0000000ef9708520 x6 : 000000001454999d
[   64.445124] x5 : 00ffffffffffffff x4 : 0000000000000015
[   64.450610] x3 : 0000000000000000 x2 : ffffffc0009ff000
[   64.456096] x1 : ffffffc078507300 x0 : ffffffc07efc2f00
[   64.461581]
[   64.463111] Process kworker/0:1 (pid: 30, stack limit = 0xffffffc0786a4020)
[   64.469942] Stack: (0xffffffc0786a7d50 to 0xffffffc0786a8000)
[   64.476319] 7d40:                                   ffffffc0786a7da0 ffffffc0009e39dc
[   64.484227] 7d60: ffffffc0786a4000 ffffffc07efc2700 ffffffc078689930 ffffffc07efc2700
[   64.492313] 7d80: ffffffc07efc2718 ffffffc0786a4000 ffffffc000dc51a0 ffffffc07efc2770
[   64.500489] 7da0: ffffffc0786a7dc0 ffffffc0000cd504 ffffffc078689900 ffffffc0000cd500
[   64.508576] 7dc0: ffffffc0786a7e20 ffffffc0000d3148 ffffffc078688c80 ffffffc000e44058
[   64.516572] 7de0: ffffffc000c29b40 ffffffc078689900 ffffffc0000cd358 0000000000000000
[   64.524568] 7e00: 0000000000000000 0000000000000000 0000000000000000 ffffffc078689900
[   64.532834] 7e20: 0000000000000000 ffffffc000085dd0 ffffffc0000d3050 ffffffc078688c80
[   64.541009] 7e40: 0000000000000000 0000000000000000 0000000000000000 ffffffc0000dc300
[   64.548917] 7e60: ffffffc0000d3050 0000000000000000 0000000000000000 ffffffc078689900
[   64.557003] 7e80: 0000000000000000 0000000000000000 ffffffc0786a7e90 ffffffc0786a7e90
[   64.565179] 7ea0: 0000000000000000 ffffffc000000000 ffffffc0786a7eb0 ffffffc0786a7eb0
[   64.573175] 7ec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.581261] 7ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.589436] 7f00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.597612] 7f20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.605520] 7f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.613695] 7f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.621692] 7f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.629778] 7fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[   64.637954] 7fc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000
[   64.646041] 7fe0: 0000000000000000 0000000000000000 5555555555555555 5555555555555555
[   64.654124] Call trace:
[   64.656287] [<ffffffc0009e3904>] __schedule+0x544/0x5d8
[   64.662035] [<ffffffc0009e39dc>] schedule+0x44/0xb8
[   64.666712] [<ffffffc0000cd504>] worker_thread+0x1ac/0x4d0
[   64.672549] [<ffffffc0000d3148>] kthread+0xf8/0x110
[   64.677581] [<ffffffc000085dd0>] ret_from_fork+0x10/0x40
[   64.683150] Code: 17fffece 900000e2 aa1603e1 aa1303e0 (f9436c42)

0
ebin-dev

ebin-dev

Posted
Posted
2 hours ago, Quintus23M said:

Armbian is booting, but after a minute or two it crashes. Same effect with older images, too.
Any help or ideas?

 

You have a hardware issue. Contact Globalscale Technologies and RMA the device.

As a workaround try 2G_800_800 or 2G_1200_750 and adapt the values in /etc/default/cpufrequtils according to available values displayed by 'cpufreq-info'. 

You may also try to switch cpufreq governor from ondemand to powersave.

0
DS Justice

DS Justice

Posted
Posted
On 11/5/2017 at 3:32 AM, Quintus23M said:

Armbian is booting, but after a minute or two it crashes. Same effect with older images, too.
Any help or ideas?

 

I had similar repeated crashes after switching to Armbian (171106) from the Espressobin Ubuntu.  It seemed to stabilize after I added `mw.l 0xd0011500 0x78e3ffff;` into my bootcmd.  Full environment linked here.

 

I really don't know what that *does*, I found it on a forum, someone suggested that it would disable the frequency governer, which they asserted was the source of *their* crashes.

 

Good luck.

0
arm-push

arm-push

Posted
Posted

Is anyone doing I/O related testing?  Just wanted to share my experience. I was trying to find out the max that the board could write, I can get around 80 MiB/s write speed for a bit old Samsung hd103SJ HDD.

Here are my configs,

 

CPU
++++++
cpufreq-set -c 1 -d 1200000 -u 1200000 -g performance
cpufreq-set -c 0 -d 1200000 -u 1200000 -g performance


Network
+++++++
ethtool -C eth0 rx-frames 1
ethtool -C eth0 rx-usecs 1

 

Disk (ext4)
++++++
tune2fs -O ^has_journal /dev/sda1
/dev/sda1 /mnt ext4 defaults,noatime,nodiratime,noacl 0 0

 

sysctl vm.dirty_expire_centisecs=10
sysctl vm.dirty_writeback_centisecs=100

 

Application (ftp upload)
++++++++++++
mirror -R -P 5 ./dirbla
3221225472 bytes transferred in 36 seconds (84.72 MiB/s)

mirror -R -P 1 ./dirbla
3221225472 bytes transferred in 37 seconds (83.75 MiB/s)

0
tkaiser

tkaiser

Posted
Posted
14 minutes ago, arm-push said:

Is anyone doing I/O related testing?

 

https://forum.armbian.com/topic/4845-marvell-based-4-ports-mpci-sata-30/?do=findComment&comment=37704

 

Please keep in mind that we've tested there 'advanced' stuff with mPCIe SATA controllers. On the onboard SATA port EspressoBin should in a single disk configuration easily exceed 500 MB/s (or in other words: Fast enough for any HDD imaginable)

 

0
arm-push

arm-push

Posted
Posted
32 minutes ago, tkaiser said:

 

https://forum.armbian.com/topic/4845-marvell-based-4-ports-mpci-sata-30/?do=findComment&comment=37704

 

Please keep in mind that we've tested there 'advanced' stuff with mPCIe SATA controllers. On the onboard SATA port EspressoBin should in a single disk configuration easily exceed 500 MB/s (or in other words: Fast enough for any HDD imaginable)

 

 

Thanks @tkaiser. I'll follow that thread as well. My intention was to find out whether the board could saturate 1Gb bandwidth during a file copy (a simple NAS use case, copying a file from my Linux Desktop to EB).

The max which I could get so far was around 80 MiB/s. I'm just wondering if someone had done that kind of a test with EB.

I hope you can give us some help/direction on that. :)

0
tkaiser

tkaiser

Posted
Posted
8 minutes ago, arm-push said:

The max which I could get so far was around 80 MiB/s. I'm just wondering if someone had done that kind of a test with EB.

 

I've my Espressobin somewhere and didn't find it since weeks so I can't answer. I remember network throughput being not that great but I tested only with pretty bad settings (800 MHz, low DRAM clockspeed). I would suggest testing in parallel with iperf3 and if there you get to the maximum (~940 Mbits/sec) then file transfers might be bottlenecked maybe by a smbd process (or NFS, or whatever) maxing out a single CPU core.

 

Different platform but also applying here: http://linux-sunxi.org/Sunxi_devices_as_NAS#Benchmarking_.2F_Identifying_bottlenecks

0
chrisf

chrisf

Posted
Posted

Can the sysfs GPIO driver be enable in the nightly kernel image? I don't see a /sys/class/gpio

Apparently there is an LED connected to gpio MPPI_2

 

image.png.9a8719c336d14c0ede5aa27ad752ee13.png

Maybe it would be better to define an LED for it in the DTS?

 

I don't see where /sys/class/leds/mmc0:: is defined, but it doesn't appear to do anything when the brightness or trigger is changed.

0
ebin-dev

ebin-dev

Posted
Posted
18 hours ago, arm-push said:

The max which I could get so far was around 80 MiB/s. I'm just wondering if someone had done that kind of a test with EB.

 

There were plenty of networking tests earlier in this thread (in particular on June 22nd). Helios Lan Tests were carried out without any optimizations and resulted in about 75MB/s (upload to EB) and about 97 MB/s (download from EB). Single large files could be downloaded with 113MB/s.

 

So your result of 80MB/s (upload to EB) is already pretty good.

 

It seems that the processor is pretty busy during upload and some further networking offload would be helpful. You can try to play with ethtool ( ethtool --offload  eth0  rx on tx on sg on tso on gso on gro on  ).

 

20 hours ago, DS Justice said:

I really don't know what that *does*, I found it on a forum, someone suggested that it would disable the frequency governer, which they asserted was the source of *their* crashes.

 

The kernel panics observed by some of you can be circumvented by switching cpufreq governor from ondemand to powersave. This does not mean that there is a software issue, because these kernel panics can also be avoided by flashing SPI with a lower CPU frequency while leaving ondemand enabled.

I have seen that myself while testing a board with hardware issues.

0
Patrick

Patrick

Posted
Posted
9 hours ago, ebin-dev said:

 

There were plenty of networking tests earlier in this thread (in particular on June 22nd). Helios Lan Tests were carried out without any optimizations and resulted in about 75MB/s (upload to EB) and about 97 MB/s (download from EB). Single large files could be downloaded with 113MB/s.

 

So your result of 80MB/s (upload to EB) is already pretty good.

 

It seems that the processor is pretty busy during upload and some further networking offload would be helpful. You can try to play with ethtool ( ethtool --offload  eth0  rx on tx on sg on tso on gso on gro on  ).

 

 

The kernel panics observed by some of you can be circumvented by switching cpufreq governor from ondemand to powersave. This does not mean that there is a software issue, because these kernel panics can also be avoided by flashing SPI with a lower CPU frequency while leaving ondemand enabled.

I have seen that myself while testing a board with hardware issues.

 

How do you get 80MB/s up-/download?

I only get 15MB/s max :(

(1200_750 Armbian_5.34.171102_Espressobin_Debian_stretch_default_4.4.95 and OMV4 (Netatalk installed)

 

kr.,

Patrick

0
Patrick

Patrick

Posted
Posted (edited)
20 hours ago, ebin-dev said:

Did you try it with 1000_800 and adapted /etc/default/cpufrequtils ? 

Tried it today, loaded the 1000_800.

cpufrequtils: 

ENABLE=true
MIN_SPEED=250000
MAX_SPEED=1000000
GOVERNOR=ondemand

SATA attached "old" Intel SSD

CapturFiles_8.png

Edited by Patrick
New info
0
arm-push

arm-push

Posted
Posted
20 hours ago, Patrick said:

Tried it today, loaded the 1000_800.

cpufrequtils: 

ENABLE=true
MIN_SPEED=250000
MAX_SPEED=1000000
GOVERNOR=ondemand

SATA attached "old" Intel SSD

 

 

 

Try changing the cpu governor and Ethernet coalesce settings,

 

cpufreq-set -c 1 -d 1200000 -u 1200000 -g performance
cpufreq-set -c 0 -d 1200000 -u 1200000 -g performance


ethtool -C eth0 rx-frames 1
ethtool -C eth0 rx-usecs 1

0
Patrick

Patrick

Posted
Posted
1 hour ago, arm-push said:

 

Try changing the cpu governor and Ethernet coalesce settings,

 

cpufreq-set -c 1 -d 1200000 -u 1200000 -g performance
cpufreq-set -c 0 -d 1200000 -u 1200000 -g performance


ethtool -C eth0 rx-frames 1
ethtool -C eth0 rx-usecs 1

I've tried it (I only could use 1000000 instead of 1200000) but the up-/downloadspeed got worse.

And now, for the first time, I've got kernel errors:

 

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1148.988014] Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1149.132228] Process kworker/1:0 (pid: 1970, stack limit = 0xffffffc03785c020)

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1149.139169] Stack: (0xffffffc03785fb60 to 0xffffffc037860000)

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1149.145112] fb60: ffffffc03785fbf0 ffffffc0009e4580 0000000000000026 0000000000000000

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1149.153482] fb80: 0000000000000001 0000000000000000 ffffffc03785fcb0 ffffffc0000e95fc

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1149.161675] fba0: 0000000100033db1 0000000000000001 ffffffc03785fbc0 ffffffc0008669cc

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1149.169868] fbc0: ffffffc03785fbd0 ffffffc000111374 ffffffc03785fc00 ffffffc00011a3ac

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1149.177978] fbe0: ffffffc03785fbf0 ffffffc0009e4578 ffffffc03785fc70 ffffffc0009e463c

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1149.185997] fc00: ffffffc039122e00 0000000000000009 ffffffc039122e20 ffffffc039122e70

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1149.194098] fc20: ffffffc03785fcf8 ffffffc0230e9630 ffffffc000dc51a0 0000000000000009

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1149.202290] fc40: ffffffc039122e20 0000000000000001 ffffffc0381af300 ffffffc0000db690

Message from syslogd@localhost at Nov 10 20:08:23 ...
 kernel:[ 1149.210312] fc60: ffffffc03785fd20 ffffffc03785fd20 ffffffc03785fc80 ffffffc0000cb03c
packet_write_wait: Connection to 10.54.1.5 port 22: Broken pipe
 

kr.,

Patrick

0
arm-push

arm-push

Posted
Posted
16 hours ago, Patrick said:

And now, for the first time, I've got kernel errors:

 

I'm using 1200_750 and I've never had any issues. I think it's the stable one.

With the settings I mentioned above, I can get around 78 to 80 MiB/s write rate over 1G Ethernet network.

As @tkaiser mentioned, I have done quick tests and it seems that the CPU is not powerful enough to exceed about 80 MiB/s. CPU utilization reaches 100%, mainly the application ( vsftpd in my case ) and kernel threads related to virtual memory contribute to that.

 

1
ebin-dev

ebin-dev

Posted
Posted (edited)
On 11/11/2017 at 1:39 PM, arm-push said:

I have done quick tests and it seems that the CPU is not powerful enough to exceed about 80 MiB/s.

 

Did you try to prioritize vsftpd:  'ionice -c1 -p $(ps -C vsftpd -o pid= )'  ? You may squeeze out some more MB/s with this.

 

EspressoBin has an envelope of about 5W and achieves remarkable results with it - in particular if you compare it with odroid HC1.

 

Edited by ebin-dev
priotitize vsftpd corrected
0
arm-push

arm-push

Posted
Posted
4 minutes ago, ebin-dev said:

 

Did you try to prioritize vsftpd:  'taskset -p 01 $(ps -C vsftpd -o pid= )'  ? You may squeeze out some more MB/s with this.

 

EspressoBin has an envelope of about 5W and achieves remarkable results with it - in particular if you compare it with odroid HC1.

 

your command sets the affinity of the vsftpd to 0 and 1. isn't it? Since the process is already running on core 0 and 1, I don't think it will make any difference.

I think you meant chrt or renice commands.

 

I have already tried setting affinity of vsftpd to only one core, but got poor performance. That means vsftpd needs both cores.

I haven't tried chrt and or renice. I will post results later.

 

Also nic offloading did not make any difference as well.

 

0
Patrick

Patrick

Posted
Posted
On 10-11-2017 at 9:30 PM, tkaiser said:

 

Please test both network and storage individually (iperf3 and iozone)

Those things are new for me so it took a while before I get results.

 

iperf3 gave me the following results:

10.54.1.5 = Espressobin

 

Connecting to host 10.54.1.98, port 5201
[  4] local 10.54.1.5 port 43888 connected to 10.54.1.98 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.01   sec  96.5 MBytes   804 Mbits/sec    0    679 KBytes       
[  4]   1.01-2.00   sec  96.2 MBytes   811 Mbits/sec    0    679 KBytes       
[  4]   2.00-3.00   sec  96.2 MBytes   807 Mbits/sec    0    701 KBytes       
[  4]   3.00-4.00   sec   100 MBytes   837 Mbits/sec    0    962 KBytes       
[  4]   4.00-5.01   sec  96.2 MBytes   807 Mbits/sec    0    962 KBytes       
[  4]   5.01-6.00   sec  96.2 MBytes   811 Mbits/sec    0    962 KBytes       
[  4]   6.00-7.01   sec   101 MBytes   843 Mbits/sec    0   1007 KBytes       
[  4]   7.01-8.01   sec  96.2 MBytes   809 Mbits/sec    0   1007 KBytes       
[  4]   8.01-9.00   sec  96.2 MBytes   811 Mbits/sec    0   1007 KBytes       
[  4]   9.00-10.01  sec  97.5 MBytes   812 Mbits/sec    0   1.10 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.01  sec   973 MBytes   815 Mbits/sec    0             sender
[  4]   0.00-10.01  sec   973 MBytes   815 Mbits/sec                  receiver


iperf Done.

 

 

Connecting to host 10.54.1.5, port 5201
[  5] local 10.54.1.98 port 54387 connected to 10.54.1.5 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  59.3 MBytes   498 Mbits/sec                  
[  5]   1.00-2.00   sec  72.8 MBytes   609 Mbits/sec                  
[  5]   2.00-3.00   sec  53.0 MBytes   445 Mbits/sec                  
[  5]   3.00-4.00   sec  66.7 MBytes   561 Mbits/sec                  
[  5]   4.00-5.00   sec  72.5 MBytes   608 Mbits/sec                  
[  5]   5.00-6.00   sec  64.5 MBytes   541 Mbits/sec                  
[  5]   6.00-7.00   sec  64.1 MBytes   537 Mbits/sec                  
[  5]   7.00-8.00   sec  65.2 MBytes   548 Mbits/sec                  
[  5]   8.00-9.00   sec  60.5 MBytes   508 Mbits/sec                  
[  5]   9.00-10.00  sec  66.0 MBytes   553 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   645 MBytes   541 Mbits/sec                  sender
[  5]   0.00-10.00  sec   644 MBytes   541 Mbits/sec                  receiver

 

iperf Done.

 

 

 

I'm still looking how to run iozone.

 

Sorry guys,

 

I did all previous tests with a wireless connected Macbook, now I use a wired connection it's a lot better (except the write speed)

 

CapturFiles_11.png

 

kr.,

Patrick

0
chrisf

chrisf

Posted
Posted

Something is wrong with the ordering of things while shutting down

 

The network interfaces are being shut down very early on, before remote file systems are stopped, which results in a timeout while NFS tries to cleanly unmount.

 

You can see just after nginx is stopped, the wan interface is disabled.

The [FAILED] message takes about 30 seconds or so to come up. Even after systemd has reached the shutdown state, there's still kernel messages about the NFS unmount failure. 

Something is telling the switch driver to shut down before the network is supposed to be shut down - those "[83850.808711] device wan left promiscuous mode" message look like debug message from the driver.

 

Quote

_____                                   _     _       
| ____|___ _ __  _ __ ___  ___ ___  ___ | |__ (_)_ __  
|  _| / __| '_ \| '__/ _ \/ __/ __|/ _ \| '_ \| | '_ \ 
| |___\__ \ |_) | | |  __/\__ \__ \ (_) | |_) | | | | |
|_____|___/ .__/|_|  \___||___/___/\___/|_.__/|_|_| |_|
          |_|                                          

Welcome to ARMBIAN 5.34.171108 nightly Debian GNU/Linux 9 (stretch) 4.4.96-mvebu64   
System load:   0.00 0.00 0.00   Up time:       23:17 hours
Memory usage:  6 % of 926MB     IP:            
Usage of /:    5% of 29G    

[ General system configuration: armbian-config ]


root@ebin:~# poweroff
[  OK  ] Stopped target RPC Port Mapper.
         Stopping User Manager for UID 0...
         Stopping NFS status monitor for NFSv2/3 l         Stopping Session 216 of user root.
[  OK  ] Stopped Clean PHP session files every 30 mins.
[  OK  ] Stopped target System Time Synchronized.
         Unmounting RPC Pipe File System...
[  OK  ] Stopped target Graphical Interface.
[  OK  ] Stopped target Multi-User System.
         Stopping LSB: Start/stop sysstat's sadc...
         Stopping System Logging Service...
         Stopping A high performance web server and a reverse proxy server...
         Stopping LSB: Armbian gathering hardware information...
         Stopping LSB: Set sysfs variables from /etc/sysfs.conf...
         Stopping LSB: disk temperature monitoring daemon...
         Stopping OpenBSD Secure Shell server...
         Stopping Unattended Upgrades Shutdown...
         Stopping Regular background program processing daemon...
[  OK  ] Stopped target Login Prompts.
         Stopping Serial Getty on ttyS0...
         Stopping Getty on tty1...
         Stopping Serial Getty on ttyMV0...
         Stopping LSB: Advanced IEEE 802.11 management daemon...
         Stopping Network Name Resolution...
         Stopping LSB: Start NTP daemon...
[  OK  ] Stopped System Logging Service.
[  OK  ] Stopped Regular background program processing daemon.
[  OK  ] Stopped Network Name Resolution.
[  OK  ] Stopped OpenBSD Secure Shell server.
[  OK  ] Stopped Authorization Manager.
[  OK  ] Stopped NFS status monitor for NFSv2/3 locking..
[  OK  ] Stopped Serial Getty on ttyMV0.
[  OK  ] Stopped Getty on tty1.
[  OK  ] Stopped User Manager for UID 0.
[  OK  ] Stopped Serial Getty on ttyS0.
[  OK  ] Stopped Session 216 of user root.
[  OK  ] Unmounted RPC Pipe File System.
[  OK  ] Stopped A high performance web server and a reverse proxy server.
[83850.808711] device wan left promiscuous mode
[83850.817680] br0: port 3(wan) entered disabled state
[  OK  ] Stopped LSB: Advanced IEEE 802.11 management[83850.831564] device lan0 left promiscuous mode
 daemon.
[83850.839749] br0: port 2(lan0) entered disabled state
[83850.881912] device lan1 left promiscuous mode
[83850.886817] br0: port 1(lan1) entered disabled state
[  OK  ] Removed slice User Slice of root.
         Stopping Login Service...
[  OK  ] Removed slice system-getty.slice.
[  OK  ] Removed slice system-serial\x2dgetty.slice.
[  OK  ] Stopped /etc/rc.local Compatibility.
         Stopping Permit User Sessions...
[  OK  ] Stopped target Host and Network Name Lookups.
[  OK  ] Stopped Login Service.
[  OK  ] Stopped LSB: Start/stop sysstat's sadc.
[  OK  ] Stopped LSB: Set sysfs variables from /etc/sysfs.conf.
[  OK  ] Stopped LSB: disk temperature monitoring daemon.
[  OK  ] Stopped Permit User Sessions.
[  OK  ] Stopped LSB: Start NTP daemon.
         Stopping LSB: set CPUFreq kernel parameters...
[  OK  ] Stopped LSB: set CPUFreq kernel parameters.
         Stopping LSB: Load kernel modules needed to enable cpufreq scaling...
[  OK  ] Stopped LSB: Armbian gathering hardware information.
[  OK  ] Stopped LSB: Load kernel modules needed to enable cpufreq scaling.
[  OK  ] Stopped target Remote File Systems.
         Unmounting /mnt/data...
[  OK  ] Stopped Unattended Upgrades Shutdown.
[FAILED] Failed unmounting /mnt/data.
[  OK  ] Stopped target Remote File Systems (Pre).
[  OK  ] Stopped target NFS client services.
[  OK  ] Stopped target Network is Online.
[  OK  ] Stopped Network Manager Wait Online.
[  OK  ] Stopped target Network.
         Stopping Network Manager...
         Stopping Raise network interfaces...
         Stopping Network Service...
[  OK  ] Stopped Network Service.
[  OK  ] Stopped Network Manager.
         Stopping D-Bus System Message Bus...
[  OK  ] Stopped D-Bus System Message Bus.
[  OK  ] Stopped target Basic System.
[  OK  ] Stopped target Slices.
[  OK  ] Removed slice User and Session Slice.
[  OK  ] Stopped target Sockets.
[  OK  ] Closed Syslog Socket.
[  OK  ] Stopped target Paths.
[  OK  ] Closed D-Bus System Message Bus Socket.
[  OK  ] Stopped target System Initialization.
         Stopping Restore / save the current clock...
         Stopping Update UTMP about System Boot/Shutdown...
         Stopping Armbian enhanced Log2Ram...
[  OK  ] Stopped target Swap.
[  OK  ] Stopped target Encrypted Volumes.
[  OK  ] Stopped Dispatch Password Requests to Console Directory Watch.
[  OK  ] Stopped Forward Password Requests to Wall Directory Watch.
         Stopping Entropy daemon using the HAVEGE algorithm...
[  OK  ] Stopped Raise network interfaces.
[  OK  ] Stopped Entropy daemon using the HAVEGE algorithm.
[  OK  ] Stopped Restore / save the current clock.
[  OK  ] Stopped Update UTMP about System Boot/Shutdown.
[  OK  ] Stopped Create Volatile Files and Directories.
         Stopping Load/Save Random Seed...
[  OK  ] Stopped Apply Kernel Variables.
[  OK  ] Stopped Load Kernel Modules.
[  OK  ] Stopped Load/Save Random Seed.
[  OK  ] Unmounted /var/log.
[  OK  ] Unmounted /var/log.hdd.
[  OK  ] Stopped Armbian enhanced Log2Ram.
[  OK  ] Stopped target Local File Systems.
         Unmounting /run/user/0...
         Unmounting /tmp...
[  OK  ] Unmounted /run/user/0.
[  OK  ] Unmounted /tmp.
[  OK  ] Reached target Unmount All Filesystems.
[  OK  ] Stopped target Local File Systems (Pre).
[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Reached target Shutdown.
[83944.353353] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[83944.376015] systemd-journald[503]: Received SIGTERM from PID 1 (systemd-shutdow).
[83944.394219] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[83944.413664] systemd-shutdown[1]: Unmounting file systems.
[83944.420285] systemd-shutdown[1]: Remounting '/mnt/data' read-only with options 'vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.6,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.0.0.6'.
[84124.436792] nfs: server 10.0.0.6 not responding, still trying
 

 

0
tkaiser

tkaiser

Posted
Posted
20 hours ago, Patrick said:

I did all previous tests with a wireless connected Macbook, now I use a wired connection it's a lot better (except the write speed)

 

I expected that already but since the numbers were a bit too good I didn't asked already. Now with wired connection write speed is still at 14.2 MB/s so I would strongly suggest to do a local storage test now (since old SSDs are often funny SSDs). Please do a cd to the SSD's mountpoint in question and then 

for i in 1 2 3 ; do iozone -e -I -a -s 300M -r 128k -i 0 -i 1 ; done
sleep 240 ; iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2

Please post the results via pastebin.com or something similar.

0
tkaiser

tkaiser

Posted
Posted
1 hour ago, bunny said:

Reading/Writing speed is around 20 MB/s via Samba Share.

No, you were bottlenecked by either using the wrong storage port combined with the wrong device (pendrive on USB2) or the wrong filesystem (NTFS). When using the SATA port, a capable HDD exceeding 150 MB/s and appropriate Samba settings the network will be the bottleneck on EspressoBin.

0
bunny

bunny

Posted
Posted

thank you tkaiser.

I will change my storage system according to your recommendation and post the results here.

In fact, I just ordered a sata cable, so I will use the SATA with my HDD drive (changed to ext4) to squeeze out the most of espressobin.

 

/bunny

0
Patrick

Patrick

Posted
Posted
7 hours ago, tkaiser said:

 

I expected that already but since the numbers were a bit too good I didn't asked already. Now with wired connection write speed is still at 14.2 MB/s so I would strongly suggest to do a local storage test now (since old SSDs are often funny SSDs). Please do a cd to the SSD's mountpoint in question and then 


for i in 1 2 3 ; do iozone -e -I -a -s 300M -r 128k -i 0 -i 1 ; done
sleep 240 ; iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2

Please post the results via pastebin.com or something similar.

Hello tkaiser,

 

Thanks for your reaction, I think it's a compliment for my wireless network.

I did the local storage test as you asked and post the results here: https://pastebin.com/qNaprd0s

 

kr.,

Patrick

 

0
tkaiser

tkaiser

Posted
Posted
39 minutes ago, Patrick said:

Thanks for your reaction, I think it's a compliment for my wireless network.

I did the local storage test as you asked and post the results here: https://pastebin.com/qNaprd0s

 

Well, if your MacBook is somewhat recent then it has at least 3 antennas and this combined with a 802.11ac MIMO capable AP with same count of antennas or more... good wireless performance (same here, Fast Ethernet almost feels broken)

 

Your SSD is not the bottleneck, the 3 first runs tried to simulate what LanTest was doing and even with 128KB blocksize write performance is at least 200 MB/s. Overall performance of this Intel thing is still pretty fast: 

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          307200     128   232575   275415   194252   188408
          307200     128   233562   275463   195358   201862
          307200     128   201101   244803   174815   178310
          102400       4    18686    30697    35260    35643    23840    32454
          102400      16    65864   103817    98304    98915    73120   103304
          102400     512   276280   344878   229520   229880   218717   344584
          102400    1024   330431   318705   227095   237980   236421   360159
          102400   16384   307209   318521   228364   228928   229110   318783

In case you didn't disable Wi-Fi completely when doing the wired tests this would be worth a try (since macOS' configd and/or the AP might have different views about which interface packets should take)

0
Patrick

Patrick

Posted
Posted
1 hour ago, tkaiser said:

 

Well, if your MacBook is somewhat recent then it has at least 3 antennas and this combined with a 802.11ac MIMO capable AP with same count of antennas or more... good wireless performance (same here, Fast Ethernet almost feels broken)

 

Your SSD is not the bottleneck, the 3 first runs tried to simulate what LanTest was doing and even with 128KB blocksize write performance is at least 200 MB/s. Overall performance of this Intel thing is still pretty fast: 


                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          307200     128   232575   275415   194252   188408
          307200     128   233562   275463   195358   201862
          307200     128   201101   244803   174815   178310
          102400       4    18686    30697    35260    35643    23840    32454
          102400      16    65864   103817    98304    98915    73120   103304
          102400     512   276280   344878   229520   229880   218717   344584
          102400    1024   330431   318705   227095   237980   236421   360159
          102400   16384   307209   318521   228364   228928   229110   318783

In case you didn't disable Wi-Fi completely when doing the wired tests this would be worth a try (since macOS' configd and/or the AP might have different views about which interface packets should take)

My MacBook Pro isn't the most recent (mid-2012 non-retina) and an Asus RT-AC3200.

I disabled Wi-Fi completely as far as I know.

 

I just loaded a fresh image and only created a Samba-share (armbian-config) but the performance got worse (approx 7MB/s).

 

kr.,

Patrick

0
tkaiser

tkaiser

Posted
Posted

Just as a short reminder (for whoever feels concerned): Current boot.cmd for EspressoBin should not be able to work with a btrfs rootfs (or any other sort of image using a separate boot partition) since there are fallbacks from 'boot/' to '/' missing everywhere. Generated two OMV images today with both kernels, searched again for my EspressoBin in the lab, thought a bit about the potential showstoppers, didn't found the board, gave up.

0
bunny

bunny

Posted
Posted
14 hours ago, Patrick said:

Hello tkaiser,

 

Thanks for your reaction, I think it's a compliment for my wireless network.

I did the local storage test as you asked and post the results here: https://pastebin.com/qNaprd0s

 

kr.,

Patrick

 

 

...and here the results for a sata attached HDD (not SSD) ext 4 formatted:

 

https://pastebin.com/HTxjsnVw

 

...and just for comparison, same HDD ntfs formatted:

 

https://pastebin.com/FiWApctm

0
chrisf

chrisf

Posted
Posted

I saw the announcement by @Igor of the RT kernel for this board, based off the 4.13.10 mainline kernel. Is there going to be a regular mainline build? @ebin-dev said there was work for the new hardware crypto engine begin done for mainline as well.

0

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept