malvcr

I have been thinking in some combination as OpenVPN (recompiled for cryptodev) with rsync or a type of simple plain data transmission. What I don't know is how will be the overhead ... if it is very high then I will need to process big chunks of data by myself. But, in general, it is a good thing to test :-)

Not so easy to do ... Latest ssh version (cloned with git from official site): checking OpenSSL library version... configure: error: OpenSSL >= 1.1.0 is not yet supported (have "10101000 (OpenSSL 1.1.1-dev xx XXX xxxx)"). And, of course, when I replace the OpenSSL many things are broken. I will need to figure a different approach. There is a Fedora patch for 1.0, but I think that to make that to work involves more work than to do something from scratch (and will be over when openssh be updated to openssl 1.1.0).

I checked interrupts, by Rider.Lee in BPI forum recommendation ... and when using the extensions there is a big quantity of them ( openssl speed -evp aes-128-cbc ). First run (with cryptodev) Total change for mtk-aes: 394682 Total change CPU : 2260 + 14616 + 14895 + 22625 = 54396 Second run (without cryptodev) Total change for mtk-aes: 0 Total change CPU : 1710 + 4332 + 4860 + 4357 = 15259 Right now I repeated the test checking the CPU. In the first case the CPU arrived at most to 50% - with cryptodev In the second case went to 100% (one core) - without cryptodev Would be interesting to go directly to the kernel interface. I don't know if openssl or even cryptodev way to do things could be wasting some CPU. My next test would be to replace the basic infrastructure to see how is the behavior with sftp. Previously I only was able to obtain 14 MB/s but this was limited by the CPU.

Detail .... that hardware engine seems to work better with more than 1024 bytes. Something to take into consideration (although I need to check openssl compiling parameters with more care).

I am writing to Gary ... but in the while, I built everything and the crypto driver really works. These are the numbers: Without the driver (standard openssl): for i in 128 192 256; do openssl speed -elapsed -evp aes-${i}-cbc ; done You have chosen to measure elapsed time instead of user CPU time. Doing aes-128-cbc for 3s on 16 size blocks: 4140500 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 64 size blocks: 1196387 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 256 size blocks: 312026 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 1024 size blocks: 78846 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 8192 size blocks: 9886 aes-128-cbc's in 3.00s OpenSSL 1.0.2g 1 Mar 2016 built on: reproducible build, date unspecified options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr) compiler: cc -I. -I.. -I../include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-128-cbc 22082.67k 25522.92k 26626.22k 26912.77k 26995.37k You have chosen to measure elapsed time instead of user CPU time. Doing aes-192-cbc for 3s on 16 size blocks: 3626398 aes-192-cbc's in 3.00s Doing aes-192-cbc for 3s on 64 size blocks: 1028081 aes-192-cbc's in 3.00s Doing aes-192-cbc for 3s on 256 size blocks: ^[[A266479 aes-192-cbc's in 3.00s Doing aes-192-cbc for 3s on 1024 size blocks: 67186 aes-192-cbc's in 3.00s Doing aes-192-cbc for 3s on 8192 size blocks: 8426 aes-192-cbc's in 3.00s OpenSSL 1.0.2g 1 Mar 2016 built on: reproducible build, date unspecified options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr) compiler: cc -I. -I.. -I../include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-192-cbc 19340.79k 21932.39k 22739.54k 22932.82k 23008.60k You have chosen to measure elapsed time instead of user CPU time. Doing aes-256-cbc for 3s on 16 size blocks: 3258679 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 64 size blocks: 910552 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 256 size blocks: 235055 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 1024 size blocks: 59249 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 8192 size blocks: 7422 aes-256-cbc's in 3.00s OpenSSL 1.0.2g 1 Mar 2016 built on: reproducible build, date unspecified options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr) compiler: cc -I. -I.. -I../include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 17379.62k 19425.11k 20058.03k 20223.66k 20267.01k With the driver ( to do this I had to compile kernel, cryptodrv and openssl ... with some quirks here and there ): for i in 128 192 256; do ./openssl speed -elapsed -evp aes-${i}-cbc ; done You have chosen to measure elapsed time instead of user CPU time. Doing aes-128-cbc for 3s on 16 size blocks: 97341 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 64 size blocks: 83631 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 256 size blocks: 74013 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 1024 size blocks: 73826 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 8192 size blocks: 45441 aes-128-cbc's in 2.99s Doing aes-128-cbc for 3s on 16384 size blocks: 31972 aes-128-cbc's in 3.00s OpenSSL 1.1.1-dev xx XXX xxxx built on: reproducible build, date unspecified options:bn(64,32) rc4(char) des(long) aes(partial) idea(int) blowfish(ptr) compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/usr/local/ssl\"" -DENGINESDIR="\"/usr/local/lib/engines-1.1\"" -march=armv7-a -Wa,--noexecstack The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-128-cbc 519.15k 1784.13k 6315.78k 25199.27k 124499.22k 174609.75k You have chosen to measure elapsed time instead of user CPU time. Doing aes-192-cbc for 3s on 16 size blocks: 96073 aes-192-cbc's in 3.00s Doing aes-192-cbc for 3s on 64 size blocks: 84094 aes-192-cbc's in 3.00s Doing aes-192-cbc for 3s on 256 size blocks: 74714 aes-192-cbc's in 3.00s Doing aes-192-cbc for 3s on 1024 size blocks: 74362 aes-192-cbc's in 3.00s Doing aes-192-cbc for 3s on 8192 size blocks: 43467 aes-192-cbc's in 3.00s Doing aes-192-cbc for 3s on 16384 size blocks: 29992 aes-192-cbc's in 3.00s OpenSSL 1.1.1-dev xx XXX xxxx built on: reproducible build, date unspecified options:bn(64,32) rc4(char) des(long) aes(partial) idea(int) blowfish(ptr) compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/usr/local/ssl\"" -DENGINESDIR="\"/usr/local/lib/engines-1.1\"" -march=armv7-a -Wa,--noexecstack The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-192-cbc 512.39k 1794.01k 6375.59k 25382.23k 118693.89k 163796.31k You have chosen to measure elapsed time instead of user CPU time. Doing aes-256-cbc for 3s on 16 size blocks: 95306 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 64 size blocks: 84143 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 256 size blocks: 74296 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 1024 size blocks: 73367 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 8192 size blocks: 41361 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 16384 size blocks: 28101 aes-256-cbc's in 3.00s OpenSSL 1.1.1-dev xx XXX xxxx built on: reproducible build, date unspecified options:bn(64,32) rc4(char) des(long) aes(partial) idea(int) blowfish(ptr) compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/usr/local/ssl\"" -DENGINESDIR="\"/usr/local/lib/engines-1.1\"" -march=armv7-a -Wa,--noexecstack The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-256-cbc 508.30k 1795.05k 6339.93k 25042.60k 112943.10k 153468.93k So ... with the R2 I "must" use the hardware help. I will check AF_ALG ... but today hours were not enough ;-)

I will give a try to what Gary Wang described here (http://forum.banana-pi.org/t/is-it-possible-to-have-the-crypto-extensions-working/4034). ... could be possible that thing is ready? In case I have good news I will share them :-)

You are right ... priorities. In fact, I would prefer to have the cpufreq type of things ready than the 'special' stuff. So ... today. 1) It is not possible to have high throughput on sequential cryptography networking involving only the CPU. 2) Plain networking (no linear CPU processing involved) is OK. 3) It is not possible to play with the CPU frequency. 4) Disk it is not tuned, and has the PCIe-line bandwidth limitation. USB3 speed it is not good at all. Then ... por specialized programming "today" 1) To increase "protected" networking speed, some sort of multithreading/multiprocessing must be used (at least there are 4 cores), or playing around with the cryptography algorithms to let the CPU to breath. 2) JIT processing it is not a good idea when dealing with lots of data. Better prepare what you can in advance when the CPU it is idle. 3) As the machine hast 2 GB RAM, it is important to use that 'asset' to reduce I/O latency. 4) Don't use USB for disks. Keep SATA and, "maybe", an extra adapter in the PCIe slot.

Yes, the problem with my networking test was a bottleneck with the CPU, not the networking equipment. Making a simply stupid client/server perl based test (only sending static memory data from one side without verifications), I was able to obtain 135 MB/s. Of course, this number the only shows is that the device it is really a gigabit one; doesn't indicate that in real life scenarios I can obtain that throughput. But if I use it, I must understand all its limitations and possibilities. Thanks for the crypto references; I still need to do many things and to organize many "possibilities", so that material it is really important for me. So if I use the R2, can't be trying to obtain the highest throughput. Checking around ... the MT7623 has 1 Gbps crypto engine for VPN, or 500 Mbps crypto engine suite (in case these are different things). What is that and how to use it? .... no idea. I sent another message asking if this could be available for the R2 (again, this is a message for Mediatek, because could be a license issue). I was making my own secure communications protocol. I will tune it for SBC machines with all this information at hand, as I see that SSH is not so well suited for these environments.

As I am developing for these platforms, that GCC version "detail" it is extremely important. Also, when using sftp it is not possible to separate the CPU throughput from the networked one, as the information must be cyphered. And there is the particular issue with the machine used. When we develop an application for everybody (as the ones in the Ubuntu distribution), it is not possible to design "for the platform"; this is particularly true when using standard sources that must be multi-platform capable. But if I am working "for the platform", then I can define particular capabilities to improve the final result even having constraints. Right now I am not at home. This night I will try the raw networking capacity ... in fact that Ubuntu has some useless things as a Docker infrastructure applied that could be interfering. Let me arrive with scissors and a broom to let it as naked as possible. About the BGI forum. My intention is to catch somebody from MTK. Also, there is something interesting: there are two forums there, one in English and a different one in Chinese (it is not a translation, it is a different forum). I will ask my wife (she is Chinese), to see what is written there... Google Translator it is not so good with that language.

0.4.12 in all them ... although (of course), I have no idea how they compiled the program for each platform, in particular for the R2. Something I found half hour ago. When sending a file with sftp, the top transference speed is 14 MB/s. This is not a conclusive test, I need to make it more precise and with the right tools, but this seems to be Fast Ethernet speed, not Gigabit. Just in case, I am using two R2 machines, with both sides using RAM disks and the machines have a 1 feet cable between them (no extra equipment involved). I also included a question in the BPI forum to see if there is more information about this subject.

sysbench --test=cpu --cpu-max-prime=20000 run --num-threads=4 Maximum prime number checked in CPU test: 20000 Test execution summary: total time: 135.3196s total number of events: 10000 total time taken by event execution: 541.1813 per-request statistics: min: 53.93ms avg: 54.12ms max: 127.76ms approx. 95 percentile: 54.71ms Threads fairness: events (avg/stddev): 2500.0000/1.87 execution time (avg/stddev): 135.2953/0.02 Just for comparison, I made the same test in an OPI-Zero (Armbian - kernel 4.13.5) and a RPI-3 (Raspbian - kernel 4.9.35). OPI cpuinfo_max_freq : 1200000 RPI cpuinfo_max_freq : 1200000 R2 -- no data -- Mediatek information specifies 1.3 GHz Total Taken by ------- Per-Req-Statistics ------ ------------ Threads fairness ------------ Event Exec Min Avg Max Approx(95p) Events(avg/stddev) Exec time (avg/stddev) OPI 121.8403 487.2788 46.72 48.73 237.59 54.89 2500.0000/1.73 121.8197/0.01 RPI 128.1032 512.2482 47.69 51.22 145.66 60.57 2500.0000/3.67 128.0620/0.01 R2 135.3196 541.1813 53.93 54.12 127.76 54.71 2500.0000/1.87 135.2953/0.02 Thanks for the SERDES and PCIe clarification.

I am very comfortable with Armbian ... a very nice distribution, congratulations to so splendid team ... and with Raspbian for the Raspberry Pi world (very closed, but it just works). I didn't like to touch LEDE because I feel that it is not oriented to what I need, but I will use some hours to see that by myself. If somebody ask me today what is the R2, I will say that it is a router platform with storage and processing capabilities. It is not a storage oriented device with many ethernet ports, neither a super computer, and if I understand that and that definition fits my needs, then it is OK. Anyway, there is that ClearFog platform, and the others referenced here. I really was not paying attention to all possible options, although I have not really enough time for researching so much because I need to design and to create things in short time, but it is important to have a clear vision about what is the current SBC landscape, "every month". In the PC world something similar happens, but the base it is very stable and the operating systems are very uniform; however, in the ARM side things are very wild (and this is refreshing, but complicated to manage). I agree ... R2 can improve, but it is not possible to do what Orange do with their machines (make one image and forget the machine). It is important to improve and improve and improve the software base. If I can help discovering problems while I assemble solutions, then I will discuss them the most my duties permits me to do it. And also, that board must evolve (not the radical change from R1 to R2 ... maybe an evolutive one from R2 to R3 some day). Manuals are not written in stone.

A38X ClearFog I think I understand now this board. - It is not an integrated board, but a "carrier" based one. You purchase the carrier board and the SOM (System on Module) to make it to works as you expect. It uses only one line for mSATA interface, the same as the R2. Then, why the difference in speed? It is necessary to add a mSATA to SATA adaptor to connect standard SATA drives, or to use mSATA drives. And uBoot must be modified to allow the PCIe slot to work as mSATA. - SolidRun Armada SOM A388 with eMMC : $69 - With ClearFog Base Carrier $129 - With ClearFog Pro Carrier $189 HummingBoard Edge Similar scenario than ClearFog (using SOM). - Only has 1 ethernet - Quad 1 GHz NXP i.MX6 version 2GB RAM and 8GB eMMC: $191 - Needs M.2 to SATA adapter - No USB 3.0 (only 2.0) For a multi-ethernet scenario with storage, the HummingBoard is limited by the lack of native multiple RJ45 and USB 2.0 (where the second ethernet could be attached). When only needing two ethernet, the ClearFog with Base Carrier seems to be enough. Cost is around 50% higher than the R2. And the Pro around 100% higher. ROCK64 ... not yet available for purchase (ships until November 3 if purchased in October 16 - Pine has their history on delays) - $60.89 without shipping ( 2GB version + 16 GB eMMC + USB-SATA cable ) ~ still lacks a secondary ethernet. - It only has one USB3 port, so the bandwidth must be shared between SATA and any secondary ethernet. ExpressoBin - $49 in amazon - Dual Core - Three Gigabit Ethernet ports (1 WAN, 2 LAN) - Independent SATA interface with its own Power Supply - Proper 12V barrel power connector - Mini PCIe - Has the place to add the eMMC but it seems must be soldered there -- R2 still has a place in its price/availability/performance ratio. If there is an ExpressoBin with eMMC included, Quad Core and 2GB, maybe it could cost around $80. In that case, would be a better option than the R2. The ROCK64 lacks interfaces to provide good bandwidth when the multi-ethernet scenario is included. It is not possible to have HummingBoard with multiple Gigabit Ethernet connections. The Armada SOM is Dual core ... could be possible to use the HummingBoard SOM with the ClearFog carrier? The main important elements are to determine if an improvement in the software can work the problems detected with the R2 for the full performance capacity. In my particular case, although performance has some importance, it is not the main driver to choose one or another product. The processing unit is more important (hence 4 cores would be better than 2), together with the integrated eMMC (I can't deal with soldering these tiny things) and the availability for connecting devices. Today all them are not so perfect (taking all factors into consideration) options. I am sure than in 6 months this will be a completely different world.

These are the results for iozone with the new parameters (without the cpufreq governor specified) ... in fact, I didn't have any cpufreq place where to define the governor, so I installed cpufreqd cpufrequtils, but noticed that there is no cpufreq module available. This mean, the Ubuntu distribution provided is not yet ready (have holes). But, anyway, the numbers are very different with the new specifications for iozone (included tests with SATA and USB3): SATA INTERFACE iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2 random random kB reclen write rewrite read reread read write 102400 4 2799 2884 36858 36836 17997 2869 102400 16 11626 12037 97141 97668 45048 11997 102400 512 114689 106011 221916 223716 186422 122568 102400 1024 120423 122761 203629 204082 92840 129277 102400 16384 122495 142574 253537 255011 236196 133064 iozone -e -I -a -s 6000M -r 16384k -i 0 -i 1 -i 2 random random kB reclen write rewrite read reread read write 6144000 16384 133880 133305 258242 258276 255023 130570 USB3 INTERFACE (with a Seagate Plus Portable Drive using USB3 interface) iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2 random random kB reclen write rewrite read reread read write 102400 4 11774 18878 16123 16102 517 2879 102400 16 37521 53680 37446 37280 2031 8850 102400 512 56314 54322 60220 62229 36569 45161 102400 1024 54660 55310 59720 60348 52624 48325 102400 16384 105629 112361 119853 113174 110712 107491 iozone -e -I -a -s 6000M -r 16384k -i 0 -i 1 -i 2 random random kB reclen write rewrite read reread read write 6144000 16384 117796 117946 118558 118424 104536 98876 I was writing about the other machines ... let me check them more carefully before providing a better feedback. The only I will advance now, is that the BPI-R2, in particular because of the SATA using only one PCIE line, is that the machine is medium-range category.

Worried about machine disk cache ... swapoff -a hdparm -W 0 /dev/sda2 iozone -a -g 4000m -s 400m -i 0 -i 1 -r 4K kB reclen write rewrite read reread 409600 4 149466 161515 579029 580156 819200 4 145173 152286 577471 580769 1638400 4 143195 145771 398831 552196 3276800 4 138856 144754 249275 250088 iozone -a -g 4000m -s 400m -i 0 -i 1 -r 1024K kB reclen write rewrite read reread 409600 1024 153636 163101 552258 555698 819200 1024 143936 148538 554790 556472 1638400 1024 140781 146167 415373 530808 3276800 1024 141351 144649 228767 228786 iozone -O -i 0 -i 1 -i 2 -e -+n -r 4K -s 2000m random random kB reclen write rewrite read reread read write 2048000 4 34323 0 63103 0 9124 829 I didn't notice the big drop in speed with the last line on iozone. But, what this mean? Writing it is almost the same all the time with or without cache (maybe this is because of the SSD factor). Then, it seems that, when using this particular machine, it is better to use bigger blocks but to control the file size (if possible), for being around 1638400 KB (more precise tests to check the particular number where this reduce so much the speed are needed). The really troublesome element is the last test with the random write and without disk cache. Why is this so important in this case? Could be another type of problem?

Here are some numbers: iozone -a -g 4000m -s 400m -i 0 -i 1 -r 4K kB reclen write rewrite read reread 409600 4 155162 175241 583219 583597 819200 4 153467 165512 583362 585346 1638400 4 151321 160254 259718 509318 3276800 4 154243 161490 249181 248769 iozone -a -g 4000m -s 400m -i 0 -i 1 -r 1024K kB reclen write rewrite read reread 409600 1024 156180 178749 555343 557200 819200 1024 153312 171974 555450 557801 1638400 1024 155625 164754 373269 507221 3276800 1024 156104 159858 223787 224892 iozone -O -i 0 -i 1 -i 2 -e -+n -r 4K -s 2000m random random kB reclen write rewrite read reread read write 2048000 4 38119 0 62234 0 9654 22192 I didn't change any performance parameter for the CPU as in previous tests with Legacy Armbian Kernel, because I really don't know how to do that on 4.4.70. The disk is a SanDisk SSD Z400s. I don't have at hand tests using this SSD and the OPI NAS (I made some tests with a WD Blue hard disk). But in general, and checking other tests on the Armbian site, the NAS test limit was defined by the USB 2.0 limiting bandwidth, not the disk. Knowing all these differences, this machine is around 348% faster writing and 1725% faster reading that an OPI Zero with the NAS expansion. It is also possible that this kernel is not well tuned, but it is clear that the machine is faster using the SATA interface. Then there are the own disk limitations (https://www.sandisk.com/business/computing/z400s) Sequential laboratory Read/Write up to 546/342MB/s ... so reading it is around the hard disk full speed, although writing is only using half disk speed (something to take into consideration).

BPI-R2 in Armbian forun has been as a thunderstorm. It is sad things are not in good terms, as the machine looks nice and I am sure that it could have a nice horizon with Armbian. I am working a security device on these devices (an appliance containing a small network), and by now, this is the best hardware option available: eMMc, many gigabit ethernet ports (two different networks and, in the future, different VPNs in one of them), USB 3, 2 GB RAM, SATA support, 12V power, GPIO, etc... for less than $100. It is clear that the machine is a new one and it has some problems, but it is possible to deal with them. - Wrong number in the partition configuration when trying to boot from eMMc fix:(http://forum.banana-pi.org/t/cant-boot-from-emmc/3826) - Have no idea how to turn on a connected FAN in the FAN socket, although the secondary SATA has 5v and 12v and there are the GPIO pins. - Very long time pressing the power button for the machine to start - optional fix (soldering): (http://forum.banana-pi.org/t/bpi-r2-boot-power-suppy/3647/22) - Two SATA interfaces in the same SoC line (as a workaround, use a SATA and a USB disk, not two SATA) - The software distribution from BPI is a mix of things ... there are even parts for Raspberry that I am sure have no relationship with the R2. Here the grease must be cleaned. There are other details, but I really don't care on them. The machine form factor it is not really to make an entertainment center, and the only I need is to be able to check the console with the debug pins or to connect through the network. If BPI and MTK could create a server distribution and not the full Mate based Ubuntu, would be very useful. And there is a LEDE available also, although it is a different point of view. -- P.S. I am waiting for some machines to make a deeper test. In this moment I have an OPi Zero in the entrance point, so I can't check the total bandwidth ... later I will provide more data, including my own disk throughput tests.

I made some tests several months ago with two Zero machines, one powered from the other (that even worked as an usb0 network device) and a hard disk from one of them and it worked "well" ... but you are right, I don't know in the long term if this method for powering the computers could produce some damage. Maybe with a low consumption disk (or without it), would be better. The next test was different. The Zero2+H5 was connected in a NAS expansion (from the internal 13 connector) and receiving the power from the barrel connector there. The hard disk was in the NAS expansion, and a Zero was connected on the Zero2+H5 GPIO pins. So, no micro USB was powering either machine. In general, if it is necessary to power some machine through the USB port, then it is better this to be the last one in the chain, and not to ask that machine to consume a lot of power by reducing the quantity of connected devices there. Now I am making a different arrange, but with a BPI-R2 machine; this receives 12V and was built for more demanding work than the OPi Zero ... then I can connect a Zero from the BPI-R2 USB port or from the BPI-R2 GPIO with less worries (in fact, I will replace the Zero with a BPI-M2+ when it arrives).

Same behavior. I decided to follow a different path for my current need (no OTG-HOST) ... but as a side project I will continue checking what is the reason this happens. When I have it working I will return with my findings. What worries me is the lack of parameters; where is defined how to work the packages structure? However, this is not Armbian decision but Linux one (the people that are creating that infrastructure part in the kernel core). Let me see if I can design a different way to do that type of connection, without musb neither g_ether.

I added the beta repository and updated there ... not yet on 4.13 but 4.11.12 ... and with just this change, several packages didn't arrive "officially" to the HOST machine arrived corrupted but with the OTG machine name an other data ... still doesn't work, but it is different than previous versions. This shows that it is different inside (it is evolving). Then I went to check 4.14 rc3 repository on the Kernel.org original place for all distributions and found this notice there: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/usb/musb/musb_core.c I think this is not ready, and this is why g_ether is still there although marked as "legacy". Could be possible that "musb" and "g_ether" are working one against the other? The problem with the dtb file is that everything is very interleaved. I will try to disable musb to see if this makes any difference. Although I also will try to find another solution "without" the gadget part until this to be officially ready.

No no, I understand it. I was trying with the Mainline because the last Legacy version has the same behavior (my mistake for not to say it, I'm sorry). I made this to work several months ago with complex scenarios involving several Zero machines, and let it at one side for a while because I was making other things. When I returned to this part using a recent Legacy update this didn't work. I thought that was my mistake, but these days I have been returning to work on this and really it doesn't work with Mainline neither with Legacy. At least you are giving me a confirmation. I can't use the OPIZ+2H5 in this moment for any serious task ... I will wait, and if I can help some way let me know. Thanks

I am having a problem and I really don't know if this is my fault or if could be a problem with mainline. My current version is: Linux orangepizero 4.11.5-sun8i #11 SMP Fri Jun 23 20:03:23 CEST 2017 armv7l armv7l armv7l GNU/Linux I have been fighting with g_ether on an OPIZero as slave (OTG mode) connected with another machine HOST USB port, but no matter what I do, it is not possible to make it to work well. Several months ago I was able to connect two OPIZero machines with legacy kernel in both of them. But now, I need to use another machine (OPIZ+2 H5 or BPI-R2) having 4.4 kernel and it doesn't work. For this I decompiled the dtb file and replaced the "peripheral" with "otg". Original: usb@01c19000 { compatible = "allwinner,sun8i-h3-musb"; reg = <0x1c19000 0x400>; clocks = <0x2 0x20>; resets = <0x2 0x11>; interrupts = <0x0 0x47 0x4>; interrupt-names = "mc"; phys = <0x11 0x0>; phy-names = "usb"; extcon = <0x11 0x0>; status = "okay"; dr_mode = "peripheral"; linux,phandle = <0x31>; phandle = <0x31>; }; New one: usb@01c19000 { compatible = "allwinner,sun8i-h3-musb"; reg = <0x1c19000 0x400>; clocks = <0x2 0x20>; resets = <0x2 0x11>; interrupts = <0x0 0x47 0x4>; interrupt-names = "mc"; phys = <0x11 0x0>; phy-names = "usb"; extcon = <0x11 0x0>; status = "okay"; dr_mode = "otg"; linux,phandle = <0x31>; phandle = <0x31>; }; Commented g_serial and added g_ether in the /etc/modules file. Disabled Network-Manager, installed Avahi in both machines. Added the following in /etc/network/interfaces file. iface usb0 inet manual up /usr/sbin/avahi-autoipd --force-bind -D $IFACE down /usr/sbin/avahi-autoipd --kill $IFACE And bring usb0 to life with ifup usb0. When I try to ping from the OPIZero to the other machine the ping fails, but tcpdump receives "something" in the destination side (the HOST one): 12:11:42.826864 ff:ff:46:7b:0e:f2 (oui Unknown) > 2e:00:ff:ff:ff:ff (oui Unknown), ethertype Unknown (0x8718), length 48: 0x0000: 0806 0001 0800 0604 0001 467b 0ef2 8718 ..........F{.... 0x0010: a9fe 0702 0000 0000 0000 a9fe 04cb dead ................ 0x0020: beef The OPIZero usb0 port configuration is this one: usb0 Link encap:Ethernet HWaddr 46:7b:0e:f2:87:18 inet6 addr: fe80::447b:eff:fef2:8718/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:525 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:28094 (28.0 KB) usb0:avahi Link encap:Ethernet HWaddr 46:7b:0e:f2:87:18 inet addr:169.254.7.2 Bcast:169.254.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 As can be seen, the source address is shifted 16 bits (the 0x8718 type it is really part of the source hardware address). To check if the destination was the problem, I connected a Raspberry Pi Zero machine with Raspbian configured as OTG and worked without any issue, and tested the HOST side with Armbian on OPIZ+2H5 and Ubuntu on RPI-R2 with the same result in both, so "I think" that the problem is with the Armbian in the OPIZero.

There is something really strange ... to have a double check, I replaced the NAS extension with direct cables to the USB port pinouts. Then, I found that my problem is consistent (it is not related with the NAS extension). Doing this, I found that the H5 machine really has been working well as the HOST side of the Ethernet over USB connection; for doing that, I connected a Raspberry Pi Zero and immediately acquired the address and performed a full perfect connection. But my Orange Pi Zero (with Legacy kernel) doesn't like to work well. Then, reading the usb0 link on the OPIZ+2 with tcpdump, I found that both machines are communicating but that my ping packages are shifted 4 bytes. This is the reason they don't like to agree with the communication. No one package arrive complete and they are broken at 28 bytes. Checking around, it seems there was a bug in some kernel driver on the Gadget ethernet implementation for kernel 4.0 series . But as it works well when connecting with the Raspberry Pi Zero, tomorrow I will check with a mainline kernel on the Orange Pi Zero to verify if the problem is with the kernel versions.

My mistake ... it really extends the USB ports in the SBC ... just it "looks" as a HUB. There is also something else I realized today with the board, I will corroborate "without it" to share in the forum. Let me review my comment and to check how the other thread is going, to complement the data there.

I have been doing a lot of tests with that NAS expansion board (two into production environments and other two in laboratory). The USB ports are shared with the SATA ports one to one. The one next to the SATA port has the same bus as that port. The other one is for the mSATA connector. When whatever USB device is connected, the corresponding SATA device will disappear (tested with both, mSATA and SATA devices). If no SATA device is connected, it works as a standard USB hub. And if both SATA are connected (if those devices are not very power hungry and the machine can deal with that), no USB port "must" be used. In fact, that expansion is a little dangerous to use ;-) ... if I attach SATA devices, I will hide physically the corresponding USB ports if I don't like to corrupt my data or being out of service. What I still don't know is how to use the microUSB in the SBC when the expansion it is in place, but that is another thread discussion.