research Marvell based 4 ports mPCI SATA 3.0
1 1

13 posts in this topic

Recommended Posts

Clearfog PRO with Linux 4.12.4-mvebu

Samsung 840 PRO utilization:

iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    62501   101691    98761    94430    34522    93050
          102400      16   183325   215091   231439   232948   101667   171875
          102400     512   301985   307789   341261   344567   328849   309560
          102400    1024   294818   309177   345099   347791   339763   309775
          102400   16384   271066   344426   384226   388587   387169   346825

2 x Samsung 840 PRO in RAID0, ext4 utilization:

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    67614    95508    94561    94984    32561    74157
          102400      16   174075   200947   222430   223840   109654   199578
          102400     512   307905   294689   317652   308007   307057   268841
          102400    1024   307821   317784   327007   330213   322851   317024
          102400   16384   286193   383463   394221   398474   397951   374333

 

Logs: http://sprunge.us/CfRS 

Controller temperature is stable at 64°C, network utilization is also at top speed.  Card was donated via Amazan wish list. Thank you!

20170731_121113.jpg

20170801_144901.jpg

 

Hummingboard 2 with Linux 3.14.79-cubox (mainline not working out of the box)


Not that impressive as Clearfog,  but works. 

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    28502    39271    37895    37983    22325    38821
          102400      16    68494    80988    69940    70511    53264    80047
          102400     512   116594   118132    93789    94280    92828   118884
          102400    1024   138185   140277   122610   123717   120853   139393
          102400   16384   183976   183169   151745   153813   151890   179652

Logs: http://sprunge.us/OeMY
 

Espressobin

2 x Samsung PRO 840, RAID 0, BTRFS
http://sprunge.us/iBFS

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    21547    20493    37509    37024    21747    17728
          102400      16    61955    56230    93068    93795    64349    45724
          102400     512   188611   186828   175688   176961   172098   185109
          102400    1024   215817   216559   189477   191846   188481   214727
          102400   16384   226764   228502   229162   238883   222826   229894

Reference: A20's native SATA
 

Spoiler

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    18190    21041    44757    45359    24223    21639
          102400      16    27700    30195   101335   105895    72086    29925
          102400     512    34835    35088   182999   184586   179342    34970
          102400    1024    34851    35237   188542   189265   187278    35235
          102400   16384    34571    36150   236663   238700   238065    35947

 


 

 

This document is work in progress ...

lanefu and StuxNet like this

Share this post


Link to post
Share on other sites

Mine arrived today :)

 

Clearfog_ASM1062_-Marvell_88SE9215.jpg

 

7 SATA ports in total now but most probably I'll focus on dual disk performance comparing

  • ASM1062 / ASM1062
  • 88SE9215 / 88SE9215
  • ASM1062 / 88SE9215
  • Armada / 88SE9215
  • Armada / ASM1062
  • Armada / Armada (using some u-boot Voodoo to transform one of the mPCIe slots into mSATA and using the other adapter to do mSATA --> SATA 'conversion')

BTW: Is there a reason why you did not publish Espressobin numbers so far? I made another attempt with this board today but stopped already again...

 

 

Share this post


Link to post
Share on other sites
30 minutes ago, tkaiser said:

BTW: Is there a reason why you did not publish Espressobin numbers so far? I made another attempt with this board today but stopped already again...


I haven't seen any mPCI card on this board yet at least on stock kernel while I could not boot mainline kernel - from SD card.

Share this post


Link to post
Share on other sites

Since I had the EspressoBin booted, the Marvell 88SE9215 SATA card in the mPCIe slot and 2 SSDs provided with external power I did some more testing comparing 

  • single SSD performance with an ext4 partition on the onboard SATA port and the 88SE9215
  • btrfs RAID-0 performance with both SSD attached to the 88SE9215 compared to one on each controller

My test setup sucks since I've no idea how fast Armada 3720 CPU cores clock (I let 7-zip benchmark run and '7z b' reported a total of just 1414) and the 2 SSDs are cheap consumer crap way too slow (see smartctl output here -- write performance with both SSDs sucks as soon as amount of data written exceeds more than 1 or 2 TB, especially the Intel 540 then gets dog slow at around ~60MB/s, we can see this at the end of the numbers when I tested btrfs RAID-0 with 2 GB filesize): 

Intel 540 / Armada 3720 SATA                                  random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    47255    70977    87288    81713    29007    71294
          102400      16   108707   103552   209717   215060    75508    71568
          102400     512   127331   195726   399994   455759   374609    82367
          102400    1024   143364   339287   468693   472660   423549   339132
          102400   16384   292534   328546   404603   407138   406887   336762

Intel 540 / Marvell 88SE9215 SATA                             random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    34074    45336    41414    41787    18210    38243
          102400      16    54685    48116   108386   102390    50574   109665
          102400     512    92229    68978   184259   197881   185550    74988
          102400    1024   108717   144664   261007   261002   230533   108468
          102400   16384   164837   191212   291178   292019   285631   252738

Samsung EVO840 / Armada 3720 SATA                             random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    44342    74243    88564    87235    40085    68688
          102400      16   138468   204412   223066   224952   127620   189047
          102400     512   392681   384072   445833   452317   425046   401434
          102400    1024   391877   383661   456881   464721   450389   399017
          102400   16384   375014   381085   434583   440510   437606   377176

Samsung EVO840 / Marvell 88SE9215 SATA                        random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    34457    50728    54088    52026    28858    51844
          102400      16   102471   131968   144037   143400    87486   131722
          102400     512   246405   252458   249820   244493   246951   251780
          102400    1024   251542   258106   261483   249819   255789   261397
          102400   16384   244643   272618   279133   292771   291875   266423

btrfs raid-0 / 88SE9215 + 3720 SATA                           random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    28214    27320    49198    49484    26154    23649
          102400      16    70382    66525   126629   127612    72779    57479
          102400     512   349803   345833   302641   308043   285616   345608
          102400    1024   399405   392532   320462   331041   314945   409883
          102400   16384   433521   426136   407682   416113   407861   410509
         2048000   16384   271999   126972   412175   412226   403249   126939

btrfs raid-0 / 2 x Marvell 88SE9215 SATA                      random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    25292    23843    42802    42662    23992    21229
          102400      16    62994    56114   105396   105729    65521    52447
          102400     512   225543   225750   193574   195734   188411   217733
          102400    1024   232385   224001   202101   207403   201948   231705
          102400   16384   270143   272697   247752   254730   260208   271869

To me it looks obvious that we're suffering currently from CPU cores being clocked too slow on Espressobin. But no idea how to adjust this (does cpufreq scaling work with https://dl.armbian.com/espressobin/nightly/Armbian_5.32.170626_Espressobin_Ubuntu_xenial_default_4.4.73.7z?)

 

11 minutes ago, Igor said:

What about some wireless card? You have some Atheros, right? Does it work for you?

 

Not tried yet. I've one AR9380. Will give it a try tomorrow.

Share this post


Link to post
Share on other sites
4 minutes ago, Igor said:

 

It works for me too, but it's not very impressive.

 

 

Well, by looking at the scores above to me it seems obvious we're bottlenecked by CPU on Espressobin currently (EVO840 directly connected to 3720 SATA port maxes out at 440 MB/s while the RAID-0 using two separate controllers/buses does not exceed 412 MB/s -- I would guess that's due to RAID/checksumming overhead here and if CPU wouldn't be the bottleneck we would see with a RAID-0 combining onboard SATA and 88SE9215 +700MB/s or even more.

 

I tried to use 7-zip numbers to get an idea at which clockspeed Armada 3720 is running but realized that I'm trying to compare Debian Stretch package (7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21) with numbers generated by Jessie/Xenial: '7-Zip (A) 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18'. Crap, need to retest with Jessie on Espressobin or test another 64-bit platform with Jessie. :(

Share this post


Link to post
Share on other sites

Testing on a Clearfog Pro with performance cpufreq governor (1600 MHz). Only interested in multi disk performance and comparing onboard SATA (labeled 'Armada' below), ASM1062 and Marvell 88SE9215 which this topic is about. The two 128GB SSDs I test with are a Samsung EVO 840 and an Intel 540 (both are not the best candidates for such a test since too slow especially wrt writes when amount of data written exceeds ~1 TB).

 

For convenience reasons and to be able to detect possible data corruption I relied on btrfs RAID-0 (striped data but mirrored metadata/checksums): 'mkfs.btrfs -m raid1 -d raid0 /dev/sda /dev/sdb'. Mounted all the time as '/mnt/raid-0 btrfs rw,relatime,ssd,space_cache,subvolid=5,subvol=/ 0 0'.

 

ATA hotplug works with both ASM1062 and 88SE9215 (externally powered the SSDs so I was able to switch them on/off) but I doubt it's wise to try out hotswap with the standard SATA connectors here so I chose to shutdown the board between all tests.

 

Below the combinations listed, the left one is always the EVO 840 and the right one the Intel 540, the sprunge links are just to document whether something strange happened (dmesg output included) and controller features are as expected (eg. NCQ / queue depth 31 all the time):

ASM1062 / ASM1062 http://sprunge.us/KFJE                      random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    41581    40231    66934    65931    30858    36496
          102400      16    78795    85162   129029   132381    75406    78708
          102400     512   235104   230833   224359   235073   230958   233557
          102400    1024   237243   237869   231138   233893   231208   236350
          102400   16384   278629   280890   346025   366094   368530   144826

88SE9215 / 88SE9215 http://sprunge.us/GiOD                    random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    42725    40902    67686    67951    30666    37689
          102400      16    88221    86106   133593   134186    75724    79272
          102400     512   230455   230557   236345   239362   234916   228943
          102400    1024   232845   228149   231002   236846   232255   228675
          102400   16384   251719   144849   342737   350257   349299   174308

88SE9215 / ASM1062 http://sprunge.us/VcaN                     random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    42550    41349    67662    67702    30974    37499
          102400      16    87533    87006   127737   133436    75295    79513
          102400     512   321095   317909   324035   329646   318073   315473
          102400    1024   319864   324307   320939   325636   320102   322490
          102400   16384   381197   349591   538466   580906   588403   357470

ASM1062 / 88SE9215 http://sprunge.us/KfCX                     random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    42143    41064    66865    67056    30924    37030
          102400      16    87302    77005   132274   132805    75252    79618
          102400     512   317596   315885   300093   328767   315876   315025
          102400    1024   322192   320242   319936   325295   316761   318602
          102400   16384   347208   277583   556563   592881   600894   117150

Armada / 88SE9215 http://sprunge.us/gAeb                      random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    43562    42551    71538    72146    31692    38201
          102400      16    91960    90462   142261   142811    78192    83157
          102400     512   320566   317511   364218   370690   351141   315569
          102400    1024   269086   262615   348363   363624   355324   319879
          102400   16384   349914   376723   531844   561922   587068   221096

Armada / ASM1062 http://sprunge.us/EOYJ                       random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    43630    41769    70908    71520    31611    37924
          102400      16    89669    88133   137684   125599    77745    82730
          102400     512   316515   314883   347953   360841   344984   220572
          102400    1024   144820   324249   349778   355869   349079   326118
          102400   16384   380195   417907   583988   580550   604784   376562

Armada / Armada                                               random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    45984    43445    75553    75726    32493    37967
          102400      16    96203    92794   150890   151756    81030    86489
          102400     512   357109   357089   363720   369983   354740   354042
          102400    1024   357485   369719   354278   358965   353952   367930
          102400   16384   406932   402512   532718   559035   554006   361749

Armada / Armada 2nd run http://sprunge.us/iIhN                random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    38889    43882    75523    75631    32143    34014
          102400      16    90735    89593   149764   150979    81072    82520
          102400     512   204282   114150   352789   365723   352634   126669
          102400    1024   122663   117834   325485   360243   353983   123490
          102400   16384   126510   120139   530389   561534   558192   125412

First conclusions:

  • The above numbers unfortunately do NOT indicate SATA controller performance but are the results of an insufficient test setup (wrong SSDs used, both my SSDs start to slow down especially write performance after a small amount of data is written, Samsung calls this TurboWrite, Intel does the same on cheap consumer SSDs and my 540 drops then down to ~60 MB/s). The effect of this can be seen with the last 2 tests executed one after the other with otherwise identical test setup. Write performance ~400 MB/s on first run (limited by the slower Intel 540 writing with ~200 MB/s), then dropping down to ~120 MB/s (now the Intel's cache is full and it slows down to 60 MB/s -- with a dual disk RAID-0 overall bandwidth is the one of the slowest RAID member * 2). To get really an idea how the 3 SATA controller setups vary I would've needed at least 2 Samsung 850 Pro with 256GB (or any other SSD that reads/writes with +500 MB/s all the time).
  • If I would do the test a second time I would ensure that the SSDs remain powered on after each test for some minutes to empty their caches to recover full performance. I skipped this above (quick&dirty mode) which resulted in a lot of write performance numbers that are lower than necessary.
  • When both disks are attached to the same PCIe controller performance of ASM1062 and 88SE9215 is identical (I would believe in this mode they're bottlenecked by the single PCIe 2.x lane). I had zero issues with our kernel and NCQ (tested also with the ASM1062 and 'echo 1 > /sys/block/sd?/device/queue_depth' but no differences in performance which is an indication that we need to look a little bit closer here)
  • It also made no difference which SSD was connected to which controller when I tested the '1 SSD per PCIe controller' modes.
  • The numbers above when one or two SSDs were connected to Armada's native SATA ports are IMO all questionable. Too much influence due to insufficient SSDs, these numbers should be dropped and another round of tests with really performant SSDs is needed
  • For that what most people are most likely interested in (attaching HDDs) the numbers are sufficient since 'fast enough'. In case anyone is asking about RAID5: At least I won't test this since I consider RAID a waste of money and resources at home and even dangerous since home/SOHO RAID afficionados never test what's important (failure scenarios and rebuilds)

Final remarks:

  • My numbers above can not be compared with Igor's since RAID-0 vs. single SSD, btrfs (CoW, checksumming, read/modify/write cycle with data smaller/equal default blocksize) vs. ext4 (so different amount of overhead involved)
  • I found it somewhat surprising getting identical performance with NCQ queue depths of 1 and 31. This is something that needs investigation with next rounds of tests (some NCQ details here)
  • The Marvell 88SE9215 on all those mPCIe cards wears a heatsink for a reason. It gets really hot during operation and I would assume consumption is also significantly higher than ASM1062 (not tested/measured yet).

Share this post


Link to post
Share on other sites
16 minutes ago, tkaiser said:

My numbers above can not be compared with Igor's since RAID-0 vs. single SSD, btrfs (CoW, checksumming, read/modify/write cycle with data smaller/equal default blocksize) vs. ext4 (so different amount of overhead involved)


I can add a test with 2 x 840 PRO 256G into RAID0 and BTRFS with Clearfog. If it makes a value?

Share this post


Link to post
Share on other sites
1 hour ago, Igor said:

I can add a test with 2 x 840 PRO 256G into RAID0 and BTRFS with Clearfog. If it makes a value?

 

Definitely. But we must really take care to not mix these numbers since RAID/btrfs/dual disk overhead massively influences performance. To get an idea I repeated single SSD tests with my EVO840 and a single ext4 partition on the Clearfog to check whether then we see some differences between the 3 controllers:

Armada EVO840 ext4 http://sprunge.us/MLGY                     random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    81488   117728   132076   135454    41432   121492                                                          
          102400      16   215389   267562   278121   283023   127460   266775                                                          
          102400     512   372032   368574   411084   423599   378666   383032                                                          
          102400    1024   362836   369646   415503   394737   411496   366720                                                          
          102400   16384   327837   379992   468341   469976   481556   387493                                                          

88SE9215 EVO840 ext4 http://sprunge.us/SiEG                   random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    80772   104881   113867   115767    45327   104451                                                          
          102400      16   183112   214327   235989   237563   131997   213925                                                          
          102400     512   310924   316593   351351   354992   342506   310179                                                          
          102400    1024   313374   313495   356670   360649   354205   307503                                                          
          102400   16384   296566   340321   372968   394449   393854   338588                                                          

ASM1062 EVO840 ext4 http://sprunge.us/dSAN                    random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    77791   101160   113403   113886    45548    98541                                                          
          102400      16
Error writing block 5446, fd= 3
write: Input/output error

iozone: interrupted

 

Edit: Stopped the board, re-inserted SATA cable (the connectors on my ASM1062 do not provide cables fitting tightly), tested on both SATA ports various times. It now looks like this with ext4/EVO840/ASM1062:

ASM1062 EVO840 ext4 http://sprunge.us/VOON                    random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    78708   101923   108780   111426    46000    95101
          102400      16   178421   214891   233270   234523   133188   214486
          102400     512   336295   332287   349769   353435   341314   340677
          102400    1024   334856   336743   344395   356114   350605   335704
          102400   16384   310091   377760   385584   390347   389789   366494

Edit 2: SMART CRC error counter attribute was 88 before (see smartctl output from yesterday). Now it's 89:

199 CRC_Error_Count         0x003e   099   099   000    Old_age   Always       -       89

In other words: The issue I ran into before was SATA connector crappiness: The 2 SATA connectors on my el cheapo ASM1062 card aren't that reliable. With such cheap SATA cards after connecting any device it's advisable to check smartctl output for attribute 199 and then immediately run a heavy disk benchmark and check dmesg output and SMART attribute 199 again. This SMART counter is there for exactly this reason: to identify cabling/contact problems! 

Share this post


Link to post
Share on other sites

Today this little guy arrived. It's a Transcend TS120GMTS420 M.2 2242 SSD I chose for good random and sequential IO performance. Full smartctl output after mounting it on my Clearfog here (this thing is pretty new so not already supported by smartmontools -- time to submit a patch open a ticket).

 

TS120GMTS420_on_Clearfog.jpg

 

Since we have a few btrfs RAID-0 numbers above I chose to repeat such tests, comparing single drive performance with possible RAID-0 modes (btrfs vs. mdraid -- the whole testing is somewhat useless and this is just to illustrate that the computation overhead with such modes on rather weak ARM boards is something to have in mind):

 

TS120GMTS420 in M.2 slot                                      random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    86865   124684   140973   140224    32522   101844
          102400      16   206105   251838   293776   282729    84623   202503
          102400     512   363707   377155   409380   401946   358966   372971
          102400    1024   364578   358095   402539   404249   387812   376067
          102400   16384   310808   407947   500231   497107   470879   398703
         3072000   16384   420958   403649   509368   507120   469027   405752

EVO840 behind 88SE9215                                        random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    77592    98082   105627   110948    42939    80088
          102400      16   161754   167814   224142   231793   131453   194794
          102400     512   254022   251896   321636   324871   314115   286720
          102400    1024   288675   293282   325360   328868   323345   294681
          102400   16384   278020   331917   371216   393551   390127   338349
         3072000   16384   335804   338734   384926   384487   383777   224859

TS120GMTS420 and EVO840 as btrfs RAID-0                       random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    42327    42858    73490    73437    30527    37809
          102400      16    91631    89645   145469   146531    80406    65167
          102400     512   336147   329055   364294   372163   336223   342325
          102400    1024   338846   344515   361248   365292   348673   331471
          102400   16384   391086   399913   619237   574991   416424   359991
         3072000   16384   415433   418134   587887   626527   620356   308949

TS120GMTS420 and EVO840 as mdraid RAID-0                      random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    83308   109760   113192   120491    38131   108471
          102400      16   191221   226954   241912   247788   105236   200482
          102400     512   314386   324797   357034   361831   338066   335216
          102400    1024   476716   500618   542100   550804   581167   549686
          102400   16384   503050   659126   745353   751381   749971   655232
         3072000   16384   677670   660505   734458   746781   739883   655215

(bash history here to check test methodology)

 

Off-topic: I bought the SSD as companion for this little USB3 M.2 enclosure (JMS578 so excellent performance expected). But since SMART attribute 194 (disk temperature) after benchmarks reads already 65°C I'm a bit concerned ;)

 

 


JMS578_M2.jpg
 

 

Edited by tkaiser
Added smartmontools ticket link for SSD

Share this post


Link to post
Share on other sites
1 hour ago, tkaiser said:

Off-topic: I bought the SSD as companion for this little USB3 M.2 enclosure (JMS578 so excellent performance expected). But since SMART attribute 194 (disk temperature) after benchmarks reads already 65°C I'm a bit concerned


Got similar one, also hot like hell ... chip was covered with some nail polish or something. 

Spoiler

20170830_161944.jpg

 

 

1st post updated with 2 x 840PRO in RAID0.

Share this post


Link to post
Share on other sites
1 1

  • Support the project

    We need your help to stay focused on the project.

    Choose the amount and currency you would like to donate in below.