10 10
tkaiser

SD card performance

Recommended Posts

https://www.sandisk.com/home/memory-cards/microsd-cards/high-endurance-microsd  I'm guessing this must be using a better binned MLC flash.

 

Why? Ever listened to marketing guys playing bullshit bingo? 'Hey, let's make a white variant and write something about endurance on the packaging! Might increase market share by 1%'

 

Better look on their numbers (32GB: 5,000 hours 'Full HD (1920x1080) video content recorded at 26 Mbps to one device' --> 5000 * 3600 * 26 / 8 / 1024 = 57,129 GB). So they claim a 1,785 ratio here. The card could be filled with roughly 2,000 times the amount of data compared to its capacity. So given that's the usual flash capable of 1000 write cycles they could simply use 'a bit more' overprovisioning (not a few percent as on normal quality SD cards but 100 percent: the 32GB variant has 512Gb capacity and the larger one is 1024Gb and both expose just half of it through their controller/FTL).

 

Since the only use case mentioned is video recording this means just 'sequential write' of large block sizes so FTL (flash translation layer) might be highly optimised for keeping write amplificatiojn as low as possible which may result in painfully low random IO performance (since FTL/controller priorize minimizing write cycles over performance, something that works pretty well if it's only about sequential writes but is challenging as soon as random IO is also concerned).

 

In other words: As soon as we think about an SD card being used in a computer things might be totally different (not that surprising but one of the issues so many people run into bad experiences with SBC, since they rely on irrelevant specs like 'speed class' and choose horribly bad performing SD cards). With random IO write amplification might be magnitues higher especially if/when this card type chose a rather large erase block size. If the latter is eg. 8Mb then writing just a single byte followed by a sync will result in a 1:1,000,000 ratio and not 1:1 as with the 'recording video' use case.

 

So still based on the assumption that we're talking here about 8Mb erase block size (no idea whether that's really the case, this is something a short flashbench run could show), using this card in an SBC used as desktop replacement and browsing the web Firefox alone (permanently saving a lot of stuff to a few SQLite databases with 32KB chunk size configured) could result in a write amplification ratio of 1:32 and the card will show failures already after just 1,785 GB written and not 57,129 GB as with the video use case.

 

TL;DR: It's all about the use case. SD card numbers are based on an entirely different use case (sequential writes as it happens in digital cameras and video recorders) so both performance and endurance numbers you get are ALWAYS WRONG when talking about SBC use case. You always have to check yourself.

Share this post


Link to post
Share on other sites

 

 

 

 

Better look on their numbers (32GB: 5,000 hours 'Full HD (1920x1080) video content recorded at 26 Mbps to one device' --> 5000 * 3600 * 26 / 8 / 1024 = 57,129 GB). So they claim a 1,785 ratio here. The card could be filled with roughly 2,000 times the amount of data compared to its capacity. So given that's the usual flash capable of 1000 write cycles they could simply use 'a bit more' overprovisioning (not a few percent as on normal quality SD cards but 100 percent: the 32GB variant has 512Gb capacity and the larger one is 1024Gb and both expose just half of it through their controller/FTL).

 

 

 

2000 p/e is actually quite a lot for an sd card. First note, that this is calculated, so it means that if you take into account write amplification you're getting closer to 3000 erase cycles. This is quite a lot for 2d mlc flash these days and especially a lot of an sd card. You must know, that most sd cards use lower bin tlc flash, which is only good for around a few 100 p/e. So this is definitely an improvement here.

 

 

In other words: As soon as we think about an SD card being used in a computer things might be totally different (not that surprising but one of the issues so many people run into bad experiences with SBC, since they rely on irrelevant specs like 'speed class' and choose horribly bad performing SD cards). With random IO write amplification might be magnitues higher especially if/when this card type chose a rather large erase block size. If the latter is eg. 8Mb then writing just a single byte followed by a sync will result in a 1:1,000,000 ratio and not 1:1 as with the 'recording video' use case.

 

So still based on the assumption that we're talking here about 8Mb erase block size (no idea whether that's really the case, this is something a short flashbench run could show), using this card in an SBC used as desktop replacement and browsing the web Firefox alone (permanently saving a lot of stuff to a few SQLite databases with 32KB chunk size configured) could result in a write amplification ratio of 1:32 and the card will show failures already after just 1,785 GB written and not 57,129 GB as with the video use case.

 

TL;DR: It's all about the use case. SD card numbers are based on an entirely different use case (sequential writes as it happens in digital cameras and video recorders) so both performance and endurance numbers you get are ALWAYS WRONG when talking about SBC use case. You always have to check yourself.

 

 

 
 
Unless we have exact specification, we don't know how controller handles random data, so we really can't predict how high write amplification will be when dealing with random data. But given the low sequential speed of the cards, i'm quite confident that this is not strictly optimized for sequential data. If it was, sequential speeds would be much better than 20MB/s.

Share this post


Link to post
Share on other sites

Unless we have exact specification, we don't know how controller handles random data, so we really can't predict how high write amplification will be when dealing with random data. But given the low sequential speed of the cards, i'm quite confident that this is not strictly optimized for sequential data. If it was, sequential speeds would be much better than 20MB/s.

 

No one knows until controller strategy has been revealed (I would assume the controller's 1st priority is always minimizing write amplification so maybe that's the reason).

 

But as already written: As soon as the use case differs from that what SD cards are made for one always has to test. And with new cards it's getting characteristics first (flashbench), then testing performance and maybe doing an endurance test (would be both interesting and somewhat braindead when done 'just in case' since card is destroyed afterwards).

 

My attempt to deal with limited endurance of SD cards is simply limiting writes to it, using filesystems that can detect corruption (btrfs together with mainline kernel or with Xenial now also zfs in a very convenient way) and check warranty parameters instead of pseudo specs like the above. Samsung EVOs have 60 months compared to 24 with the SanDisks so I already know which to buy. But my use cases are different. And by using superiour filesystems write amplification remains low.

 

We started testing one rsyslogd central log host in May: H3 board with kernel 4.4, 64 GB EVO and btrfs parameters as follows: 'commit=300,compress-force=zlib,discard'. Holds now approx. 400 GB log data, might store another 400 GB until the first erase cycle has to happen :)

Share this post


Link to post
Share on other sites

Some benchmarks on PC2 using ubuntu_xfce 1.0.0 image from XunLong

Card used is Samsung 32GB EVO+


root@Orangepi:/home/orangepi# iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
    Iozone: Performance Test of File I/O
            Version $Revision: 3.429 $
        Compiled for 64 bit mode.
        Build: linux

    Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                 Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                 Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                 Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                 Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                 Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                 Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                 Vangel Bojaxhi, Ben England, Vikentsi Lapa.

    Run began: Fri Dec  9 17:36:07 2016

    Include fsync in write timing
    O_DIRECT feature enabled
    Auto Mode
    File size set to 102400 kB
    Record Size 4 kB
    Record Size 16 kB
    Record Size 512 kB
    Record Size 1024 kB
    Record Size 16384 kB
    Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
    Output is in kBytes/sec
    Time Resolution = 0.000001 seconds.
    Processor cache size set to 1024 kBytes.
    Processor cache line size set to 32 bytes.
    File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4     1489     2265     7510     7502     7476     2884                                                          
          102400      16     9689    11278    14578    14538    14423    11931                                                          
          102400     512    17672    18220    23048    23049    23049    18416                                                          
          102400    1024    17325    19293    23017    23031    23016    21126                                                          
          102400   16384    15370    17377    23040    23042    23042    18059                                                          

iozone test complete.

As suspected, H5 sd controller is still shit, just like it was in h3

Share this post


Link to post
Share on other sites

As suspected, H5 sd controller is still shit, just like it was in h3

 

Wrong: http://linux-sunxi.org/H5#Overview

 

It's a board design 'problem' or let's better say design decision. Without providing the abillity to switch voltage (3.3V and also 1.8V) no faster modes are possible. And once such a board exists (maybe Olimex A64 board allows that) drivers have to be written.

 

So the sequential read/write limitation of approx 23 MB/s is due to board design decisions and not the SoC's controller. BTW: I always have a good laugh when people complain about literally $anything since we're talking about devices for $20 or even less here.

 

Did you check cpufreq while running the test? Governor set to performance before to ensure running on at least 1008 MHz all the time?

Share this post


Link to post
Share on other sites

Wrong: http://linux-sunxi.org/H5#Overview

 

It's a board design 'problem' or let's better say design decision. Without providing the abillity to switch voltage (3.3V and also 1.8V) no faster modes are possible. And once such a board exists (maybe Olimex A64 board allows that) drivers have to be written.

 

So the sequential read/write limitation of approx 23 MB/s is due to board design decisions and not the SoC's controller. BTW: I always have a good laugh when people complain about literally $anything since we're talking about devices for $20 or even less here.

 

Did you check cpufreq while running the test? Governor set to performance before to ensure running on at least 1008 MHz all the time?

 

I used nmon and cpufreq panel and frequency was always at 1008Mhz.

 

As for the whole sd card controller; isn't the actual controller in the soc still limited to 25MB/s for SD cards ? Or can you use mmc controller for sd cards as well (obviously that would boost sequential performance given how that is faster) 

Share this post


Link to post
Share on other sites
I always have a good laugh when people complain about literally $anything since we're talking about devices for $20 or even less here.

 

 

Well, complaining makes progress, otherwise we would still be stuck with CRAPPY raspberries for 35$. Single core arm6, shit IO, shit everything else.

It really doesn't hurt to be critical, regardless of price. Things like better sd controller don't exactly add that much more cost.

Share this post


Link to post
Share on other sites

And dmesg will tell you the reason: Filesystem corrupted due to using faulty or counterfeit SD card and kernel trying to save your FS by setting it to read-only :)

 

Call dmesg and watch for ext4/corruption, then follow the checking procedure on another system as outlined in our documentation: http://docs.armbian.com/User-Guide_Getting-Started/#how-to-prepare-a-sd-card or throw the card away.

 

Thanks but I tested SD cart with H2testw and no issue detected. output log:

 

 

Warning: Only 14786 of 14787 MByte tested.
Test finished without errors.
You can now delete the test files *.h2w or verify them again.
Writing speed: 8.07 MByte/s
Reading speed: 20.6 MByte/s
H2testw v1.4

and demsg log after filesystem crash :

 

[16631.756436] systemd-logind[25616]: New seat seat0.
[16631.809140] systemd-logind[25616]: Failed to start user service: Unknown unit: user@0.service
[16631.852567] systemd-logind[25616]: New session 1450 of user root.
[16980.635700] systemd-logind[25616]: New session 1482 of user root.
[22881.545071] systemd-logind[25616]: New session 2003 of user root.
[23419.144228] mmcblk0: mmc_blk_err_check: general error sending status command, card status 0x80e00
[23419.146272] mmcblk0: retrying write for general error
[23419.148499] mmcblk0: mmc_blk_cmd_recovery: general error sending stop or status command, stop cmd response 0x0, card status 0x80d00
[23419.151556] mmcblk0: timed out sending r/w cmd command, card status 0x80d00
[23419.154673] mmcblk0: not retrying timeout
[23419.157856] end_request: I/O error, dev mmcblk0, sector 23929992
[23423.502223] mmcblk0: mmc_blk_cmd_recovery: general error sending stop or status command, stop cmd response 0x0, card status 0x80d00
[23423.507197] mmcblk0: timed out sending r/w cmd command, card status 0x80d00
[23423.512222] mmcblk0: not retrying timeout
[23423.517266] end_request: I/O error, dev mmcblk0, sector 3936304
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936312
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936320
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936328
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936336
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936344
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936352
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936360
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936368
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936376
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936384
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936392
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936400
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936408
[23423.522180] end_request: I/O error, dev mmcblk0, sector 3936416
[23423.598498] Aborting journal on device mmcblk0p1-8.
[23423.598852] mmcblk0: mmc_blk_cmd_recovery: general error sending stop or status command, stop cmd response 0x0, card status 0x80d00
[23423.598945] mmcblk0: timed out sending r/w cmd command, card status 0x80d00
[23423.598964] mmcblk0: not retrying timeout
[23423.598985] end_request: I/O error, dev mmcblk0, sector 5197824
[23423.599008] end_request: I/O error, dev mmcblk0, sector 5197832
[23423.599030] end_request: I/O error, dev mmcblk0, sector 5197840
[23423.599050] end_request: I/O error, dev mmcblk0, sector 5197848
[23423.599070] end_request: I/O error, dev mmcblk0, sector 5197856
[23423.599090] end_request: I/O error, dev mmcblk0, sector 5197864
[23423.599110] end_request: I/O error, dev mmcblk0, sector 5197872
[23423.599130] end_request: I/O error, dev mmcblk0, sector 5197880
[23423.599149] end_request: I/O error, dev mmcblk0, sector 5197888
[23423.599169] end_request: I/O error, dev mmcblk0, sector 5197896
[23423.599189] end_request: I/O error, dev mmcblk0, sector 5197904
[23423.599208] end_request: I/O error, dev mmcblk0, sector 5197912
[23423.599228] end_request: I/O error, dev mmcblk0, sector 5197920
[23423.599248] end_request: I/O error, dev mmcblk0, sector 5197928
[23423.599267] end_request: I/O error, dev mmcblk0, sector 5197936
[23423.599286] end_request: I/O error, dev mmcblk0, sector 5197944
[23423.599382] Buffer I/O error on device mmcblk0p1, logical block 649472
[23423.599446] Buffer I/O error on device mmcblk0p1, logical block 649473
[23423.599507] Buffer I/O error on device mmcblk0p1, logical block 649474
[23423.599554] Buffer I/O error on device mmcblk0p1, logical block 649475
[23423.599600] Buffer I/O error on device mmcblk0p1, logical block 649476
[23423.599643] Buffer I/O error on device mmcblk0p1, logical block 649477
[23423.599689] Buffer I/O error on device mmcblk0p1, logical block 649478
[23423.599728] Buffer I/O error on device mmcblk0p1, logical block 649479
[23423.599767] Buffer I/O error on device mmcblk0p1, logical block 649480
[23423.599806] Buffer I/O error on device mmcblk0p1, logical block 649481
[23423.599845] Buffer I/O error on device mmcblk0p1, logical block 649482
[23423.599884] Buffer I/O error on device mmcblk0p1, logical block 649483
[23423.599923] Buffer I/O error on device mmcblk0p1, logical block 649484
[23423.599962] Buffer I/O error on device mmcblk0p1, logical block 649485
[23423.600002] Buffer I/O error on device mmcblk0p1, logical block 649486
[23423.600028] Buffer I/O error on device mmcblk0p1, logical block 649487
[23423.600062] EXT4-fs warning (device mmcblk0p1): ext4_end_bio:249: I/O error writing to inode 386396 (offset 1048576 size 65536 starting block 649728)
[23423.696785] mmcblk0: mmc_blk_cmd_recovery: general error sending stop or status command, stop cmd response 0x0, card status 0x80d00
[23423.826926] mmcblk0: timed out sending r/w cmd command, card status 0x80d00
[23423.832883] mmcblk0: not retrying timeout
[23423.838534] end_request: I/O error, dev mmcblk0, sector 3825800
[23423.842833] Buffer I/O error on device mmcblk0p1, logical block 477969
[23423.842833] lost page write due to I/O error on mmcblk0p1
[23423.857179] JBD2: Error -5 detected when updating journal superblock for mmcblk0p1-8.
[23423.857591] mmcblk0: mmc_blk_cmd_recovery: general error sending stop or status command, stop cmd response 0x0, card status 0x80d00
[23423.857678] mmcblk0: timed out sending r/w cmd command, card status 0x80d00
[23423.857696] mmcblk0: not retrying timeout
[23423.857716] end_request: I/O error, dev mmcblk0, sector 2048
[23423.857745] Buffer I/O error on device mmcblk0p1, logical block 0
[23423.857763] lost page write due to I/O error on mmcblk0p1
[23423.857822] EXT4-fs error (device mmcblk0p1): ext4_journal_start_sb:328: Detected aborted journal
[23423.857853] EXT4-fs (mmcblk0p1): Remounting filesystem read-only
[23423.857878] ------------[ cut here ]------------
[23423.857917] WARNING: at fs/buffer.c:1112 mark_buffer_dirty+0x40/0xe4()
[23423.857933] Modules linked in: 8188eu bnep bluetooth 8189es [last unloaded: 8188eu]
[23423.858016] [<c0016a20>] (unwind_backtrace+0x0/0xe8) from [<c0695870>] (dump_stack+0x20/0x24)
[23423.858060] [<c0695870>] (dump_stack+0x20/0x24) from [<c0029720>] (warn_slowpath_common+0x5c/0x74)
[23423.858090] mmcblk0: mmc_blk_cmd_recovery: general error sending stop or status command, stop cmd response 0x0, card status 0x80d00
[23423.858130] [<c0029720>] (warn_slowpath_common+0x5c/0x74) from [<c00297f4>] (warn_slowpath_null+0x2c/0x34)
[23423.858172] [<c00297f4>] (warn_slowpath_null+0x2c/0x34) from [<c0149b6c>] (mark_buffer_dirty+0x40/0xe4)
[23423.858198] mmcblk0: timed out sending r/w cmd command, card status 0x80d00
[23423.858238] [<c0149b6c>] (mark_buffer_dirty+0x40/0xe4) from [<c01a21e8>] (ext4_commit_super+0x1a4/0x218)
[23423.858258] mmcblk0: not retrying timeout
[23423.858287] [<c01a21e8>] (ext4_commit_super+0x1a4/0x218) from [<c01a2284>] (save_error_info+0x28/0x2c)
[23423.858310] end_request: I/O error, dev mmcblk0, sector 2048
[23423.858341] [<c01a2284>] (save_error_info+0x28/0x2c) from [<c01a2974>] (__ext4_abort+0xbc/0xf0)
[23423.858361] Buffer I/O error on device mmcblk0p1, logical block 0
[23423.858394] [<c01a2974>] (__ext4_abort+0xbc/0xf0) from [<c01a2acc>] (ext4_journal_start_sb+0x124/0x1a4)
[23423.858414] lost page write due to I/O error on mmcblk0p1
[23423.858446] [<c01a2acc>] (ext4_journal_start_sb+0x124/0x1a4) from [<c018ce68>] (ext4_dirty_inode+0x24/0x50)
[23423.858474] EXT4-fs error (device mmcblk0p1): ext4_journal_start_sb:328: [<c018ce68>] (ext4_dirty_inode+0x24/0x50) from [<c01417d0>] (__mark_inode_dirty+0x3c/0x1c8)
[23423.858521] Detected aborted journal[<c01417d0>] (__mark_inode_dirty+0x3c/0x1c8) from [<c0134d88>] (file_update_time+0xf8/0x114)

[23423.858600] [<c0134d88>] (file_update_time+0xf8/0x114) from [<c00dccc0>] (__generic_file_aio_write+0x2f0/0x434)
[23423.858666] [<c00dccc0>] (__generic_file_aio_write+0x2f0/0x434) from [<c00dce74>] (generic_file_aio_write+0x70/0xd8)
[23423.858722] [<c00dce74>] (generic_file_aio_write+0x70/0xd8) from [<c01832ec>] (ext4_file_write+0x188/0x2c0)
[23423.858784] [<c01832ec>] (ext4_file_write+0x188/0x2c0) from [<c011d8cc>] (do_sync_write+0xac/0xe8)
[23423.858826] [<c011d8cc>] (do_sync_write+0xac/0xe8) from [<c011dfc4>] (vfs_write+0xc0/0x16c)
[23423.858863] [<c011dfc4>] (vfs_write+0xc0/0x16c) from [<c011e360>] (sys_write+0x4c/0x78)
[23423.858902] [<c011e360>] (sys_write+0x4c/0x78) from [<c000df60>] (ret_fast_syscall+0x0/0x30)
[23423.858928] ---[ end trace 7415e919e2cb81fd ]---
[23423.859219] mmcblk0: mmc_blk_cmd_recovery: general error sending stop or status command, stop cmd response 0x0, card status 0x80d00
[23423.859293] mmcblk0: timed out sending r/w cmd command, card status 0x80d00
[23423.859312] mmcblk0: not retrying timeout
[23423.859332] end_request: I/O error, dev mmcblk0, sector 2048
[23423.859361] Buffer I/O error on device mmcblk0p1, logical block 0
[23423.859379] lost page write due to I/O error on mmcblk0p1
[23448.500734] mmcblk0: mmc_blk_cmd_recovery: general error sending stop or status command, stop cmd response 0x0, card status 0x80d00
[23448.515357] mmcblk0: timed out sending r/w cmd command, card status 0x80d00
[23448.530133] mmcblk0: not retrying timeout
[23448.544804] end_request: I/O error, dev mmcblk0, sector 4199624
[23448.554755] Buffer I/O error on device mmcblk0p1, logical block 524697
[23448.554755] lost page write due to I/O error on mmcblk0p1
[23448.590706] mmcblk0: mmc_blk_cmd_recovery: general error sending stop or status command, stop cmd response 0x0, card status 0x80d00
[23448.607128] mmcblk0: timed out sending r/w cmd command, card status 0x80d00
[23448.623677] mmcblk0: not retrying timeout
[23448.640295] end_request: I/O error, dev mmcblk0, sector 4201016
[23448.650254] Buffer I/O error on device mmcblk0p1, logical block 524871
[23448.650254] lost page write due to I/O error on mmcblk0p1
[23474.133018] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23474.148595] mmcblk0: not retrying timeout
[23474.163811] end_request: I/O error, dev mmcblk0, sector 12875400
[23474.514701] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23474.529646] mmcblk0: not retrying timeout
[23474.544383] end_request: I/O error, dev mmcblk0, sector 12875400
[23534.270797] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23534.284202] mmcblk0: not retrying timeout
[23534.297107] end_request: I/O error, dev mmcblk0, sector 12875400
[23553.805118] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23553.818195] mmcblk0: not retrying timeout
[23553.830636] end_request: I/O error, dev mmcblk0, sector 12875400
[23594.555994] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23594.567161] mmcblk0: not retrying timeout
[23594.578194] end_request: I/O error, dev mmcblk0, sector 12875400
[23652.753856] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23652.763856] mmcblk0: not retrying timeout
[23652.773565] end_request: I/O error, dev mmcblk0, sector 12875400
[23653.505575] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23653.515097] mmcblk0: not retrying timeout
[23653.524382] end_request: I/O error, dev mmcblk0, sector 12875400
[23660.556280] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23660.565967] mmcblk0: not retrying timeout
[23660.575746] end_request: I/O error, dev mmcblk0, sector 12875400
[23691.298438] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23691.308043] mmcblk0: not retrying timeout
[23691.317425] end_request: I/O error, dev mmcblk0, sector 12875400
[23709.104535] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23709.113909] mmcblk0: not retrying timeout
[23709.123413] end_request: I/O error, dev mmcblk0, sector 12875400
[23713.866279] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23713.876803] mmcblk0: not retrying timeout
[23713.887318] end_request: I/O error, dev mmcblk0, sector 12875400
[23714.233959] mmcblk0: timed out sending r/w cmd command, card status 0x80b00
[23714.244696] mmcblk0: not retrying timeout
[23714.255197] end_request: I/O error, dev mmcblk0, sector 12875400

 

 

Please Help  :(

Edited by zador.blood.stained
Added spoiler tags

Share this post


Link to post
Share on other sites

Please Help  :(

What do you expect to hear? Some magic spells that will turn a crappy SD card into a Samsung Evo Plus? Sorry, this probably won't happen. If sequential write/read tests don't show you any errors it doesn't mean that this card will behave perfectly in other scenarios - i.e. combined reads and writes of blocks of different sizes.

 

There is only one reliable solution:

 

 

post-480-0-37098900-1481649542_thumb.png

 

 

Share this post


Link to post
Share on other sites

I just read this article...Any relevance for OPi?

 

It talks about tweaking settings to get much better memory performance. To me good SD cards cost a lot compared with OPi prices themselves, and have usability and reliability issues.

 

Can "slower" SD cards be used with some tricks? Perhaps with onboard SPI NOR bootloader, with or without netbooting?

 

http://hackaday.com/2016/12/29/improving-raspberry-pi-disk-performance/

Share this post


Link to post
Share on other sites

Any optimizations are a tradeoff between read/write speeds (sequential and random, observed by application, not actual SD speeds since they don't change), CPU time utilization and RAM utilization (and reliability if you don't have an UPS or are using unstable kernel drivers), so different things can be tweaked to achieve different performance improvements in different tasks.

 

Regarding reliability - it is more a statistics problem, while minimizing writes to the card may help in theory, you can't predict when or how the SD card will malfunction (this applies both to "good" and "bad" ones). You can get anything depending on your luck - completely read-only card, partially read-only card, silent data corruption that won't be noticed until it's too late, or a completely "bricked" (unreadable) device.

 

If we are not talking about "usual" use cases, there are different possibilities depending on your usage scenarios:

  • Read-only rootfs on SD
  • Rootfs on local device (HDD, SSD, USB flash)
  • Rootfs on NFS

SPI flash can help with eliminating the SD as a boot device in last 2 scenarios.

 

Edit: Well, measuring SD card "performance" with "dd"... you may want to read the first post of this thread (and links at the end of it) before @tkaiser says that he thinks about this article

Share this post


Link to post
Share on other sites

Samsung Evo 64GB microsd. Brand new, just run through crystaldiskmark and h2testw

Orange pi PC

Linux orangepipc 3.4.112-sun8i #10 SMP PREEMPT Sun Oct 23 16:06:55 CEST 2016 armv7l GNU/Linux

root@orangepipc:~# iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.429 $
                Compiled for 32 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                     Vangel Bojaxhi, Ben England, Vikentsi Lapa.

        Run began: Mon Jan 16 22:16:47 2017

        Include fsync in write timing
        O_DIRECT feature enabled
        Auto Mode
        File size set to 102400 kB
        Record Size 4 kB
        Record Size 16 kB
        Record Size 512 kB
        Record Size 1024 kB
        Record Size 16384 kB
        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                                                                   
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  frer                                                  ead
          102400       4     1004      689     4453     4434     4381      839
          102400      16     2827     3298    11694    11903    11507     2441
          102400     512    11333    13769    21382    21393    21106    12297
          102400    1024    13565    13151    22074    22067    21998    11406
          102400   16384    13517    14777    22850    22838    22844    11913

iozone test complete.

It seems samsung have crippled the later evos!

it's showing:

date: 10/2016

second run with performance governor

 

 

 

root@orangepipc:~# iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.429 $
                Compiled for 32 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                     Vangel Bojaxhi, Ben England, Vikentsi Lapa.

        Run began: Mon Jan 16 22:43:53 2017

        Include fsync in write timing
        O_DIRECT feature enabled
        Auto Mode
        File size set to 102400 kB
        Record Size 4 kB
        Record Size 16 kB
        Record Size 512 kB
        Record Size 1024 kB
        Record Size 16384 kB
        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4      590      732     4510     4394     4251      805
          102400      16     2927     3948    11782    11798    11642     2904
          102400     512    12633    14333    21418    21406    21334    12595
          102400    1024    14782    14092    21998    21981    21950    12414
          102400   16384    15195    12667    22833    22845    22839    13129

iozone test complete.

 

 

Share this post


Link to post
Share on other sites

32GB sandisk ultra

root@orangepipcplus:~# iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.429 $
                Compiled for 32 bit mode.
                Build: linux
        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4     2087     3278     7660     7666     6585     2525
          102400      16     2193     2680    15009    15004    14165     4382
          102400     512     7311    10130    21708    21690    21675     9219
          102400    1024    11290    16439    22156    22163    22137    13792
          102400   16384    17462    18827    22777    22779    22776    18479

Share this post


Link to post
Share on other sites

 

 

Samsung Evo 64GB microsd. Brand new, just run through crystaldiskmark and h2testw

Orange pi PC

Linux orangepipc 3.4.112-sun8i #10 SMP PREEMPT Sun Oct 23 16:06:55 CEST 2016 armv7l GNU/Linux

root@orangepipc:~# iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.429 $
                Compiled for 32 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                     Vangel Bojaxhi, Ben England, Vikentsi Lapa.

        Run began: Mon Jan 16 22:16:47 2017

        Include fsync in write timing
        O_DIRECT feature enabled
        Auto Mode
        File size set to 102400 kB
        Record Size 4 kB
        Record Size 16 kB
        Record Size 512 kB
        Record Size 1024 kB
        Record Size 16384 kB
        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                                                                   
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  frer                                                  ead
          102400       4     1004      689     4453     4434     4381      839
          102400      16     2827     3298    11694    11903    11507     2441
          102400     512    11333    13769    21382    21393    21106    12297
          102400    1024    13565    13151    22074    22067    21998    11406
          102400   16384    13517    14777    22850    22838    22844    11913

iozone test complete.

It seems samsung have crippled the later evos!

it's showing:

date: 10/2016

second run with performance governor

 

 

 

root@orangepipc:~# iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.429 $
                Compiled for 32 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                     Vangel Bojaxhi, Ben England, Vikentsi Lapa.

        Run began: Mon Jan 16 22:43:53 2017

        Include fsync in write timing
        O_DIRECT feature enabled
        Auto Mode
        File size set to 102400 kB
        Record Size 4 kB
        Record Size 16 kB
        Record Size 512 kB
        Record Size 1024 kB
        Record Size 16384 kB
        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4      590      732     4510     4394     4251      805
          102400      16     2927     3948    11782    11798    11642     2904
          102400     512    12633    14333    21418    21406    21334    12595
          102400    1024    14782    14092    21998    21981    21950    12414
          102400   16384    15195    12667    22833    22845    22839    13129

iozone test complete.

 

 

 

 

Maybe you got a fake one.

Edited by tkaiser
Hide useless fullquote

Share this post


Link to post
Share on other sites

Maybe you got a fake one.

 

I thought that too, but it looks real to me. It has the EU model number, made in Philippines, the print quality on the card and packet are top notch

And it passes h2testw

Share this post


Link to post
Share on other sites

I thought that too, but it looks real to me. It has the EU model number, made in Philippines, the print quality on the card and packet are top notch and it was from Amazon (not marketplace)

And it passes h2testw

Sadly all of this can be faked just as well. And amazon is no way a guarantee to get a real one sadly.

Share this post


Link to post
Share on other sites

PNY SDXC 64GB UHS-I - rated 50MB/s

        Include fsync in write timing
        O_DIRECT feature enabled
        Auto Mode
        File size set to 102400 kB
        Record Size 4 kB
        Record Size 16 kB
        Record Size 512 kB
        Record Size 1024 kB
        Record Size 16384 kB
        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                                                                                                                  
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  frer                                                                                                 ead
          102400       4     1182     1182     7421     7419     3163     1029
          102400      16     2577     3512    14447    14524     8940     3230
          102400     512    11793    16506    21625    21623    20965    17871
          102400    1024    15319    16764    22187    22184    21804    18210
          102400   16384    15889    17232    22911    22908    22886    18651

Share this post


Link to post
Share on other sites

Good news! The SDA (SD Association) responsible for specifications around SD/TF cards released SD 5.1 specifications just recently which define the first time performance numbers for random read/write performance (just check the link and the press release from Nov 2016)! SD cards that want to be compliant to A1 performance standard must exceed at least random 1500/500 IOPS (read/write) with 4K blocksize.

 

If you compare with the Samsung EVO numbers from this thread then 500 IOPS 4k random write look somewhat laughable but the good thing is that there's now a standard that defines random IO performance and not only sequential performance as it's today (the so called 'speed class'). So while Samsung today might produce EVOs that still fulfill 'class 10' performance criteria (10 MB/s sequential write performance) they could decrease random IO performance by factor 100 and still would meet 'class 10' criteria.

 

It's long known that especially Android really sucks when running off slow storage (slow as in 'slow random write performance') which is why most Android devices are equipped with fast eMMC or older ones with raw NAND (showing high random IO numbers). Starting with Android 7.0/6.0 a concept called 'Adoptable Storage' has been introduced that tries to integrate storage on an additional SD card with the device's fast internal storage. Android does a performance test of the SD card in question before allowing 'adopted storage' and this is where this 'A1 performance' profile originates from.

 

As most use cases for devices running with Armbian also benefit from high random IO performance you should watch out from now on for SD cards that show the A1 logo as can be seen here: http://heise.de/-3612646

 

But please keep in mind that meeting A1 specs means only 500/1500 IOPS (4k write/read). The 'DSP Memory' SD card shown in the Heise link is a pretty good example for a company known not for product quality but brilliant marketing (they grew with relabeled DRAM sold to clueless Apple users). This SD card (most probably some cheap noname chips with Phison or Silicon Motion controller) claiming A1 conformance slightly exceeds the A1 criteria and the 32GB variant will be sold for 23€ while at the same time you can get a Samsung EVO+ that costs only half as much and is multiple times more performant (combining quality NAND flash dies with quality controller!): http://geizhals.de/?cat=sm_sdhc&asuch=EVO&asd=on&v=e&hloc=at&hloc=de&hloc=pl&hloc=uk&hloc=eu&filter=aktualisieren&xf=1032_Samsung~2456_10~298_microSDHC~298_microSDXC~307_32~5950_2&sort=p

 

If you compare with performance numbers from 1st post of this thread at least my Samsung EVO/EVO+ with 32 and 64 GB all show ~7500 read and +3000 write IOPS at 4k -- that's 5 times the A1 read performance and 6 times the write performance!

Share this post


Link to post
Share on other sites

Are evos still good at random performance ? can anyone test newly purchased cards ?

 

While I highly appreciate that you don't do a full quote of mostly unrelated stuff someone else wrote before as usual why not just looking a few posts above? There are numbers of 'brand new' EVOs bought just recently. Maybe it's important to keep in mind what I wrote in the very first post of this thread:

 

I used three of these cards the first time and they all show identical behaviour especially regarding writes with small record sizes: pretty slow in the beginning and getting faster over time (the Samsung Pro for example started with only 1400KB/s 4K writes and 3 runs later the same test showed 3270KB/s...

Share this post


Link to post
Share on other sites

For comparison, a 32GB Sandisk Extreme Pro 95MB/s MicroSD, a 32GB Samsung EVO+ MicroSD and the Orange Pi PC Plus eMMC speeds:


(using 'iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2')

 

32GB Sandisk Extreme Pro 95MB/s MicroSD

                                                              random    random     bkwd    record    stride                                                                                                                                  
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  frer                                                                                                 ead
          102400       4     2891     2973     8610     8609     8378     4764
          102400      16     7914     8058    16077    16097    16064    11118
          102400     512    20219    20331    21772    21724    21764    19582
          102400    1024    20450    20479    22279    22277    22269    20117
          102400   16384    21585    21620    22957    22967    22956    21358

 

32GB Samsung EVO+ MicroSD

                                                              random    random     bkwd    record    stride                                                                                                                                  
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  frer                                                                                                 ead
          102400       4     1510     2426     7017     7380     7496     3326
          102400      16    10629    11033    14575    14563    14579    11048
          102400     512    20020    20167    21592    21594    21584    20095
          102400    1024    20283    20370    22130    22129    22128    20340
          102400   16384    21053    21198    22906    22905    22903    21190

 

Orange Pi PC Plus eMMC

                                                              random    random     bkwd    record    stride                                                                                                                                  
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  frer                                                                                                 ead
          102400       4     6204     6558    15817    15812    13358     6422
          102400      16    21499    22650    38919    38938    34894    21409
          102400     512    35602    35566    64903    64908    64912    34244
          102400    1024    36147    36359    69756    70083    69976    35780
          102400   16384    38198    38697    78658    78768    78775    38495

 

MicroSD clearly bottlenecks (particularly obvious with one of the fastest UHS-1 MicroSD cards in the Extreme Pro) and eMMC is definitely the better option for OS by the looks.

Edited by tkaiser
Formatted for better reading

Share this post


Link to post
Share on other sites
10 hours ago, duyfken said:

32GB Sandisk Extreme Pro 95MB/s MicroSD

 

Thank you for the numbers, though both 4K and 16K Sandisk numbers look very suspicious (random IO faster than sequential? Hard to believe since this would mean a crippled controller 'optimizing' something terribly wrong). In case you have the time it would be great if you could re-test, ensure that you've chosen performance governor and provide 'armbianmonitor -u' output or at least the MMC info block as can be seen here: https://forum.armbian.com/index.php?/topic/990-testers-wanted-sd-card-performance/&do=findComment&comment=8172

 

There a SanDisk Extreme Plus with 32GB was tested. Since I got a 16GB SanDisk Extreme Plus 2 days ago for just €17 here are the numbers (debug output -- you might want to search for '### mmc0' in there) that look quite comparable to the above from last year:

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4     3331     3305     7823     7834     6722     2905
          102400      16     8308     8672    16278    16290    16290     7830
          102400     512    20892    22040    22913    22951    22949    22038
          102400    1024    22346    22455    23074    22594    23071    22127
          102400   16384    22206    22633    23289    23290    23289    22624

This was the fourth run and numbers remained stable (I don't trust in single 'fire and forget' benchmark runs and numbers with factory new Samsung cards from last year prove this attempt to be right since they seem to do some sort of initial 'calibration' at least they get significantly faster after some short time of use -- see post #1 from this thread for details if interested)

 

As you already noticed the maximum throughput is limited to 22/23 MB/s anyway (this is an SDIO limitation unfortunately most boards Armbian supports share, one known exception is the ASUS Tinkerboard now where +50MB/s are reported with capable SD cards)

 

10 hours ago, duyfken said:

Orange Pi PC Plus eMMC

 

O yes, Xunlong seems to be the only vendor using pretty fast eMMC also on the 'small' or better say inexpensive boards (according to Aliexpress pictures same eMMC is also used on 'Zero Plus 2' now). FriendlyELEC and SinoVoip for example chose both pretty slow 8GB eMMC modules on their NanoPi and BananaPi (at least FriendlyELEC responded and said they will do some investigations so maybe we get faster eMMC here too)

Share this post


Link to post
Share on other sites
1 hour ago, tkaiser said:

 

Thank you for the numbers, though both 4K and 16K numbers look very suspicious (random IO faster than sequential? Hard to believe since this would mean a crippled controller 'optimizing' something terribly wrong). In case you have the time it would be great if you could re-test, ensure that you've chosen performance governor and provide 'armbianmonitor -u' output or at least the MMC info block as can be seen here: https://forum.armbian.com/index.php?/topic/990-testers-wanted-sd-card-performance/&do=findComment&comment=8172

 

 

 

which results are you looking at ? They look fine to me (bigger the block size, faster performance).

Share this post


Link to post
Share on other sites
Quote

This was the fourth run and numbers remained stable (I don't trust in single 'fire and forget' benchmark runs and numbers with factory new Samsung cards from last year prove this attempt to be right since they seem to do some sort of initial 'calibration' at least they get significantly faster after some short time of use -- see post #1 from this thread for details if interested)

 

that seems very interesting. I guess there is a chance sd card is doing some internal garbage collection (old data from burn in testing ?) which can obviously yield worse results at the beginning.

Share this post


Link to post
Share on other sites
46 minutes ago, hojnikb said:

which results are you looking at ? They look fine to me (bigger the block size, faster performance).


First 2 lines of SanDisk numbers. Random writes being 60 percent faster than sequential at 4K blocksize is simply strange:

          102400       4     2891     2973     8610     8609     8378     4764
          102400      16     7914     8058    16077    16097    16064    11118
19 minutes ago, hojnikb said:

I guess there is a chance sd card is doing some internal garbage collection (old data from burn in testing ?) which can obviously yield worse results at the beginning.

 

Whatever it is... it happened above also, just look at the 4K results for the EVO:

          102400       4     1510     2426     7017     7380     7496     3326

1.5MB/s when writing the 1st 100 MB, then 2.4MB/s when re-writing 100 MB and finally 3.3MB/s when writing the next 100 MB with a random pattern. Doing benchmarks is not just producing numbers without meaning but interpreting/understanding results (which leads in many or even most cases to the tester throwing away results and starting again).

 

And now please let's remember that this here is not a chatroom but a research/educational thread trying to collect valuable information to be used by people coming from anywhere. This whole threads gets useless when flooded with posts!

Share this post


Link to post
Share on other sites
10 10