Jump to content

tkaiser

Members
  • Posts

    5462
  • Joined

Reputation Activity

  1. Like
    tkaiser got a reaction from NicoD in rock-5b with vendor u-boot & vendor kernel; PR and test images   
    https://github.com/ThomasKaiser/Knowledge/blob/master/articles/Quick_Preview_of_ROCK_5B.md#important-insights-and-suggested-optimisations
     
    BTW: The device is called Rock 5B as such congratulations on the subforum name...
  2. Like
    tkaiser got a reaction from lanefu in rock-5b with vendor u-boot & vendor kernel; PR and test images   
    https://github.com/ThomasKaiser/Knowledge/blob/master/articles/Quick_Preview_of_ROCK_5B.md#important-insights-and-suggested-optimisations
     
    BTW: The device is called Rock 5B as such congratulations on the subforum name...
  3. Like
    tkaiser got a reaction from balbes150 in Firefly Station M3 (rk3588s)   
    It's not about mainline kernel vs. BSP kernel but the latter having settings that might fit for Android (use cases like watching video, playing games) but not for Linux.
     
    You seem to be using these defaults without questioning them. Rockchip's BSP kernel defaults to powersave for ASPM (Active State Power Management) which of course negatively affects NVMe performance. As such you need to either eliminate CONFIG_PCIEASPM_POWERSAVE=y from kernel config or need to execute somewhere after booting:
    echo default >/sys/module/pcie_aspm/parameters/policy  
    Also the BSP kernel when the dmc/dfi device-tree nodes are enabled (seems to be the case with your RK3588 kernel fork since the 7-zip scores you and @blondu are sharing are below 14800 while they could be around 16500) defaults to dmc_ondemand governor which can be changed by doing this:
    echo performance >/sys/class/devfreq/dmc/governor (similar way as I've done this for RK3399 years ago: https://github.com/armbian/build/blob/fdf73a025ba56124523baefaf705792b74170fb8/packages/bsp/common/usr/lib/armbian/armbian-hardware-optimization#L241-L244 )
     
    And this here:
    prefix="/sys/devices/system/cpu" CPUFreqPolicies=($(ls -d ${prefix}/cpufreq/policy? | sed 's/freq\/policy//')) if [ ${#CPUFreqPolicies[@]} -eq 1 -a -d "${prefix}/cpufreq" ]; then # if there's just a single cpufreq policy ondemand sysfs entries differ CPUFreqPolicies=${prefix} fi for i in ${CPUFreqPolicies[@]}; do affected_cpu=$(tr -d -c '[:digit:]' <<< ${i}) echo ondemand >${prefix}/cpu${affected_cpu:-0}/cpufreq/scaling_governor echo 1 >${i}/cpufreq/ondemand/io_is_busy echo 25 >${i}/cpufreq/ondemand/up_threshold echo 10 >${i}/cpufreq/ondemand/sampling_down_factor echo 200000 >${i}/cpufreq/ondemand/sampling_rate done  
    is the exact replacement for lines 81-89 in armbian-hardware-optimization: https://github.com/armbian/build/blob/fdf73a025ba56124523baefaf705792b74170fb8/packages/bsp/common/usr/lib/armbian/armbian-hardware-optimization#L81-L89
     
    Without this I/O performance with Armbian sucks on a variety of boards for example ODROID N2/N2+, VIM3 or now the RK3588/RK3588S based boards. Unfortunately @lanefu seems to be way too biased or limited in his thinking to understand this when he creates bizarre tickets that are rotting around somewhere: https://armbian.atlassian.net/browse/AR-1262
     
    I really don't care whether these fixes will be incorporated into Armbian. But if you benchmark stuff the settings should be adjusted accordingly. And we (we as a broader community – not this place here) already know how ASPM settings negatively affect performance of PCIe devices (like for example NVMe SSDs): https://forum.radxa.com/t/rock-5b-debug-party-invitation/10483/86?u=tkaiser, we know which role io_is_busy has and what the benefits and drawbacks of the chosen dmc governor are.  
     
    A quality NVMe SSD even when just connected with a single Gen2 lane should always outperform any SATA SSD if it's about what really matters: random I/O. If the SSD is cheap garbage or the settings are garbage it might look differently.
  4. Like
    tkaiser got a reaction from Myron in The partition is not resized to full SD card size.   
    So this is a 64GB Samsung. EVO, EVO+, Pro or Pro+? Anyway there's still the 1% reserve present and this is not 'the average' SD card but Samsungs all contain rather good controllers and NAND dies.
     
    Regarding 'reserve sectors' and overprovisioning: If a card claims it's n bytes in capacity it has internally a larger capacity. This is used as reserve (if the controller detects bad sectors, then reserve sectors are mapped in) and to allow somewhat ok-ish write performance when the card gets full. On flash media you can't overwrite directly, it's always a very time consuming read/delete/write cycle, the number a flash cell can be written to is determined by the count of program/erase (P/E) cycles it is designed for, the controller has to take care of this so that all flash cells wear out equally (wear leveling).
     
    Since the controller has no idea which sectors contain real data and which not (there's no TRIM support for SD cards) as soon as you completely fill the card once (all space partitioned) from now on the controller considers every sector containing useful data (even if you deleted the data in the meantime -- since there's no TRIM support the controller doesn't know what's empty or not, from now on the whole capacity is considered in use). Now only the 'reserve sectors' are available to perform read/delete/write cycles and if this amount of sectors is small things slow down a lot on average SD cards (not those more recent Samsung).
     
    Just check articles explaining SSDs and keep in mind that SD cards behave like crappy SSDs from a decade ago containing slow/primitive controllers and do not support TRIM.
     
    BTW: This is the only great use case for SD Association's 'SD Formatter'. This tool is used to format SD cards appropriately (partitions it while choosing the 'correct' file system which is either FAT or exFAT) which obviously is pretty useless from Armbian's perspective since burning an OS image as next step both overwrites the partition table and the filesystems present before. So why using SD Formatter in the first place? Since this tool implements ERASE CMD38. It tells the card's controller that every sector/block of the card does not contain any real data any more and can be considered empty. On 'the average' SD card this also might restore horribly low performance back to 'factory default' performance. But more recent SD card controllers especially when paired with many reserve sectors aren't that much affected.
     
    With Armbian's auto resize behaviour leaving at least 1% unpartitioned all that's happening in the background is that we ensure there are more sectors available for the controller's housekeeping which prevents older cards from becoming slow like hell as soon as they've been 'filled completely'.
     
    Edit: A small note regarding 'SD cards don't support TRIM'. The SD protocol defines a block erase command and tools like fstrim are supposed to do the job. Whether your kernel + SD card combination supports that or not a simple 'sudo fstrim -v /' might tell. Whether this has the desired effect or not is a different question though (see this attempt to test this -- I'm not sure whether the method is sufficient since the point of marking data segments as already erased should not involve overwriting them, it's just that the SD card's controller knows that specific sectors/pages can be added to the wear level pool since marked 'emtpy' now)
  5. Like
    tkaiser got a reaction from iav in Why I prefer ZFS over btrfs   
    Guess why you hear so often about data loss with btrfs?
     
    Three main reasons:
     
    1) Shooting the messenger: Putting a btrfs on top of hardware raid, mdraid or lvm is as 'great' as doing the same with ZFS but 'Linux experts' and NAS vendors did it and still do it. Data corruption at the mdraid layer resulting in a broken btrfs on top and guess who's blamed then? https://github.com/openmediavault/openmediavault/issues/101#issuecomment-473920806
     
    2) btrfs lives inside the kernel and as such you need a recent kernel version to escape old and well known bugs. Now look at kernel versions in popular distros like Debian and you get the problem. With ZFS that's different, most recent ZoL versions still run well with horribly outdated kernels. Hobbyists look into btrfs wiki, see that a certain issue has been fixed recently so they're gonna use it forgetting that the 'everything outdated as hell' distro (AKA Debian) they're using is still on an ancient 4.9 kernel or even worse.
     
    3) Choice of hardware. ZFS is most of the time used on more reliable hardware than btrfs and that's mostly due to some BS spread by selected FreeNAS/TrueNAS forum members who created the two urban myths that 'running ZFS without ECC RAM will kill your data' (nope) and 'you'll need at least 8GB of RAM for ZFS' (also BS, I'm running lots of VMs with ZFS with as less as 1GB). While both myths are technically irrelevant a lot of people believe into and therefore invest in better hardware. Check which mainboards feature ECC memory and you'll realize that exactly those mainboards are the ones with proper onboard storage controllers. If you believe you need at least 8 GB of RAM this also rules out a lot of crappy hardware (like eg. vast majority of SBCs).
     
    The main technical challenge for majority of modern filesystem attempts (even for journaled ext4) is correct write-barrier semantics (more on this in the referenced github issue above). Without those both ZFS and btrfs will fail in the same way. So using an SBC with flaky USB host controller combined with crappy USB-to-SATA controller in external enclosure is recipe for desaster with both ZFS and btrfs. ZFS is more mature than btrfs but guess what has happened when ZFS was rather new over a decade ago: people lost their pools due to broken write-barrier semantics caused by erratic HBAs and drive cache behavior (reporting back 'yeah, I've written the sector to spinning rust' while keeping data in some internal cache and after a power loss the pool was gone).
     
    The last time I wasted time with ZFS on ARM it was a true desaster. Over a year ago I tried to run several ODROID HC2 as Znapzend targets for a bunch of fileservers but always ran into stability issues (freezes) after a couple of days/weeks. Switched to btrfs just to check whether all the HC2 might have a hardware problem: a year later realized by accident that the HC2's in question all had an uptime of 300 days or above (I stopped doing kernel updates on isolated Armbian machines since Igor bricked already way too much of my boards over the past years).
     
    Do I prefer btrfs over ZFS? Nope. Being a real ZFS fanboi almost all the storage implementations here and at customers are based on ZFS. On x86 hardware due to not only relying on OpenZFS team but also the guys at Canonical who care about 'ZFS correctly built and fully working with the kernel in question' (on Debian installations pulling in the Proxmox kernel which is latest Ubuntu kernel with additional tweaks/fixes).
     
    And that's the main difference compared to the situation with Armbian. There is no such team that takes care about you being able to access your ZFS pools flawlessly after the next kernel update. Which is the reason why the choice of filesystem (btrfs or ZFS) at my place depends on x86 (ZFS) or ARM (btrfs). And that's not because of the CPU architecture but solely due to with x86 being able to rely on professionals who take care about the updates they roll out vs. situation on ARM with a team of hobbyists who still try harder and harder to provide OS images for as much different SBC as possible so it's absolutely impossible to properly test/support all of them. Playground mode.
  6. Like
    tkaiser got a reaction from schwar3kat in Armbian running on Pine64 (and other A64/H5 devices)   
    A little update regarding 'network performance' since people wrote me the results with our Armbian/Xenial image look too good to be true.
     
    So back to the basics: most benchmarks you find somewhere on the internet regarding SBC performance are showing numbers without meaning since they were made in 'passive benchmarking' mode without taking care of what's important.
      What's different on SBCs compared to real servers (be it x86, Sparc, MIPS, ARMv8)? Network and IO performance on cheap ARM SoCs are affected by current CPU clockspeed (that's different on 'real' servers) A benchmarking tool that adds to CPU utilization on a real server in a negligible fashion might max out CPU ressources on a weak ARM SoC so the tool used for benchmarking might bastardize performance numbers itself. Also when acting single-threaded it might in reality test CPU and not network -- true for iperf in most operation modes The OS distribution might use horribly wrong settings (cpufreq governor, IRQ distribution and so on), might contain mechanisms that are counterproductive (screensavers that start after periods of inactivity and influence performance numbers massively) and might be optimized for different use cases (for example on any ARM SoC around CPU and GPU cores share access to DRAM so we already know that by disabling GPU/HDMI on headless servers we automagically improve performance due to less consumption/temperature now leading to better throttling behaviour under load and more memory bandwidth leading to higher throughput numbers for some tasks) Since we already know that common tools like iperf are CPU intensive let's try out the upper and lower clockspeed on Pine64 to get the idea how a wrong cpufreq governor might influence results (switching way too slow from lower to upper clockspeeds when starting short benchmark executions) and how bottlenecked iperf is by CPU anyway.   Results as follows (please remember: this is always the same hardware and test setup, the only real difference is the OS image used!): TX RX Armbian Xenial @ 480 MHz: 630 / 940 Mbits/sec Armbian Jessie @ 480 MHz: 620 / 600 Mbits/sec pine64.pro Jessie @ 480 MHz: 410 / 595 Mbits/sec TX RX Armbian Xenial @ 1152 MHz: 920 / 940 Mbits/sec Armbian Jessie @ 1152 MHz: 920 / 810 Mbits/sec pine64.pro Jessie @ 1152 MHz: 740 / 770 Mbits/sec What does this mean? Obviously the OS image matters. Someone wanting to benchmark Pine64+ and relying on the OS images from their official download location gets 500 Mbits/sec on average while someone choosing our Xenial image gets 930 Mbits/sec on average.
     
    So let's switch from passive to active benchmarking mode (monitoring the benchmark itself) and check what's different:
    When using Xenial's iperf the tool acts single-threaded only in one direction (TX), with the iperf version in Jessie it's single-threaded in both directions. So by using Jessie you ensure that CPU clockspeed tampers network throughput always in both directions (iperf maxing out one CPU core at 100%) while with Xenial that happens only in TX direction. With Xenial you would also see full 940 Mbits/sec in both directions by adjusting maximum cpufreq from 1152 to 1200 MHz. The performance numbers also show that compiler switches do not affect iperf performance that much when comparing Xenial with Jessie (with other tools like sysbench you get a whopping 30 percent better numbers on Xenial compared to Jessie) The differences between Armbian Jessie and the one from pine64.pro are: my Jessie image is headless while pine64.pro runs a desktop and more importantly that the pine64.pro OS image uses the wrong cpufreq governor (ondemand instead of interactive) which affects standard iperf/iperf3 test execution using the 10s defaults. When using longer test executions benchmark numbers improve but this is obviously the wrong way to handle the problem -- switching to interactive is the real solution With iperf in single-threaded mode you will also see performance differences caused by CPU affinity. If for whatever reasons the kernel decides to assign the iperf task to cpu0 performance numbers might differ compared to running on cpu1 (depends on IRQ distribution). So to get really a clue what's going on you have to monitor IRQ distribution (/proc/interrupts) and assign the benchmark tool to a specific core using taskset to get the idea whether tool and IRQ handling on the same core improve performance or not (since then you could decide whether improving IRQ distribution is worth a try) What do these numbers tell us?
     
    iperf/iperf3 are both not able to measure network througput realiably on SBCs since they are too much CPU bound (as used by most people using default window sizes the results are useless anyway) on the other hand that means that actual CPU clockspeeds matters a lot and therefore choosing an inappropriate cpufreq governor results in worse performance (simple solution: switch to performance or interactive with Allwinner's BSP kernel or schedutil on mainline kernel 4.7 or later) Also inapproriate heat dissipation leads to benchmarks showing lower network performance so in case you cramped your Pine64 in a tiny enclosure without using a heatsink you ensure that performance sucks iperf seems to perform better when using Xenial since it behaves differently there (maybe caused by Jessie compiling distro packages with GCC 4.9 while Xenial uses GCC 5.4 instead). So you get better performance numbers by switching from Jessie to Xenial but it's important to understand that this ONLY affects meaningless benchmark numbers! Real world workloads behave differently. Don't trust in any benchmark blindly iperf when used with default settings (that's 10 seconds test execution here) might show weird/random numbers since we have to deal with a a phenomenon called TX/RX delay and cpufreq scaling governor chosen adds to random behaviour. With iperf you should always test 5 times with 10 seconds, then 1 x 60 seconds and 1 x 300 seconds since then you immediately get the idea why ondemand is wrong for this workload and how the other phenomenon might influence results. Use iperf3 also since this outputs 1 second statistics and might report different numbers (so you learn at least to not blindly trust into benchmark tools!) Most important lesson: Try to understand how these iperf/iperf3 numbers correlate with reality and then measure/monitor real world workloads. If you let Pine64 run as a web server for example with Jessie from pine64.pro then with light workloads Pine64 will be magnitudes slower than when using Armbian (remaining at 480 MHz vs. jumping to 1152 MHz when needed)... or switch to interactive governor with the pine64.pro image. The numbers above do not tell you this real difference for this specific use case. It's always important to try to understand what a benchmark actually tells you We at Armbian should ask ourselves whether we will provide desktop images for Pine64 at all (please, please not, let's prevent that our forum gets flooded with all the HDMI and DVI issues and 'why does firefox not play youtube videos?' and crap like that) and whether we make our CLI images 'true headless' (disabling GPU/HDMI like we did with H3 boards since both SoCs are pretty identical in this regard and I would assume that this helps with specific server workloads a lot due to more memory bandwidth, more useable RAM and better throttling behaviour)
  7. Like
    tkaiser got a reaction from Willy Moto in How to provide and interpret Debug output   
    Armbian implements some basic 'system logging' at every startup and shutdown and contains a little utility to provide this collected information combined with some more useful debug info from user installations. All that's needed is executing 'sudo armbianmonitor -u' (or armbian-config --> Software --> Diagnostics) and then all this support related information is uploaded to an online pasteboard service automagically (see example output) 
     

     
    How to interpret this wall of text? The output is best read from bottom to top since the most important information is collected during upload:
     
    At the bottom /proc/interrupts contents to check for IRQ affinity problems (interrupt collissions on some CPU cores negatively affecting performance) Then last 250 lines dmesg output are included. Here you might find important information wrt the last (kernel) events that happened on the machine uptime output including average load statistics (1, 5 and 15 min) free output telling how much physical memory is available and how much swap and/or zram is used (you need to look directly above whether zram is active or not to interpret the 'Swap:' line) vmstat output contains virtual memory usage information since last reboot iostat output contains the same but allows for a 'per device' view since all devices are listed with individual statistics (so it's easy to spot IO bottlenecks by looking at these numbers and also looking at %iowait value) 'Current system health' displays what the system is actually doing while uploading the debug log (on systems where DC-IN monitoring is available also allowing for underpowering diagnosis -- if you read here numbers below 5.0V stop reading the log and tell the user to fix his underpowering issues first) In case the installation has been moved from SD card to other storage nand-sata-install.log will be included in the output 'Loaded modules' allow to look for module related problems If it's an Allwinner board running legacy kernel the whole script.bin contents are included 'Installed packages' shows version numbers of relevant Armbian packages 'Group membership of' should list all groups the user is member of. If this line is missing ignore the whole contents and ask the user to re-submit debug info, this time doing it correctly not as root but using 'sudo armbianmonitor -u' (group memberships are important to understand certain problems, eg. users not being member of audio group won't have success getting noise out of their devices) If the board is PCIe capable list of attached PCIe devices is included The lsusb output lists all connected USB devices and also information about speed (12M, 480M, 5000M) and protocol/connection details (mass-storage vs uas for example) If the user installed the lshw utility and verbosity is set to 4 or above in /boot/armbianEnv.txt some more disk related information will be included  
    Important: The debug output also contains all collected support files that follow this naming scheme: /tmp/armbianmonitor_checks_* -- so if a user complains about 'transmission so slow' or 'latest files are always missing' ask him to run 'armbianmonitor -c /path/to/torrent-storage' and afterwards 'sudo armbianmonitor -u' without a reboot in between since then the checking results will also be contained.
     
    Everything above of this information at the output's bottom is result of regular logging at startup and shutdown (the contents of /var/log/armhwinfo.log). 
     
    At startup the following items are logged: dmesg output, /etc/armbian-release and /boot/armbianEnv.txt contents, lsusb and lscpu output, /proc/cpuinfo and /proc/meminfo contents, network interface information, available partitions and filesystems, on Allwinner boards where /boot/script.bin points to, some metadata information for all MMC media connected to the host (eg. SD card and/or eMMC) and some system health information.
     
    At shutdown iostat, vmstat and free output are added to /var/log/armhwinfo.log as well as the last 100 lines from dmesg output. If these '### shutdown' entries are missing after reboots the system crashed while shutting down.
  8. Like
    tkaiser got a reaction from iav in Marvell based 4 ports mPCI SATA 3.0   
    The 4 port things are based on Marvell 88SE9215 (this is a 4 port SATA to single PCIe 2.x lane controller, so suitable for mPCIe where only one lane is available). I got mine from Aliexpress, Igor got his for twice the price from Amazon, you also find them sometimes at local resellers, just search for '88SE9215'.
     
    (there's also a x2 variant called 88SE9235 -- just like there's ASM1062 vs. ASM1061 -- but at least behind mPCIe there's no advantage in choosing this over 88SE9215, it would need M.2 or a real PCIe port to make use of the second available lane)
  9. Like
    tkaiser got a reaction from hartraft in How to provide and interpret Debug output   
    Armbian implements some basic 'system logging' at every startup and shutdown and contains a little utility to provide this collected information combined with some more useful debug info from user installations. All that's needed is executing 'sudo armbianmonitor -u' (or armbian-config --> Software --> Diagnostics) and then all this support related information is uploaded to an online pasteboard service automagically (see example output) 
     

     
    How to interpret this wall of text? The output is best read from bottom to top since the most important information is collected during upload:
     
    At the bottom /proc/interrupts contents to check for IRQ affinity problems (interrupt collissions on some CPU cores negatively affecting performance) Then last 250 lines dmesg output are included. Here you might find important information wrt the last (kernel) events that happened on the machine uptime output including average load statistics (1, 5 and 15 min) free output telling how much physical memory is available and how much swap and/or zram is used (you need to look directly above whether zram is active or not to interpret the 'Swap:' line) vmstat output contains virtual memory usage information since last reboot iostat output contains the same but allows for a 'per device' view since all devices are listed with individual statistics (so it's easy to spot IO bottlenecks by looking at these numbers and also looking at %iowait value) 'Current system health' displays what the system is actually doing while uploading the debug log (on systems where DC-IN monitoring is available also allowing for underpowering diagnosis -- if you read here numbers below 5.0V stop reading the log and tell the user to fix his underpowering issues first) In case the installation has been moved from SD card to other storage nand-sata-install.log will be included in the output 'Loaded modules' allow to look for module related problems If it's an Allwinner board running legacy kernel the whole script.bin contents are included 'Installed packages' shows version numbers of relevant Armbian packages 'Group membership of' should list all groups the user is member of. If this line is missing ignore the whole contents and ask the user to re-submit debug info, this time doing it correctly not as root but using 'sudo armbianmonitor -u' (group memberships are important to understand certain problems, eg. users not being member of audio group won't have success getting noise out of their devices) If the board is PCIe capable list of attached PCIe devices is included The lsusb output lists all connected USB devices and also information about speed (12M, 480M, 5000M) and protocol/connection details (mass-storage vs uas for example) If the user installed the lshw utility and verbosity is set to 4 or above in /boot/armbianEnv.txt some more disk related information will be included  
    Important: The debug output also contains all collected support files that follow this naming scheme: /tmp/armbianmonitor_checks_* -- so if a user complains about 'transmission so slow' or 'latest files are always missing' ask him to run 'armbianmonitor -c /path/to/torrent-storage' and afterwards 'sudo armbianmonitor -u' without a reboot in between since then the checking results will also be contained.
     
    Everything above of this information at the output's bottom is result of regular logging at startup and shutdown (the contents of /var/log/armhwinfo.log). 
     
    At startup the following items are logged: dmesg output, /etc/armbian-release and /boot/armbianEnv.txt contents, lsusb and lscpu output, /proc/cpuinfo and /proc/meminfo contents, network interface information, available partitions and filesystems, on Allwinner boards where /boot/script.bin points to, some metadata information for all MMC media connected to the host (eg. SD card and/or eMMC) and some system health information.
     
    At shutdown iostat, vmstat and free output are added to /var/log/armhwinfo.log as well as the last 100 lines from dmesg output. If these '### shutdown' entries are missing after reboots the system crashed while shutting down.
  10. Like
    tkaiser got a reaction from TRS-80 in NanoPi Neo 2, memory leak in proftpd, even worse if SSL encrypted   
    Maybe you were drunk at the time and can't remember? https://github.com/armbian/build/commit/7f1c5b19cd58f100218c0175f9e81df1b5276003#commitcomment-33848416
     
    Moving posts to threads that are inaccessible is the same as deletion (but maybe you're not able to get this). Apropos deletion. Asides that you have not the slightest idea what you babble here about (totally missing the context)
    You claim 'it is written in the internet or in one of TKs posts, ahh wait, he deleted everything here'. Care to elaborate what you mean? Here is the list of my 5432 posts so far: https://forum.armbian.com/profile/7-tkaiser/content/ -- you almost singlehandedly stopped me from posting more in this forum since it makes absolutely no sense to post in a place where a dumbass with moderator privileges deletes posts (and either doesn't get what he does or simply lies).
     
    In case you want to censor again be aware that this post is already archived: https://archive.fo/8eMcV
  11. Like
    tkaiser got a reaction from Werner in Banana Pi BPI-M5 with Amlogic S905X3 chip design   
    The BPi M5 is just an ODROID C4 clone (check also the serial console output in the wall of text above) so simply look there. They even took the same VL817 hub (a way better choice than the Genesis Logic thingy on the N2+) but of course decided to change the power circuitry so that their users suffer from the usual Banana undervoltage drama (unlike the C4 which is up to the task powering a number of USB3 consumers with stable 5V unlike the M5 here).
  12. Like
    tkaiser got a reaction from Werner in How to provide and interpret Debug output   
    Armbian implements some basic 'system logging' at every startup and shutdown and contains a little utility to provide this collected information combined with some more useful debug info from user installations. All that's needed is executing 'sudo armbianmonitor -u' (or armbian-config --> Software --> Diagnostics) and then all this support related information is uploaded to an online pasteboard service automagically (see example output) 
     

     
    How to interpret this wall of text? The output is best read from bottom to top since the most important information is collected during upload:
     
    At the bottom /proc/interrupts contents to check for IRQ affinity problems (interrupt collissions on some CPU cores negatively affecting performance) Then last 250 lines dmesg output are included. Here you might find important information wrt the last (kernel) events that happened on the machine uptime output including average load statistics (1, 5 and 15 min) free output telling how much physical memory is available and how much swap and/or zram is used (you need to look directly above whether zram is active or not to interpret the 'Swap:' line) vmstat output contains virtual memory usage information since last reboot iostat output contains the same but allows for a 'per device' view since all devices are listed with individual statistics (so it's easy to spot IO bottlenecks by looking at these numbers and also looking at %iowait value) 'Current system health' displays what the system is actually doing while uploading the debug log (on systems where DC-IN monitoring is available also allowing for underpowering diagnosis -- if you read here numbers below 5.0V stop reading the log and tell the user to fix his underpowering issues first) In case the installation has been moved from SD card to other storage nand-sata-install.log will be included in the output 'Loaded modules' allow to look for module related problems If it's an Allwinner board running legacy kernel the whole script.bin contents are included 'Installed packages' shows version numbers of relevant Armbian packages 'Group membership of' should list all groups the user is member of. If this line is missing ignore the whole contents and ask the user to re-submit debug info, this time doing it correctly not as root but using 'sudo armbianmonitor -u' (group memberships are important to understand certain problems, eg. users not being member of audio group won't have success getting noise out of their devices) If the board is PCIe capable list of attached PCIe devices is included The lsusb output lists all connected USB devices and also information about speed (12M, 480M, 5000M) and protocol/connection details (mass-storage vs uas for example) If the user installed the lshw utility and verbosity is set to 4 or above in /boot/armbianEnv.txt some more disk related information will be included  
    Important: The debug output also contains all collected support files that follow this naming scheme: /tmp/armbianmonitor_checks_* -- so if a user complains about 'transmission so slow' or 'latest files are always missing' ask him to run 'armbianmonitor -c /path/to/torrent-storage' and afterwards 'sudo armbianmonitor -u' without a reboot in between since then the checking results will also be contained.
     
    Everything above of this information at the output's bottom is result of regular logging at startup and shutdown (the contents of /var/log/armhwinfo.log). 
     
    At startup the following items are logged: dmesg output, /etc/armbian-release and /boot/armbianEnv.txt contents, lsusb and lscpu output, /proc/cpuinfo and /proc/meminfo contents, network interface information, available partitions and filesystems, on Allwinner boards where /boot/script.bin points to, some metadata information for all MMC media connected to the host (eg. SD card and/or eMMC) and some system health information.
     
    At shutdown iostat, vmstat and free output are added to /var/log/armhwinfo.log as well as the last 100 lines from dmesg output. If these '### shutdown' entries are missing after reboots the system crashed while shutting down.
  13. Like
    tkaiser got a reaction from Werner in Amlogic still cheating with clockspeeds   
    TL;DR: the following is a simple summary of the issue:
     
    Amlogic SoCs contain an own embedded microcontroller (Cortex-M3) used for controlling power and clocks A proprietary and closed source firmware is loaded on the M3 at boot. This stuff is contained in the bl30.bin blob we have to include This firmware controls the real clockspeeds, ignores what the cpufreq framework running in the Linux kernel wants and even reports back bogus clockspeeds (the kernel wants to set 1512 MHz, the M3 ignores this and sets 1416 MHz instead but the code returns faked 1512 MHz) Various tests showed that this is not related to thermal protection but just to $something we currently don't understand and out of our control Hardkernel are the only ones who managed to get a bl30.bin blob from Amlogic for their ODROID-C2 that does not cheat on us but honours the cpufreq framework, sets the wanted clockspeeds and also returns real and not faked cpufreq values. On all other Amlogic SBC situation is different  
     
     
    Talking about 'not entirely honest' when it's about bold lies is funny
     
    It's some proprietary crap that controls DVFS on Amlogic SoCs (a bl30.bin BLOB loading some firmware on the embedded Cortex-M3 which controls DVFS/cpufreq on its own) and Hardkernel is the only vendor that got this BLOB from Amlogic in a way where the installation does not cheat on you. In case the BLOB does also DRAM initialization (most likely) it should be hard to exchange it between boards.
     
    https://forum.armbian.com/topic/2138-armbian-for-amlogic-s912/?do=findComment&comment=43338 (S912 and S905X are both known to cheat on the Linux kernel. The cpufreq values are all faked. Most probably this does also apply to all S905 devices except ODROID-C2 since Hardkernel managed to get a fixed BLOB from Amlogic)
     
  14. Like
    tkaiser got a reaction from aaditya in [Mini review] ROCK64 SATA cable   
    No idea. I've one ASM1153E here but never measured consumption (only pretty unprecise and then I didn't notice differences to JMS567 or JMS578). Since the basic problem is always one USB3 SuperSpeed PHY and one SATA 3.0 PHY being active I also doubt that there are real power savings possible,
     
    Though for JMicron SATA bridges we learned recently that there exists the possibility to update/modify the firmware so maybe there are some tweaks possible. While I'm currently in direct contact with JMicron (issue here) I don't want to ask these questions now since waiting for firmware fixes that provide full SAT compliance for JMS567/JMS578.
  15. Like
    tkaiser got a reaction from lanefu in Odroid N2 single CPU handling all the interrupts?   
    https://github.com/armbian/build/commit/1a04b50674626cf0165c84ef463c2b9e3df07061#commitcomment-40258499
  16. Like
    tkaiser got a reaction from manuti in Nanopi R2S overheating and throttling   
    Are you kidding? Again: 
     
    There's no problem with the RK3328, this is just another boring Quad-Core A53 in 28nm (just like the H6). So the question remains what's wrong with NanoPi R2S and not the RK SoCs.
     
    And I really don't understand why so many blindly trust in numbers. There's a thermal sensor inside the SoC, there's a reference voltage, there's some calibration needed, there's driver code. The idea that the numbers this driver spits out are somewhat or even closely related to the actual temperature of the SoC in question is just a hope!
     
    Care to remember what you yourself already reported?
     
  17. Like
    tkaiser got a reaction from iav in SD card performance   
    2018 SD card update
     
    It's 2018 now, SD Association's A1 'performance class' spec is out over a year now and in the meantime we can buy products trying to be compliant to this performance class. SD cards carrying the A1 logo must be able to perform at least 1500 random read input-output operations per second (IOPS) with 4KB block size, 500 random write IOPS and 10 MB/s sustained sequential performance (see here for more details and background info)
     
    Why is this important? Since what we do on SBC at least for the rootfs is mostly random IO and not sequential IO as it's common in cameras or video recorders (that's the stuff SD cards have been invented to be used with in the beginning). As an SBC (or Android) user we're mostly interested in high random IO performance with smaller blocksizes since this is how 'real world' IO patterns mostly look like. Prior to A1 and A2 performance classes there was no way to know how SD cards perform in this area prior to buying. Fortunately this has changed now.
     
    Last week arrived an ODROID N1 dev sample so I bought two SanDisk A1 cards with 32GB capacity each. An el cheapo 'Ultra A1' for 13€ (~$15) and an 'Extreme A1' for 23€. I wanted to buy a slightly more expensive 'Extreme Plus A1' (since even more performance and especially reliability/longevity) but ordered the wrong one  Please keep in mind that the 'Extreme Plus' numbers shown below are made with an older card missing the A1 logo.
     
    Let's look how these things perform, this time on a new platform: RK3399 with an SD card interface that supports higher speed modes (requires kernel support and switching between 3.3V to 1.8V at the hardware layer). So results aren't comparable with the numbers we generated the last two years in this and other threads but that's not important any more... see at the bottom.
     
    A1 conformance requires at least 10 MB/s sequential performance and 500/1500 (write/read) IOPS with 4K blocksize. I tested also with 1K and 16K blocksizes for the simple reason to get an idea whether 4K results are useful to determine performance with smaller or larger blocksizes (since we already know that the vast majority of cheap SD cards out there shows a severe 16K random write performance drop which is the real reason so many people consider all SD cards being crap from a performance point of view).
     
    I tested with 7 cards, 4 of them SanDisk, two Samsung and the 'Crappy card' being a results mixture of a 4GB Kingston I started to test with and old results from a 4GB Intenso from two years ago (see first post of this thread). The Kingston died when testing with 4K blocksize and the performance of all these crappy 'noname class' cards doesn't vary that much:
    1K w/r 4K w/r 16K w/r Crappy card 4GB 32 1854 35 1595 2 603 Samsung EVO+ 128GB 141 1549 160 1471 579 1161 Ultra A1 32GB 456 3171 843 2791 548 1777 Extreme A1 32GB 833 3289 1507 3281 1126 2113 Samsung Pro 64GB 1091 4786 1124 3898 478 2296 Extreme Plus 16GB 566 2998 731 2738 557 2037 Extreme Pro 8GB 304 2779 323 2754 221 1821 (All results in IOPS --> IO operations per second) For A1 compliance we only need to look at the middle column and have to expect at least 500/1500 IOPS minimum here. The 'Crappy card' fails as expected, the Samsung EVO+ too (but we already knew that for whatever reasons newer EVO+ or those with larger capacity perform worse than the 32GB and 64GB variants we tested two years ago), the Samsung Pro shows the best performance here while one of the 4 SanDisk also fails. But my Extreme Pro 8GB is now 3 years old, the other one I had showed signs of data corruption few months ago and when testing 2 years ago (see 1st post in this thread) random write performance was at 800. So most probably this card is about to die soon and the numbers above are partially irrelevant..
     
    What about sequential performance? Well, 'Crappy card' also not able to meet specs and all the better cards being 'bottlenecked' by ODROID N1 (some of these cards show 80 MB/s in my MacBook's card reader but Hardkernel chose to use some safety headroom for good reasons and limits the maximum speed for improved reliability)
    MB/s write MB/s read Crappy card 4GB 9 15 Samsung EVO+ 128GB 21 65 Ultra A1 32GB 20 66 Extreme A1 32GB 59 68 Samsung Pro 64GB 61 66 Extreme Plus 16GB 63 67 Extreme Pro 8GB 50 67 Well, sequential transfer speeds are close to irrelevant with single board computers or Android but it's good to know that boards that allow for higher SD card speed modes (e.g. almost all ODROIDs and the Tinkerboard) also show an improvement in random IO performance if the card is a good one. The ODROID N1 was limited to DDR50 (slowest SD card mode) until today when Hardkernel unlocked UHS capabilities so that my cards (except of 'Crappy card') could all use SDR104 mode. With DDR50 mode sequential performance is limited to 22.5/23.5MB/s (write/read) but more interestingly random IO performance also differs. See IOPS results with the two SanDisk A1 cards, one time limited to DDR50 and then with SDR104:
    1K w/r 4K w/r 16K w/r Ultra A1 DDR50 449 2966 678 2191 445 985 Ultra A1 SDR104 456 3171 843 2791 548 1777 1K w/r 4K w/r 16K w/r Extreme A1 DDR50 740 3049 1039 2408 747 1068 Extreme A1 SDR104 833 3289 1507 3281 1126 2113 We can clearly see that the larger the blocksize the more the interface speed influences also random IO performance (look especially at 16K random reads that double with SDR104)
     
    Some conclusions:
    When comparing results above the somewhat older Samsung Pro performs pretty similar to the Extreme A1. But great random IO performance is only guaranteed with cards carrying the A1 logo (or A2 soon) so it might happen to you that buying another Samsung Pro today results in way lower random IO performance (see Hardkernel's results with a Samsung Pro Plus showing 224/3023 4k IOPS which is way below the 1124/3898 my old Pro achieves with especially write performance 5 times worse and below A1 criteria) We still need to focus on the correct performance metrics. Sequential performance is more or less irrelevant ('Class 10', 'UHS' and so on), all that matters is random IO (A1 and A2 soon). Please keep in mind that you can buy a nice looking UHS card from 'reputable' brands like Kingston, Verbatim, PNY and the like that might achieve theoretical 80MB/s or even 100MB/s sequential performance (you're not able to benefit from anyway since your board's SD card interface will be the bottleneck) but simply sucks with random IO performance. We're talking about up to 500 times worse performance when trusting in 'renowned' brands and ignoring performance reality (see 16k random writes comparing 'Crappy card' and 'Extreme A1') Only a few vendors on this planet run NAND flash memory fabs, only a few companies produce flash memory controllers and have the necessary know-how in house. And only a few combine their own NAND flash with their own controllers to their own retail products. That's the simple reason why at least I only buy SD cards from these 4 brands: Samsung, SanDisk, Toshiba, Transcend The A1 performance speed class is a great and necessary improvement since now we can rely on getting covenant random IO performance. This also helps in fighting counterfeit flash memory products since even if fraudsters in the meantime produce fake SD cards that look real and show same capacity usually these fakes suck at random IO performance. So after testing new cards with either F3 or H2testw it's now another iozone or CrystalDiskMark test to check for overall performance including random IO (!) and if performance sucks you simply return the cards asking for a refund. TL;DR: If you buy new SD cards choose those carrying an A1 or A2 logo. Buy only good brands (their names start with either S or T). Don't trust in getting genuine products but always expect counterfeit stuff. That's why you should only buy at sellers with a 'no questions asked' return/refund policy and why you have to immediately check your cards directly after purchase. If you also care about reliability/resilience buy more expensive cards (e.g. the twice as expensive Extreme Plus A1 instead of Ultra A1) and choose larger capacities than needed.
     
    Finally: All detailed SD card test results can be found here: https://pastebin.com/2wxPWcWr As a comparison performance numbers made with same ODROID N1, same settings but vendor's orange eMMC modules based on Samsung eMMC and varying only in size: https://pastebin.com/ePUCXyg6
  18. Like
    tkaiser got a reaction from aaditya in SD card performance   
    In this 'Let's start a collective benchmark' thread might appear a few other results (and 4 cards are already there that are missing here): 
     
    But IMO we can already sum it up: 
    On boards that only implement the slowest SDIO mode (4 bit @ 50 MHz) the SDIO interface between SoC and SD card becomes the bottleneck when we're talking about sequential transfer speeds. You won't exceed ~23MB/s anyway. Therefore choosing cards where sequential write/read exceed this treshold is somewhat useless unless you use them also somewhere else where the host is capable of higher speeds (this was one of my former use cases: burning OS images in minimum time with +80 MB/s on my laptop -- now mostly irrelevant since FEL/NFS booting is way more fast and convenient when it's about testing new Armbian images) Many 'normal' SanDisk, Toshiba and Transcend cards aren't that fast and show a strange random I/O write weakness with 16k record size (does not apply to the more/most expensive series from the same vendors!) The speed differences when we're talking about small record sizes (the typical card accesses when you put your rootfs or user homes on SD card) vary a lot, especially when it's about random I/O and especially random writes To my surprise the rather cheap Samsung EVOs in our scenario were as fast as the more expensive EVO+ (these show higher potential sequential performance you can't make use of if the SDIO implementation is already the bottleneck) and outperform the even more expensive Samsung Pros in most areas. The EVO's sequential write speeds are advertised as '10 MB/s' but in reality that's something +20 MB/s at least with the few samples we tested with The top performer was Hardkernel's eMMC module combined with their Micro SD adapter. Sequential transfer speeds limited as with the others but random I/O simply rocks. You get what you pay for  Things to mention:
    Boards exist that do not share the slow SDIO limitation as our test boards using Allwinner's A20. On these random I/O might be nearly identical but the sequential speed limitation of ~22/23 MB/s isn't existent. With these boards you might see huge differences when using more expensive branded cards as long as we're talking about sequential transfer speeds (writing/reading huge amounts of data constantly) Regarding more recent SoCs you always have to differentiate between SoC and implementation/board. For example Allwinner's cheap 64-bit A64 is able to use the faster SDIO modes. But on cheap boards like Pine64 this isn't implemented (requires on-the-fly voltage switching between 3.3V and 1.8V for the faster modes). A64 is also able to access eMMC which could be magnitudes faster than SD cards. But since some of the needed eMMC pins are already used for other stuff on the Pine64 you're limited to the slowest SDIO access mode (which also means: all performance numbers above are valid for Pine64 -- I did some cross checks to confirm) Please keep in mind that the Kingston results above as well as all results for cheap 'normal' SanDisk, Toshiba or Transcend cards are hard to interpret. Kingston buys flash chips and controllers on spot markets and combines whatever is cheap at the moment. Therefore you get the irrelevant sequential speeds as advertised but random I/O might vary between different batches a lot (please remember: On SBCs random I/O is more important) Same applies to the cheap SanDisk, Toshiba and Transcend cards, especially if the tested cards are a few years old. Chances are great that you now get a card that is labeled identical but features both a new controller and new NAND chips that are many times faster when we're talking about random I/O Always remember that there are a few renowned storage brands that just buy bunches of unknown flash chips to combine them with unknown controllers. Both performance and reliability might vary a lot between different batches Then there exist manufacturers that own NAND factories, produce their own controllers, have engineers that do not just think about optimal combinations but realize them and ship final products. Think about the difference to the aforementioned group when buying cards And finally... reliability:
    You read all of the above and thought 'Well, what can be wrong ordering 5 of these cheap and performant EVO?' Maybe, that you get something different. Fraud/counterfeit cards are still a massive problem, so always test any card you buy directly after purchase. Always! No excuses! It's really easy and worth the efforts (more info) What about longevity? SD cards were made for cameras/recorders (sequential transfers, access pattern: mostly idle). This is an absolutely different use case than using these cards with an SBC, especially when a lot of random I/O happens (Armbian already helps here with a rather high commit interval: with our defaults all disk related changes are collected and then written to the card just every 10 minutes) I talked recently to someone who deploys lots of devices built of SBCs relying on SD cards. His conclusion: Never ever buy anything again but SanDisk Extreme Plus (not Pro -- yes, those that are twice as expensive ). The costs to replace a broken card at a customer's location are many times higher than buying reliable hardware in the beginning What to do if this is not your situation but you still care about reliability/longevity? You can try to do burn-in tests (maybe lasting a few months/years of continual random writes/reads and results with no meaning since the manufacturer exchanged the controller and/or NAND dies in the meantime) or you take the trust the vendor has into his own products into account. Choosing cards with a warranty of 10 years or even more is not the worst choice in such a situation.
  19. Like
    tkaiser got a reaction from aaditya in SD card performance   
    Warning: This whole thread is only about historical information now since it's 2018 and we can buy inexpensive and great performing A1 rated SD cards in the meantime. Buying anything else is a mistake so directly jump to the end of the thread for performance numbers and recommendations.
     
     
     
    Edit: See an early 2017 update at the end of the thread regarding new SD specs also covering random IO performance.
     
    Edit 2: Some thoughts/observations on lifespan/reliability of SBC storage: http://tech.scargill.net/a-question-of-lifespan/ (see also/especially the comments there)
     
    Edit 3: CNX Software picked up recent performance reports (eg. by Andreas Spiess) and other important issues around SD cards: http://www.cnx-software.com/2017/06/13/micro-sd-cards-for-development-boards-classes-tools-benchmarks-reliability-and-tips-tricks/
     
    Edit 4: See an early 2018 update testing real world A1 performance class products at the end of the thread.
     
    Edit 5: See here and there for some rather boring but very important information about Armbian's tries to prevent SD cards wearing out too fast.
     
     

     
    Preparations:
     
    I tested 8 different SD/TF cards under identical conditions. I created an Armbian 5.07 image to be used on Banana Pi (A20 SoC with 4.4.6 kernel, ext4 rootfs (Armbian defaults == no journal), 960MHz scaling_max_cpufreq, scaling_governor == performance. All test runs were done using 'iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2' and monitored using 'sudo iostat 5' for anomalies (none detected).
      Kernel version matters (since with Allwinner's 3.4 kernel sequential throughput is limited to ~16MB/s, with mainline kernel we get close to Banana Pi SDIO implementation's max: ~23MB/s) and filesystem settings matter too (enabled journal for example slows down 4K writes a lot).   Sequential speeds:     Random I/O:     The 4 Samsung cards were bought within the last 3 weeks and manufactured between 09/2015 and 12/2015 according to the card's metadata. Interesting observation: I used three of these cards the first time and they all show identical behaviour especially regarding writes with small record sizes: pretty slow in the beginning and getting faster over time (the Samsung Pro for example started with only 1400KB/s 4K writes and 3 runs later the same test showed 3270KB/s -- maybe an indication that some sort of calibration happened. Anyway: I know why I always repeat tests and do not rely on a single test run)   Sequential speed mostly irrelevant / random I/O differs and matters a lot!   The SanDisk Extreme Pro has been bought nearly 3 years ago. This card shows superior sequential read/write performance compared to the three Samsung EVOs. But only when used in combination with a host that can make use of these speeds. My MacBook writes 4 times faster an OS image to the Extreme Pro compared to the EVOs. But this doesn't matter at all since the SDIO implementation of the board in question is limited to ~23MB/s (50MHz @ 4 bit). The same sequential write/read speed limitation applies to most SBCs since to be able to exceed this slow mode the voltage the SD/TF card is fed with would've to be adjusted (default 3.3V, the faster modes require a dynamic switch to 1.8V instead which some/most SoCs can perform but if the SBC vendor doesn't implement this you're limited to ~23MB/s).   Therefore cards labeled as being capable of "up to 90MB/s" do not perform different than those that can only do "up to 20MB/s" as long as we're talking about sequential transfers since the SD card interface is already the bottleneck. But since we're using SD/TF cards not in cameras but as storage media for the rootfs of an SBC something different is more important: Random I/O. And here performance of cards that are labeled identical ('class 10' for example) differs a lot.   All 4 Samsungs outperform the other cards easily in this area. The SanDisk Extreme Pro can not compete regarding random I/O compared to superiour (but mostly irrelevant) sequential transfer speeds. And funnily the 3 other cards show horribly slow random write performance, especially with 16k record size. According to card metadata the 2 Intenso are oemid 0x5048 / manfid: 0x000027 (cheap crap known to die way too early) and I would believe the SanDisk 'class 10' card is a fake or at least uses the same controller as the 2 Intenso since 16K random writes are also way slower than 4K writes.   Detailed results (summary table also available as .ods, .xlsx or .txt):   Samsung Pro 64GB (brand new) http://sprunge.us/DEYd   Samsung EVO+ 64GB (brand new) http://sprunge.us/NLQd   Samsung EVO+ 32GB (brand new) http://sprunge.us/heGL   Samsung EVO 64GB (already used some time) http://sprunge.us/DUOS   SanDisk Extreme Pro 8GB (already used some time) http://sprunge.us/ZPFZ   SanDisk 'Class 10' 8GB (already used some time) http://sprunge.us/OHjT   Intenso 'Class 10' 16GB (already used some time) http://sprunge.us/LUMZ   Intenso 'Class 4' 4GB (already used some time) http://sprunge.us/GbAY   Some updates from Igor (still showing superiour EVO results but the clear winner is ODROID's eMMC module with SD card adapter):   Transcend Premium 300x 16GB (almost new) http://sprunge.us/UcSD  
     
     
    Sandisk Extreme Pro 8Gb (almost new) http://sprunge.us/aDJJ
     
     
     
    Hardkernel eMMC 8G via SD reader (brand new) http://sprunge.us/SHXF
     
     
     
    Sandisk Ultra 8Gb (old and very used) http://sprunge.us/OWZf
     
     
     
    Sandisk 8GB (almost new) http://sprunge.us/iViT
     
     
     
    Sandisk 8G (new) http://sprunge.us/XHjj
     
     
     
    Transcend 8Gb (used) http://sprunge.us/NTRT
     
     
     
    Samsung EVO 32Gb (brand new) http://sprunge.us/bTQh
     
     
        Further readings: http://www.bunniestudios.com/blog/?page_id=1022 http://thewirecutter.com/reviews/best-microsd-card/ http://www.jeffgeerling.com/blogs/jeff-geerling/raspberry-pi-microsd-card http://www.bradfordembedded.com/2014/05/flashbenching/ Conclusions:
    If the board's SD card interface is the bottleneck since it's not supporting the faster SDIO modes using expensive cards that exceed the interface's maximum sequential bandwidth is useless. An expensive Samsung Pro Plus won't be faster than a way more cheap EVO when it's about sequential transfer speeds since you will stay at ~22MB/s anyway Sequential read and especially write speeds are all the SD association's speed ratings are about (to ensure reliable recording of videos/images in cameras/recorders) When an SD card is used in an SBC sequential speeds aren't that important. It's all about random I/O, especially with small block sizes (reading and writing small random chunks of data from/to the card) No commonly used 'random I/O' speed ratings exist so you have to check the card in question prior to usage or rely on appropriate benchmarks (see the two links directly above). Again: the 'speed class' won't tell you anything. You can get two different 'class 10' cards that differ by 500% or even more regarding real world storage performance (again: random I/O matters). In the example above the Intenso 'class 10' card is 385 slower compared to the EVOs when it's about 16K random writes (good luck if you have a database running that uses this page size) Interestingly more expensive cards are outperformed by cheaper ones (the EVOs show a better overall performance compared to the Samsung Pro since sequential speeds are limited by the interface) One extreme example: Using an identical cloned installation that was somewhat outdated on the small 4GB Intenso card and on the 64GB EVO resulted in the following times for an 'apt-get upgrade' (+200 packages): EVO less than 6 minutes vs. 390 minutes (yes, ~6.5 hours) with the Intenso. The time to finish depends largely on fast random writes. It's easy to test the card in question when running Armbian since we ship with iozone. Therefore simply execute the iozone call from the first paragraph after logging in as a normal user. Starting with Armbian 5.06 a even better method exists that also tests the whole card for errors: armbianmonitor -c will report precisely both performance and health state of your card
  20. Like
    tkaiser got a reaction from Werner in Banana PI BPI-W2   
    Anyone interested in RTD1295/RTD1296 platform would need to do something like this first
    check out upstream mainline kernel at version 4.9.119 import changes from RealTek's kernel above (you then end up with one giant commit like this -- context) Now you can spend some days or weeks to have a look what's different, where security holes exist (remember Allwinner's 'rootmydevice'?) and what's not properly licensed If this step is skipped there exists no reason to trust into RealTek's 4.9 at all. Especially if this step is skipped it's impossible to start with Armbian support for RTD1295/RTD1295 devices since it would harm the project repeating mistakes like with Allwinner's H3 BSP kernel again (where something like rootmydevice was only discovered by accident).
     
    We can't trust Android vendor's BSP kernels since those employees forced to hack driver support into something they're not interested in (the Linux kernel and its concepts) never worked with mechanisms like peer review or code quality control. If those employees would've started to submit patches upstream where all this stuff happens (Linux kernel community) then this would be something entirely different. But RealTek just like Allwinner and the majority of ARM SoC vendors has no interest in properly upstreaming their code so this code can't be trusted.
     
    If their stuff is not properly licensed this will most likely prevent developing mainline drivers for the affected hardware (but I doubt anyone will do this anyway since so far only Suse's Andreas Färber worked on the platform -- tried to inform him via CNX but no idea whether he's aware that some RealTek kernel sources are now provided).
     
    Having now sources does not mean everything's fine now. Next step is looking at the mess contained (if anyone wants to spend weeks of his life with something like this).
     
     
  21. Like
    tkaiser got a reaction from TheOWL in pyCurl installation Failed: arm-linux-gnueabihf-gcc   
    So it seems, GNUTLS development stuff is missing. Did you try to install libcurl4-gnutls-dev already? Or try 
    apt-cache search gnutls | grep dev and install the packages listed one by one.
  22. Like
    tkaiser got a reaction from lomady in SD card performance   
    Warning: This whole thread is only about historical information now since it's 2018 and we can buy inexpensive and great performing A1 rated SD cards in the meantime. Buying anything else is a mistake so directly jump to the end of the thread for performance numbers and recommendations.
     
     
     
    Edit: See an early 2017 update at the end of the thread regarding new SD specs also covering random IO performance.
     
    Edit 2: Some thoughts/observations on lifespan/reliability of SBC storage: http://tech.scargill.net/a-question-of-lifespan/ (see also/especially the comments there)
     
    Edit 3: CNX Software picked up recent performance reports (eg. by Andreas Spiess) and other important issues around SD cards: http://www.cnx-software.com/2017/06/13/micro-sd-cards-for-development-boards-classes-tools-benchmarks-reliability-and-tips-tricks/
     
    Edit 4: See an early 2018 update testing real world A1 performance class products at the end of the thread.
     
    Edit 5: See here and there for some rather boring but very important information about Armbian's tries to prevent SD cards wearing out too fast.
     
     

     
    Preparations:
     
    I tested 8 different SD/TF cards under identical conditions. I created an Armbian 5.07 image to be used on Banana Pi (A20 SoC with 4.4.6 kernel, ext4 rootfs (Armbian defaults == no journal), 960MHz scaling_max_cpufreq, scaling_governor == performance. All test runs were done using 'iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2' and monitored using 'sudo iostat 5' for anomalies (none detected).
      Kernel version matters (since with Allwinner's 3.4 kernel sequential throughput is limited to ~16MB/s, with mainline kernel we get close to Banana Pi SDIO implementation's max: ~23MB/s) and filesystem settings matter too (enabled journal for example slows down 4K writes a lot).   Sequential speeds:     Random I/O:     The 4 Samsung cards were bought within the last 3 weeks and manufactured between 09/2015 and 12/2015 according to the card's metadata. Interesting observation: I used three of these cards the first time and they all show identical behaviour especially regarding writes with small record sizes: pretty slow in the beginning and getting faster over time (the Samsung Pro for example started with only 1400KB/s 4K writes and 3 runs later the same test showed 3270KB/s -- maybe an indication that some sort of calibration happened. Anyway: I know why I always repeat tests and do not rely on a single test run)   Sequential speed mostly irrelevant / random I/O differs and matters a lot!   The SanDisk Extreme Pro has been bought nearly 3 years ago. This card shows superior sequential read/write performance compared to the three Samsung EVOs. But only when used in combination with a host that can make use of these speeds. My MacBook writes 4 times faster an OS image to the Extreme Pro compared to the EVOs. But this doesn't matter at all since the SDIO implementation of the board in question is limited to ~23MB/s (50MHz @ 4 bit). The same sequential write/read speed limitation applies to most SBCs since to be able to exceed this slow mode the voltage the SD/TF card is fed with would've to be adjusted (default 3.3V, the faster modes require a dynamic switch to 1.8V instead which some/most SoCs can perform but if the SBC vendor doesn't implement this you're limited to ~23MB/s).   Therefore cards labeled as being capable of "up to 90MB/s" do not perform different than those that can only do "up to 20MB/s" as long as we're talking about sequential transfers since the SD card interface is already the bottleneck. But since we're using SD/TF cards not in cameras but as storage media for the rootfs of an SBC something different is more important: Random I/O. And here performance of cards that are labeled identical ('class 10' for example) differs a lot.   All 4 Samsungs outperform the other cards easily in this area. The SanDisk Extreme Pro can not compete regarding random I/O compared to superiour (but mostly irrelevant) sequential transfer speeds. And funnily the 3 other cards show horribly slow random write performance, especially with 16k record size. According to card metadata the 2 Intenso are oemid 0x5048 / manfid: 0x000027 (cheap crap known to die way too early) and I would believe the SanDisk 'class 10' card is a fake or at least uses the same controller as the 2 Intenso since 16K random writes are also way slower than 4K writes.   Detailed results (summary table also available as .ods, .xlsx or .txt):   Samsung Pro 64GB (brand new) http://sprunge.us/DEYd   Samsung EVO+ 64GB (brand new) http://sprunge.us/NLQd   Samsung EVO+ 32GB (brand new) http://sprunge.us/heGL   Samsung EVO 64GB (already used some time) http://sprunge.us/DUOS   SanDisk Extreme Pro 8GB (already used some time) http://sprunge.us/ZPFZ   SanDisk 'Class 10' 8GB (already used some time) http://sprunge.us/OHjT   Intenso 'Class 10' 16GB (already used some time) http://sprunge.us/LUMZ   Intenso 'Class 4' 4GB (already used some time) http://sprunge.us/GbAY   Some updates from Igor (still showing superiour EVO results but the clear winner is ODROID's eMMC module with SD card adapter):   Transcend Premium 300x 16GB (almost new) http://sprunge.us/UcSD  
     
     
    Sandisk Extreme Pro 8Gb (almost new) http://sprunge.us/aDJJ
     
     
     
    Hardkernel eMMC 8G via SD reader (brand new) http://sprunge.us/SHXF
     
     
     
    Sandisk Ultra 8Gb (old and very used) http://sprunge.us/OWZf
     
     
     
    Sandisk 8GB (almost new) http://sprunge.us/iViT
     
     
     
    Sandisk 8G (new) http://sprunge.us/XHjj
     
     
     
    Transcend 8Gb (used) http://sprunge.us/NTRT
     
     
     
    Samsung EVO 32Gb (brand new) http://sprunge.us/bTQh
     
     
        Further readings: http://www.bunniestudios.com/blog/?page_id=1022 http://thewirecutter.com/reviews/best-microsd-card/ http://www.jeffgeerling.com/blogs/jeff-geerling/raspberry-pi-microsd-card http://www.bradfordembedded.com/2014/05/flashbenching/ Conclusions:
    If the board's SD card interface is the bottleneck since it's not supporting the faster SDIO modes using expensive cards that exceed the interface's maximum sequential bandwidth is useless. An expensive Samsung Pro Plus won't be faster than a way more cheap EVO when it's about sequential transfer speeds since you will stay at ~22MB/s anyway Sequential read and especially write speeds are all the SD association's speed ratings are about (to ensure reliable recording of videos/images in cameras/recorders) When an SD card is used in an SBC sequential speeds aren't that important. It's all about random I/O, especially with small block sizes (reading and writing small random chunks of data from/to the card) No commonly used 'random I/O' speed ratings exist so you have to check the card in question prior to usage or rely on appropriate benchmarks (see the two links directly above). Again: the 'speed class' won't tell you anything. You can get two different 'class 10' cards that differ by 500% or even more regarding real world storage performance (again: random I/O matters). In the example above the Intenso 'class 10' card is 385 slower compared to the EVOs when it's about 16K random writes (good luck if you have a database running that uses this page size) Interestingly more expensive cards are outperformed by cheaper ones (the EVOs show a better overall performance compared to the Samsung Pro since sequential speeds are limited by the interface) One extreme example: Using an identical cloned installation that was somewhat outdated on the small 4GB Intenso card and on the 64GB EVO resulted in the following times for an 'apt-get upgrade' (+200 packages): EVO less than 6 minutes vs. 390 minutes (yes, ~6.5 hours) with the Intenso. The time to finish depends largely on fast random writes. It's easy to test the card in question when running Armbian since we ship with iozone. Therefore simply execute the iozone call from the first paragraph after logging in as a normal user. Starting with Armbian 5.06 a even better method exists that also tests the whole card for errors: armbianmonitor -c will report precisely both performance and health state of your card
  23. Like
    tkaiser got a reaction from lomady in SD card performance   
    2018 SD card update
     
    It's 2018 now, SD Association's A1 'performance class' spec is out over a year now and in the meantime we can buy products trying to be compliant to this performance class. SD cards carrying the A1 logo must be able to perform at least 1500 random read input-output operations per second (IOPS) with 4KB block size, 500 random write IOPS and 10 MB/s sustained sequential performance (see here for more details and background info)
     
    Why is this important? Since what we do on SBC at least for the rootfs is mostly random IO and not sequential IO as it's common in cameras or video recorders (that's the stuff SD cards have been invented to be used with in the beginning). As an SBC (or Android) user we're mostly interested in high random IO performance with smaller blocksizes since this is how 'real world' IO patterns mostly look like. Prior to A1 and A2 performance classes there was no way to know how SD cards perform in this area prior to buying. Fortunately this has changed now.
     
    Last week arrived an ODROID N1 dev sample so I bought two SanDisk A1 cards with 32GB capacity each. An el cheapo 'Ultra A1' for 13€ (~$15) and an 'Extreme A1' for 23€. I wanted to buy a slightly more expensive 'Extreme Plus A1' (since even more performance and especially reliability/longevity) but ordered the wrong one  Please keep in mind that the 'Extreme Plus' numbers shown below are made with an older card missing the A1 logo.
     
    Let's look how these things perform, this time on a new platform: RK3399 with an SD card interface that supports higher speed modes (requires kernel support and switching between 3.3V to 1.8V at the hardware layer). So results aren't comparable with the numbers we generated the last two years in this and other threads but that's not important any more... see at the bottom.
     
    A1 conformance requires at least 10 MB/s sequential performance and 500/1500 (write/read) IOPS with 4K blocksize. I tested also with 1K and 16K blocksizes for the simple reason to get an idea whether 4K results are useful to determine performance with smaller or larger blocksizes (since we already know that the vast majority of cheap SD cards out there shows a severe 16K random write performance drop which is the real reason so many people consider all SD cards being crap from a performance point of view).
     
    I tested with 7 cards, 4 of them SanDisk, two Samsung and the 'Crappy card' being a results mixture of a 4GB Kingston I started to test with and old results from a 4GB Intenso from two years ago (see first post of this thread). The Kingston died when testing with 4K blocksize and the performance of all these crappy 'noname class' cards doesn't vary that much:
    1K w/r 4K w/r 16K w/r Crappy card 4GB 32 1854 35 1595 2 603 Samsung EVO+ 128GB 141 1549 160 1471 579 1161 Ultra A1 32GB 456 3171 843 2791 548 1777 Extreme A1 32GB 833 3289 1507 3281 1126 2113 Samsung Pro 64GB 1091 4786 1124 3898 478 2296 Extreme Plus 16GB 566 2998 731 2738 557 2037 Extreme Pro 8GB 304 2779 323 2754 221 1821 (All results in IOPS --> IO operations per second) For A1 compliance we only need to look at the middle column and have to expect at least 500/1500 IOPS minimum here. The 'Crappy card' fails as expected, the Samsung EVO+ too (but we already knew that for whatever reasons newer EVO+ or those with larger capacity perform worse than the 32GB and 64GB variants we tested two years ago), the Samsung Pro shows the best performance here while one of the 4 SanDisk also fails. But my Extreme Pro 8GB is now 3 years old, the other one I had showed signs of data corruption few months ago and when testing 2 years ago (see 1st post in this thread) random write performance was at 800. So most probably this card is about to die soon and the numbers above are partially irrelevant..
     
    What about sequential performance? Well, 'Crappy card' also not able to meet specs and all the better cards being 'bottlenecked' by ODROID N1 (some of these cards show 80 MB/s in my MacBook's card reader but Hardkernel chose to use some safety headroom for good reasons and limits the maximum speed for improved reliability)
    MB/s write MB/s read Crappy card 4GB 9 15 Samsung EVO+ 128GB 21 65 Ultra A1 32GB 20 66 Extreme A1 32GB 59 68 Samsung Pro 64GB 61 66 Extreme Plus 16GB 63 67 Extreme Pro 8GB 50 67 Well, sequential transfer speeds are close to irrelevant with single board computers or Android but it's good to know that boards that allow for higher SD card speed modes (e.g. almost all ODROIDs and the Tinkerboard) also show an improvement in random IO performance if the card is a good one. The ODROID N1 was limited to DDR50 (slowest SD card mode) until today when Hardkernel unlocked UHS capabilities so that my cards (except of 'Crappy card') could all use SDR104 mode. With DDR50 mode sequential performance is limited to 22.5/23.5MB/s (write/read) but more interestingly random IO performance also differs. See IOPS results with the two SanDisk A1 cards, one time limited to DDR50 and then with SDR104:
    1K w/r 4K w/r 16K w/r Ultra A1 DDR50 449 2966 678 2191 445 985 Ultra A1 SDR104 456 3171 843 2791 548 1777 1K w/r 4K w/r 16K w/r Extreme A1 DDR50 740 3049 1039 2408 747 1068 Extreme A1 SDR104 833 3289 1507 3281 1126 2113 We can clearly see that the larger the blocksize the more the interface speed influences also random IO performance (look especially at 16K random reads that double with SDR104)
     
    Some conclusions:
    When comparing results above the somewhat older Samsung Pro performs pretty similar to the Extreme A1. But great random IO performance is only guaranteed with cards carrying the A1 logo (or A2 soon) so it might happen to you that buying another Samsung Pro today results in way lower random IO performance (see Hardkernel's results with a Samsung Pro Plus showing 224/3023 4k IOPS which is way below the 1124/3898 my old Pro achieves with especially write performance 5 times worse and below A1 criteria) We still need to focus on the correct performance metrics. Sequential performance is more or less irrelevant ('Class 10', 'UHS' and so on), all that matters is random IO (A1 and A2 soon). Please keep in mind that you can buy a nice looking UHS card from 'reputable' brands like Kingston, Verbatim, PNY and the like that might achieve theoretical 80MB/s or even 100MB/s sequential performance (you're not able to benefit from anyway since your board's SD card interface will be the bottleneck) but simply sucks with random IO performance. We're talking about up to 500 times worse performance when trusting in 'renowned' brands and ignoring performance reality (see 16k random writes comparing 'Crappy card' and 'Extreme A1') Only a few vendors on this planet run NAND flash memory fabs, only a few companies produce flash memory controllers and have the necessary know-how in house. And only a few combine their own NAND flash with their own controllers to their own retail products. That's the simple reason why at least I only buy SD cards from these 4 brands: Samsung, SanDisk, Toshiba, Transcend The A1 performance speed class is a great and necessary improvement since now we can rely on getting covenant random IO performance. This also helps in fighting counterfeit flash memory products since even if fraudsters in the meantime produce fake SD cards that look real and show same capacity usually these fakes suck at random IO performance. So after testing new cards with either F3 or H2testw it's now another iozone or CrystalDiskMark test to check for overall performance including random IO (!) and if performance sucks you simply return the cards asking for a refund. TL;DR: If you buy new SD cards choose those carrying an A1 or A2 logo. Buy only good brands (their names start with either S or T). Don't trust in getting genuine products but always expect counterfeit stuff. That's why you should only buy at sellers with a 'no questions asked' return/refund policy and why you have to immediately check your cards directly after purchase. If you also care about reliability/resilience buy more expensive cards (e.g. the twice as expensive Extreme Plus A1 instead of Ultra A1) and choose larger capacities than needed.
     
    Finally: All detailed SD card test results can be found here: https://pastebin.com/2wxPWcWr As a comparison performance numbers made with same ODROID N1, same settings but vendor's orange eMMC modules based on Samsung eMMC and varying only in size: https://pastebin.com/ePUCXyg6
  24. Like
    tkaiser got a reaction from pask in Announcement : Odroid N2   
    No, you do NOT know when you're finished firing up your next round of passive benchmarking tests spitting out some numbers. You need to know what's going on to generate answers to the question 'why is A faster than B' and also 'what is the limiter on A and why can't A not be twice as fast'? You missed this change and attributed faster RockPi scores to DDR4 vs. DDR3 memory while in reality you compared old Armbian kernel config with newer one. This whole passive benchmarking approach (and IMO 'technical' Youtube videos in general) always only contributes to confusion and generates zero insights.
     
    The general rule of passive benchmarking and what in fact happened here is: you benchmark A, but actually measure B, and conclude you've measured C (what's needed instead is Active Benchmarking)
     
    What matters are insights, settings and software. And this not only applies to RK3399 but to N2/S922X too of course. So with your use case that utilizes all CPU cores in parallel you surely should go with S922X (since being definitely faster than RK3399 with everything that's multi-threaded), then use a kernel with CONFIG_HZ=100 and an optimized software stack. With Gentoo for example, most recent GCC, optimal/aggressive compiler flags for the A53/A73 combo and a CONFIG_HZ=100 kernel on the N2 your Blender job might finish in less than 40 minutes (maybe just 20 or even less if a programmer equipped with knowledge starts to look into Blender code and adds NEON optimizations to the performance critical code parts -- see here for a great example of Active Benchmarking and adding NEON optimizations on ARM to a software that utilizes SIMD Extensions only on x86 so far just like SSE2 optimized Blender does)
     
    The price you'll pay is a few weeks of your life needed to become a Linux expert since there's nothing ready. Choosing Armbian is usually a matter of convenience but if you want software that performs as fast as possible it's a problematic choice since the software stack is not meant to be performant but to be compatible and stable (funny joke since Armbian's own approach with kernel/bootloader updates is the opposite). The two distro variants Armbian provides are
    Ubuntu also known as 'everything outdated' Debian also known as 'everything outdated as hell' (there are areas where Armbian is in fact faster than other Ubuntu/Debian based images or distros using a more modern software stack like Gentoo or Fedora but this is due to what separates Armbian from the stock Debian/Ubuntu stuff: settings like CPU/IRQ affinity, thermal/DVFS tuning, zram, log2ram and such stuff. But two guys who mostly took care of this contributed nothing to Armbian for a longer time now and no idea how Armbian evolves here in the future. On EoL platforms like Allwinner H5 IRQ affinity is already broken but nobody cares)
     
  25. Like
    tkaiser got a reaction from TRS-80 in Odroid XU4 - Ubuntu Xenial doesn't run on eMMC   
    XU4 and HC1 are fully software compatible (check the readme) and heat dissipation on HC1 is WAY BETTER than with XU4. But you're limited to 2.5" disks here and should keep in mind that Hardkernel already announced a HC2 without that much details (so maybe using the same JMS561 as on Cloudshell 2 minus the interconnection problems -- better ask in their forum).
     
    Wrt new boards being interesting for NAS use cases:
    Pine folks want to provide something called 'Cloudmedia Transformer' (same idea as XU4 vs. HC1, the Transformer is a 100% software compatible ROCK64 + JMS578 on the PCB in an heat emitting metal enclosure that fits a 2.5" HDD, they also thought about a 3.5" variant) Similar to 'Le Potato' a so called 'Le Fly' also with RK3328 might appear (the bottom one here) A bunch of RK3399 devices will be available (all with PCIe 2.x x4 and USB3), I expect software support next year on par with RK3328 EspressoBin might have stable software support next year (you can use the native SATA port there and add mPCIe SATA cards with 2 or even 4 additional real SATA ports) Allwinner H6 boards with both PCIe (single lane, PCIe 2.x) and USB3 will appear (see here and keep in mind that Banana Pi people also have an H6 board in the works) Banana Pi R2 might be ready next year (MTK software support looks promising so maybe next year when mainline kernel support matured we'll support the board) The GnuBees are not listed by intention.
     
     
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines