tkaiser Posted March 1, 2018 Posted March 1, 2018 The nice dashboard screenshot above is used by @Fourdee to explain why DietPi is superiour to Armbian: 'With #DietPi, logs and DietPi scripts are mounted to RAM , this reduces SD card write operations vastly' -- while I don't understand the purpose to 'mount scripts to RAM' of course the idea to cache logs into RAM is great! That's why Armbian does it since 2014 already. While the above 'proof' is somewhat questionable (watching a 5 min period in a dashboard and once there's activity in one graph taking a screenshot with numbers without meaning) let's look into what makes DietPi that superiour compared to Armbian since it's always a great idea to improve even if that means taking over other project's USPs. For whatever reasons DietPi dropped support for all Orange and Banana Pis recently (seems this started with a conversation between @Igor and @Fourdee on Twitter, then continued here and ended up there) so I had to take another board to do a direct comparison. The only boards that are supported by both projects are now Pine64, Rock64, Tinkerboard, some NanoPi and the ODROIDs. I chose Rock64 mostly to ensure that we use same kernel and almost same settings (Armbian's philosophy is to fix as much as possible upstream so our usual performance fixes went into ayufan's Rock64 build scripts DietPi in this case is relying on by accident so even DietPi users can continue to benefit from our work ) I took latest official DietPi image for Rock64 and the first surprise was the rootfs being pretty small and entirely full so no way to proceed: /dev/mmcblk1p7 466M 453M 0 100% / For whatever reasons DietPi chose to overtake ayufan's partition layout (for users new to DietPi: this is always just someone else's Debian image processed manually and by some scripts until it becames 'DietPi') but their 'dietpi-drive_manager' responsible to resize the rootfs seems not to be able to cope with this (I wanted to report it to DietPi but there's already a report that gets ignored and it seems I can't comment there). Edit: Ah, it seems @Fourdee blocked me from helping them entirely. I wanted to assist DietPi folks over at https://github.com/Fourdee/DietPi/issues/1550 but can't point them to fix the thermal issues they're running into again or why it's a bit weird to reintroduce the 'rootmydevice' issue again or why the new Allwinner BSP code is not such a great idea due to non-existing dvfs/thermal support Fortunately our scripts below /usr/local/sbin/ were not deleted by DietPi so I simply called /usr/local/sbin/resize_rootfs.sh which instantly resized the rootfs partition and was then able to continue. For whatever reasons it took 3 whole reboots to get DietPi upgraded to their latest version v6.2 but then I was able to do so some measurements: root@DietPi:/var/log# df -h Filesystem Size Used Avail Use% Mounted on udev 963M 0 963M 0% /dev tmpfs 193M 7.6M 186M 4% /run /dev/mmcblk1p7 466M 453M 0 100% / tmpfs 965M 0 965M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 965M 0 965M 0% /sys/fs/cgroup tmpfs 965M 4.0K 965M 1% /tmp tmpfs 20M 16K 20M 1% /var/log tmpfs 10M 1.4M 8.7M 14% /DietPi /dev/mmcblk1p6 100M 42M 59M 42% /boot/efi root@DietPi:/var/log# which resize_rootfs.sh /usr/local/sbin/resize_rootfs.sh root@DietPi:/var/log# df -h Filesystem Size Used Avail Use% Mounted on udev 963M 0 963M 0% /dev tmpfs 193M 5.2M 188M 3% /run /dev/mmcblk1p7 30G 457M 28G 2% / tmpfs 965M 0 965M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 965M 0 965M 0% /sys/fs/cgroup tmpfs 965M 0 965M 0% /tmp tmpfs 20M 12K 20M 1% /var/log tmpfs 10M 1.4M 8.7M 14% /DietPi 3 Reboots later: root@DietPi:~# uname -a Linux DietPi 4.4.77-rockchip-ayufan-136 #1 SMP Thu Oct 12 09:14:48 UTC 2017 aarch64 GNU/Linux root@DietPi:~# cpu [ OK ] Root access verified. ───────────────────────────────────────────────────── DietPi CPU Info Use dietpi-config to change CPU / performance options ───────────────────────────────────────────────────── Architecture | aarch64 Temp | 46'c : 114'f | Optimal temperature. Governor | ondemand Throttle up | 50% CPU usage Current Freq Min Freq Max Freq CPU0 | 1296 MHz 408 MHz 1296 MHz CPU1 | 1296 MHz 408 MHz 1296 MHz CPU2 | 1296 MHz 408 MHz 1296 MHz CPU3 | 1296 MHz 408 MHz 1296 MHz [ INFO ] DietPi-CPU_info | CPU current frequency, may be affected by this script, due to the processing required to run it. root@DietPi:~# dpkg -l | wc -l 229 root@DietPi:~# free total used free shared buff/cache available Mem: 1974728 39780 1782724 7348 152224 1906564 Swap: 0 0 0 root@DietPi:~# ps auxwww | wc -l 120 root@DietPi:~# lsb_release -c -bash: lsb_release: command not found root@DietPi:~# grep -v "^#" /etc/fstab | sed '/^\s*$/d' proc /proc proc defaults 0 0 tmpfs /tmp tmpfs defaults,noatime,nodev,nosuid,mode=1777 0 0 tmpfs /var/log tmpfs defaults,size=20m,noatime,nodev,nosuid,mode=1777 0 0 tmpfs /DietPi tmpfs defaults,size=10m,noatime,nodev,nosuid,mode=1777 0 0 UUID=6393-85D3 /boot/efi auto defaults,noatime,nofail,x-systemd.automount 0 0 UUID=75d6fd2e-96b1-4f46-a075-5f2aa3de02d3 / auto defaults,noatime 0 0 root@DietPi:~# iostat -y 3600 Linux 4.4.77-rockchip-ayufan-136 (DietPi) 03/01/18 _aarch64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.02 0.00 0.03 0.00 0.00 99.95 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn mmcblk1 0.03 0.28 0.04 1008 156 ^C root@DietPi:~# cat /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state 408000 375774 600000 1971 816000 88 1008000 15 1200000 14 1296000 7803 root@DietPi:~# dumpe2fs /dev/mmcblk1p7 dumpe2fs 1.43.4 (31-Jan-2017) Filesystem volume name: linux-root Last mounted on: /root Filesystem UUID: 75d6fd2e-96b1-4f46-a075-5f2aa3de02d3 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: journal_data_writeback user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 1941504 Block count: 7758971 Reserved block count: 233752 Free blocks: 7510676 Free inodes: 1928761 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 126 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 Flex block group size: 16 Filesystem created: Thu Oct 12 10:30:30 2017 Last mount time: Thu Jan 1 01:00:03 1970 Last write time: Thu Jan 1 01:00:03 1970 Mount count: 8 Maximum mount count: -1 Last checked: Sat Jan 20 15:04:11 2018 Check interval: 0 (<none>) Lifetime writes: 4972 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: fc1fe5c2-59d3-479b-a585-14344b02bec8 Journal backup: inode blocks Journal features: journal_incompat_revoke Journal size: 64M Journal length: 16384 Journal sequence: 0x000044a8 Journal start: 1 I then downloaded our Rock64 nightly image (based on Ubuntu Xenial but that doesn't matter that much -- as we all know the userland stuff is close to irrelevant since kernel and settings matter) and did the same thing. But no reboot needed since for whatever reasons DietPi remained on pretty outdated 4.4.77 kernel so I chose to not update Armbian's kernel to our 4.4.115 but to remain at 4.4.77 too: root@rock64:~# df -h Filesystem Size Used Avail Use% Mounted on udev 962M 0 962M 0% /dev tmpfs 193M 2.9M 191M 2% /run /dev/mmcblk1p1 29G 1.2G 28G 4% / tmpfs 965M 0 965M 0% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 965M 0 965M 0% /sys/fs/cgroup tmpfs 965M 0 965M 0% /tmp log2ram 50M 1.6M 49M 4% /var/log tmpfs 193M 0 193M 0% /run/user/0 root@rock64:~# uname -a Linux rock64 4.4.77-rk3328 #29 SMP Mon Nov 20 03:26:28 CET 2017 aarch64 aarch64 aarch64 GNU/Linux root@rock64:~# armbianmonitor -m Stop monitoring using [ctrl]-[c] Time CPU load %cpu %sys %usr %nice %io %irq CPU C.St. 10:24:41: 1200MHz 0.01 0% 0% 0% 0% 0% 0% 43.6°C 0/5 10:24:46: 408MHz 0.01 0% 0% 0% 0% 0% 0% 43.2°C 0/5 10:24:51: 408MHz 0.01 0% 0% 0% 0% 0% 0% 41.4°C 0/5 10:24:56: 408MHz 0.01 0% 0% 0% 0% 0% 0% 45.5°C 0/5 10:25:01: 408MHz 0.01 0% 0% 0% 0% 0% 0% 43.6°C 0/5^C root@rock64:~# dpkg -l | wc -l 436 root@rock64:~# free total used free shared buff/cache available Mem: 1975120 45140 1832900 7592 97080 1814712 Swap: 987552 0 987552 root@rock64:~# ps auxwww | wc -l 134 root@rock64:~# lsb_release -c Codename: xenial root@rock64:~# grep -v "^#" /etc/fstab | sed '/^\s*$/d' UUID=52ddb7ff-487f-4f7c-8606-4e27c00ab708 / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1 tmpfs /tmp tmpfs defaults,nosuid 0 0 root@rock64:~# iostat -y 3600 Linux 4.4.77-rk3328 (rock64) 03/01/2018 _aarch64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.01 0.00 0.01 0.00 0.00 99.98 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn mmcblk1 0.08 1.64 0.12 5912 448 zram0 0.00 0.00 0.00 0 0 zram1 0.00 0.00 0.00 0 0 zram2 0.00 0.00 0.00 0 0 zram3 0.00 0.00 0.00 0 0 root@rock64:~# sleep 3480 ; cat /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state 408000 373420 600000 62 816000 50 1008000 172 1200000 76 1296000 452 root@rock64:~# dumpe2fs /dev/mmcblk1p1 dumpe2fs 1.42.13 (17-May-2015) Filesystem volume name: <none> Last mounted on: / Filesystem UUID: 52ddb7ff-487f-4f7c-8606-4e27c00ab708 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: journal_data_writeback user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 1850240 Block count: 7709728 Reserved block count: 80976 Free blocks: 7286933 Free inodes: 1815116 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 98 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 7840 Inode blocks per group: 490 Flex block group size: 16 Filesystem created: Mon Nov 20 11:21:03 2017 Last mount time: Thu Feb 11 16:28:00 2016 Last write time: Thu Jan 1 00:00:10 1970 Mount count: 3 Maximum mount count: -1 Last checked: Mon Nov 20 11:21:03 2017 Check interval: 0 (<none>) Lifetime writes: 2295 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: aa162a1d-aa71-437f-a4fb-602658e3c4bb Journal backup: inode blocks Journal features: journal_incompat_revoke Journal size: 32M Journal length: 8192 Journal sequence: 0x00000ffd Journal start: 1 Let's look at the results leaving aside the various performance and security issues DietPi suffers from since not relevant if we want to look at stuff where DietPi outperforms Armbian. First 'idle behaviour': DietPi Armbian DRAM used: 39 MB (2%) 44 MB (2%) processes: 120 134 cpufreq lowest: 97.5% 99.8% cpufreq highest: 2.0% 0.1% idle temp: 46°C 43.5°C %idle percent: 99.95% 99.98% So we're talking more or less about identical numbers. 'Used' memory after booting is 2% of the available 2GB (anyone thinking 'free' RAM would be desirable on Linux... please try to educate yourself: https://www.linuxatemyram.com), the count of processes reported by ps is almost the same, cpufreq behaviour, %idle percentage and temperatures are also the same (DietPi temperature readout is somewhat flawed since their 'cpu' tool affects system behaviour negatively). Even if Armbian ships with almost twice as much packages installed by default the process count doesn't differ that much (and idling processes really don't hurt anyway) and used memory after booting also doesn't differ significantly. But this 'boot and sit there in idle' use case isn't that relevant anyway and in situations when RAM is really needed I would assume Armbian users are in a much better position since we ship with zram active allowed to use half of the physical DRAM (see here for a brief introduction to zram). So far I don't see that much advantages (none to be honest) but most probably I missed something? Anyway: let's continue focussing on storage utilization and 'use': DietPi Armbian size img.7z: 104 MB 223 MB (x 2.1) size img: 668 MB 1.6 GB (x 2.5) rootfs size: 457 MB 1.2 GB (x 2.7) packages: 229 436 (x 1.9) commit interval: 5 s 600 s kB_wrtn: 156 KB 448 KB (x 2.9) kB_read: 1008 KB 5912 KB (x 5.9) So both compressed and uncompressed image sizes are much larger with Armbian, same goes for used space on the rootfs which is understandable given that Armbian does not try to be as minimalistic as possible (see the count of pre-installed packages). I don't think going minimalistic is something desirable though we could think about removing development related packages from default installations as @zador.blood.stained suggested already. Maybe it's worth to adjust the rootfs partition size calculation to use slightly less so the uncompressed image size can be a little bit smaller? Anyway: for people being concerned about smallest image size possible even without leaving out packages from default install simply building an own image and then switching from ext4 to btrfs does the job since reducing image size to around ~60% (one of Armbian's advantages is that our images are not hand-crafted unique 'gems' but the fully automated result of our build system so everyone on this earth can simply build his own Armbian images suiting his own needs). And besides that I really see no benefit in trying to get the rootfs size smaller since we surely don't want to start to encourage users to write Armbian images to old and crappy SD cards with less than 4GB size (though I already consider 4GB cards nothing anyone should use these days since almost all those cards are insanely slow). Let's better continue to educate our users about the importance to choose good and reliable SD cards! Now looking at the last 3 lines above. I executed an 'iostat -y 3600' to query the kernel about the total amount of data read and written at the block device layer. within one whole hour With DietPi/Stretch 156KB/1008KB (write/read) were reported and with Armbian/Xenial 448KB/5912KB (write/read). All numbers are too low for further investigations though something is worth a look: that's the default rootfs 'commit interval.' DietPi seems to use ext4 defaults (sync every 5 seconds to SD card) while in Armbian we choose a somewhat high 10 minute value (commit=600). So while with Armbian and 448 KB written in one hour almost three times as much data has been written at the block device layer it might be possible that the 156 KB written by the DietPi installation caused more wear at the flash layer below due to a phenomenon called Write Amplification (TL;DR version: writes at the flash layer happen at 'page sizes', usually 8K, and by using a high commit interval somewhat larger data chunks will be written only every few minutes which can result in significantly less page writes at the flash layer compared to writing every few seconds smaller chunks of data. Adding to the problem once a card is 'full' now we're talking about much higher Write Amplification since now not just pages are written but usually whole Erase Blocks are affected that are much larger. So please choose your SD card wisely and always use a much larger capacity than needed since there's no TRIM with SD cards in Linux!) It would need a lot of more detailled analysis about this write behaviour but IMO it's not worth the efforts and Armbian's 10 min commit interval does a great job reducing further SD card wearout (anyone with too much spare time? Grab 'iostat 5' and 'iotop -o -b -d5 -q -t -k | grep -v Total' and start to analyse what's happening at the block device and application layer forgetting about the filesystem layer in between!) So where's some room for improvement when comparing our defaults with DietPi's? Maybe removing development related packages from default package list? Maybe tuning rootfs partition creation to use slightly less space? Mostly unrelated but an issue: improving our log2ram behaviour as already discussed? 3
tkaiser Posted March 1, 2018 Author Posted March 1, 2018 Oh, just found this: http://dietpi.com/phpbb/viewtopic.php?f=9&t=2794 (archived version) -- this is a really funny compilation
Igor Posted March 1, 2018 Posted March 1, 2018 1 hour ago, tkaiser said: Maybe removing development related packages from default package list? We need those packages:https://github.com/armbian/build/blob/master/lib/configuration.sh#L121 only here:https://github.com/armbian/build/blob/master/lib/distributions.sh#L112 https://github.com/armbian/build/blob/master/lib/image-helpers.sh#L80 ?
tkaiser Posted March 1, 2018 Author Posted March 1, 2018 1 hour ago, Igor said: We need those packages:https://github.com/armbian/build/blob/master/lib/configuration.sh#L121 only here:https://github.com/armbian/build/blob/master/lib/distributions.sh#L112 https://github.com/armbian/build/blob/master/lib/image-helpers.sh#L80 I was more talking about bundling all this compiler stuff with default installations. Just a quick check which packages consume most space as already done a year ago: root@rock64:/etc/cron.d# wajig large Package Size (KB) Status =================================-==========-============ libc6-dev 11,015 installed locales 13,684 installed binutils 14,288 installed libstdc++-5-dev 14,832 installed cpp-5 15,474 installed iso-codes 16,035 installed g++-5 16,580 installed systemd 16,872 installed perl-modules-5.22 17,260 installed gcc-5 17,622 installed libperl5.22 19,863 installed git 20,868 installed vim-runtime 26,870 installed libicu55 30,046 installed unicode-data 31,681 installed linux-image-rk3328 55,808 installed linux-firmware 207,968 installed root@rock64:/etc/cron.d# wajig large | awk -F" " '/installed$/ {print $1}' | while read ; do > echo -e "${REPLY}: \c" > apt-get --assume-no --purge remove "${REPLY}" | grep "be freed" | awk '{print $4, $5}' > done libc6-dev: 50.4 MB locales: 14.1 MB binutils: 51.5 MB libstdc++-5-dev: 32.2 MB cpp-5: 51.0 MB iso-codes: 16.7 MB g++-5: 17.0 MB systemd: 23.5 MB perl-modules-5.22: 67.5 MB gcc-5: 35.1 MB libperl5.22: 49.8 MB git: 21.4 MB vim-runtime: 29.8 MB libicu55: 47.3 MB unicode-data: 32.4 MB linux-image-rk3328: 57.1 MB linux-firmware: 213 MB Edit: This is how it looks like with DietPi on Rock64: root@DietPi:~# wajig large Package Size (KB) Status =================================-==========-============ systemd 11,739 installed firmware-brcm80211 12,605 installed coreutils 14,572 installed linux-image-4.4.77-rockchip-ayufan-136 51,280 installed linux-headers-4.4.77-rockchip-ayufan-136 69,955 installed root@DietPi:~# wajig large | awk -F" " '/installed$/ {print $1}' | while read ; do > echo -e "${REPLY}: \c" > apt-get --assume-no --purge remove "${REPLY}" | grep "be freed" | awk '{print $4, $5}' > done systemd: 11.4 MB firmware-brcm80211: 12.9 MB coreutils: 16.6 MB
Igor Posted March 1, 2018 Posted March 1, 2018 OK. I left dev stuff in by default and only removed vim package, change apt list to compressed one and tweaked image size (overhead from +40% down to +10% for CLI images): [ o.k. ] Preparing image file for rootfs [ tinkerboard stretch ] [ o.k. ] Current rootfs size [ 753 MiB ] [ o.k. ] Creating blank image for rootfs [ 961 MiB ] I am not sure if 10% is enough ... perhaps 15% will be right number.
chwe Posted March 1, 2018 Posted March 1, 2018 Both comparisons is what I call a 'goal oriented report'. @Fourdee picked topics where DietPi shines and you picked up topics where armbian shines... I avoid to decide who did it more objective, as a 'never used dietPi' guy my opinion wouldn't be objective. And I've a preference for 'not that much' customized systems cause it makes it easier to fix things with informations besides the official forum of a distribution. It's obvious that armbian outperforms dietPi on a kernel level, there's more knowledge here on a devs side and it's part of the project since the beginning whereas DietPi never touched it (correct me if I'm wrong). Both projects have different goals: DietPi is minimalistic and tries to be unique, they purge out everything from debian which they think is not important and fill it with their scripts for an easy to install things for a use case. As a result you get a minimalistic OS. To quote @zador.blood.stained: Quote I see it as Ubuntu/Debian + bootloader with customizations + available kernel(s) with customizations + minimal OS (userspace) customizations and some optional tools/scripts like armbian-config. What we can learn (personal opinion): - 'customer informations' From the rumors I read, their approach seems to be more 'beginners friendly' - use the dietPi scripts to install *random feature* and it will work 'out of the box'. Their tutorials are similar to their OS - minimalistic. You want a 'DietPi - Kodi + BitTorrent Combo': Step1 --> Step2 --> Step3 etc. This is for sure more user friendly that our tutorials section in the forum whereas sometimes 2-3 pages of discussions are filled. I personally appreciate that we allow our users to discuss after a tutorial, a tutorial can be improved things that are wrong can be corrected or different opinions can be discussed but the starting topic of the tutorial should be updated with the new findings, maybe this needs some stronger moderation? DietPi.com is 'more informative' than ours or at least they tell their userbase better why you should use DietPi instead of a *random Distribution*. Whenever this informations are correct or not is out of my knowledge I don't have the time and endurance to check this statement by statement. It's also of no interest to me cause I don't use DietPi. But IMO they explain better what they are than we do. Before you insist that they don't explain well what they aren't - that wasn't part of my statement... I guess and hope that our new web design allows it to improve this slightly so that beginners read through tutorials beginners guides etc. before they start the same questions again and again. - 'public relations' (edit: you can also call it project presentation - but I like those buzz word bs bingo.. ) DietPi is more visible in reviews, reports etc. It might be easier to get attention when you do things different to other distributions as when you try to be a 'boring debian/ubuntu' with just better performance than the images provided by the boardmaker. Maybe we should make our efforts more visible. Most devs/researcher hate public relations as I do too but it's 'part of the deal'. When nobody recognizes what you do better than your competitor 'the world' doesn't recognize your work. Our release history is quite minimalistic which makes it fast to read but not very informative. E.g.: Quote v5.37 / 23.1.2018 bugfix release armbianmonitor -u fix ... What is fixed in armbianmonitor -u? I know that I get all those informations when I look through the forum and through the GitHub history, but we must accept that the average user is to lazy to do it. It's not that every small change needs a 200 words comment what we fixed but a 'predictable place' where you find those announcements could be helpful. E.g.: Quote *random SBC* reaches wip status: what works: -boot via SD-Card -Ethernet -*random feature* what doesn't works/ not tested: -wifi -installation to eMMC -*random feature* When Bloggers, news sites etc. know where they can pick up those informations they might talk more about the project. In case we want to be more visible, we have to spend more time in summarizing our efforts, if this on minor interest the current approach is fine. When I read through @Igor website (the whole posts are not readable anymore and for sure a bit outdated), you spend more time explaining what you did than now. It's clear that the project growth a lot and your workload is for sure high enough that you don't have time to do all of this on your own. Maybe we can make a call if someone wants to fill this position? 3 hours ago, Igor said: OK. I left dev stuff in by default and only removed vim package, change apt list to compressed one and tweaked image size (overhead from +40% down to +10% for CLI images): Is this more a reaction to @Fourdee decision to drop 'armbian based' dietPi images or is it driven by 'rational reasons'? On SD-card driven SBCs image size is IMO not relevant.. (ok, some 8gb large images from the boardsupplier are annoying but armbian isn't that big ) You don't find any new reliable SD-card below 16GB (and to my knowledge no device has less than 8GB eMMC which should also be enough for our images). 10% more or less aren't that important. IMO the SD-Card shouldn't be used as datastorage (but that's a personal opinion too). Wouldn't it make sense to explain our users how you can purge all the stuff they don't need with the buildscript - means that we explain how you manipulate this part of the buildscript with a clear reminder that we don't suggest it for the *average user* and that we don't provide support for it when you run into trouble after doing it. Packages should be removed by reasons, e.g. a driver which affects reliability of the system (talking about XR819 driver... ) or it forces the user to do stupid things (that's where you're the experts knowing it better). huh... It got a bit long...
zador.blood.stained Posted March 1, 2018 Posted March 1, 2018 Since I was mentioned in this thread a couple times already, as much as I wanted to avoid such discussions... 4 minutes ago, chwe said: - 'public relations' please don't make me start on "public relations". I missed the Twitter arguing mentioned in the OP but I don't think that throwing crap at other distribution in the field where winning a "competition" means almost nothing is a good example of "public relations" 8 minutes ago, chwe said: DietPi is minimalistic and tries to be unique, they purge out everything from debian which they think is not important and fill it with their scripts for an easy to install things for a use case. As a result you get a minimalistic OS. DietPi is "diet". Whether it is a healthy diet or not is a good question, but I would call it lightweight, not minimalistic. It contains too many custom scripts on top of the base distribution to be minimalistic IMO. Regarding why people may choose DietPi over other images - this is what I noticed while reading Orange Pi related threads on another forum Read-only rootfs with overlayfs (or whatever is used) and smaller base image size is more robust for bad and slow SD cards Software installer and helper scripts are more user (or should I say "noob") friendly As it was said already - better project presentation (website, tutorials) 6 hours ago, tkaiser said: Maybe it's worth to adjust the rootfs partition size calculation to use slightly less so the uncompressed image size can be a little bit smaller? 6 hours ago, tkaiser said: So where's some room for improvement when comparing our defaults with DietPi's? ... Is this really worth doing right now, just for the purpose of throwing dirt at each other (I mean DietPi <-> Armbian) as I see it?
tkaiser Posted March 1, 2018 Author Posted March 1, 2018 33 minutes ago, zador.blood.stained said: Is this really worth doing right now, just for the purpose of throwing dirt at each other (I mean DietPi <-> Armbian) as I see it? Nope, you're right and apologies. While I tried hard to focus on 'doing it better' you nailed it that I was annoyed by being confrontated again with bizarre 'performance metrics' like those from their Twitter conversation. Trying to calm down now and focus on our problems (the many 5.38 update issues being almost enough for me to throw the towel)
zador.blood.stained Posted March 2, 2018 Posted March 2, 2018 15 hours ago, tkaiser said: While I tried hard to focus on 'doing it better' you nailed it that I was annoyed by being confrontated again with bizarre 'performance metrics' like those from their Twitter conversation. I have to note that dietpi.com front page claims optimizations towards disk space, memory consumption and CPU usage by the OS, not overall CPU, network and I/O performance in real world scenarios which you are interested the most. While it's easy to show that counting processes or checking RAM usage produces numbers without meaning, I don't think we should focus on that (arguing related activities). 1
tkaiser Posted March 2, 2018 Author Posted March 2, 2018 39 minutes ago, zador.blood.stained said: I have to note that dietpi.com front page claims optimizations towards disk space, memory consumption and CPU usage by the OS, not overall CPU, network and I/O performance in real world scenarios which you are interested the most. Oh, I'm also very much interested in reliable operation and one of the key aspects with SBC is reducing wear on flash media like SD cards since at least I (being a low consumption fetishist) want to put the rootfs on energy efficient flash storage and not more 'wasting' SSD or HDD storage. @Fourdee's 'Why DietPi is better' list http://dietpi.com/phpbb/viewtopic.php?f=9&t=2794 shows at least one very dangerous misunderstanding about Linux' VM / page cache implementation but at least that explains his strange conclusions in his Tests 1) and 4). While this is only dangerous for DietPi users it's a bit tough not trying to explain why 'dirty pages' do NOT 'show wait times, before the requested memory data can be written to disk. This generally means bottle-necking has occurred and the overall system performance will be effected until the data is written'. In contrast dirty pages are exactly what we want to reduce wear on flash memory and result of our way larger commit interval at the fs / block device layer. Same with the comparison of the log attempts... once DietPi users switch to full logging (rsyslog + logrotate) their flash media is at risk while it's possible to combine rsyslog + logrotate + logs in RAM. Speaking of that: Do you have an opinion on improving log2ram? https://forum.armbian.com/topic/6626-var-easily-get-full-with-log2ram/?tab=comments#comment-50286
zador.blood.stained Posted March 2, 2018 Posted March 2, 2018 6 minutes ago, tkaiser said: Speaking of that: Do you have an opinion on improving log2ram? https://forum.armbian.com/topic/6626-var-easily-get-full-with-log2ram/?tab=comments#comment-50286 I'm starting to regret that we added it in the past, and we need something that works reliably regardless of the daily log size. I have some thoughts, but different ideas have different requirements and expected reliability issues, and anything requires extensive tests. 3
sgjava Posted March 3, 2018 Posted March 3, 2018 As a dev I vote to keep the dev packages in. Sure is nice to have the build tools (even git client) installed even if things like libtool and pkg-config are missing from the base. If you are going for a minimal install I understand stripping all this stuff out, but I'm not sure how important it is to have a 700K vs 1.6 G image in today's terms. As @tkaiser points out most people have moved on beyond 4G SD cards. I've been using 32G for years and the price/performance is good for what I need it for. I've have some home made security cameras that write/read/delete 100s of movies and images a day 24/7 for years without failure. I realize there may be more intensive usages scenarios (more write intensive), but at the current pricing levels I'll toss the SD out every few years if I have to. As with all things the lowest common denominator or race to the bottom isn't always the best strategy. Stability and standardization are more important to me then a little extra RAM, a little more SD life, etc. Not that these are not important, but the bigger picture I think is more important to focus on. 3
TonyMac32 Posted March 3, 2018 Posted March 3, 2018 1 hour ago, sgjava said: most people have moved on beyond 4G SD cards I completely agree. I think, if a desire to be more minimalist exists, that we create those images specifically labelled as such, for devices that fit the case (OPi IoT, nanopi NEO, Duo, etc). Otherwise I don't see the value. My honest reaction to the discussion is that it fits perfectly with the "What is Armbian?" discussion. I don't think playing games to save a few megabytes fits something we really care about, it isn't really our problem if an extreme subset of users wants to pull the 512 MB SD card out of their 10 year old Sandisk Sansa MP3 player and want to put Linux on it... There are 2 main deliverables that I see: A build system for people to make their flavor of Debian for their needs, and our pre-packaged images. IMHO, if we want to make it more "lean", this can be an option available in the build system deliverable, and if we want to make questionably valuable tiny images, then we can do that in parallel with the current desktop and server images. 4
tkaiser Posted March 4, 2018 Author Posted March 4, 2018 16 hours ago, TonyMac32 said: I don't think playing games to save a few megabytes fits something we really care about, it isn't really our problem if an extreme subset of users wants to pull the 512 MB SD card out of their 10 year old Sandisk Sansa MP3 player and want to put Linux on it... Well, as already explained above: Since Armbian is a build system everyone really keen on using a really small installation media can do (512 MB is not a great idea but possible). Simply by switching from ext4 to btrfs and deleting the linux-firmware package from default package list. When choosing btrfs rootfs size requirement drops down to ~60% since we use zlib compression at image creation time: https://github.com/armbian/build/commit/b14da27a4181e8e232bd8f526e71d2a931a8252f#commitcomment-19484855 When leaving out the linux-firmware package 200 MB less garbage get installed (no idea why this package is included in the first place but I stopped to join any firmware related discussions already since too frustrating, it seems it's up to @Igor to decide about anything firmware related) Using btrfs works quite well (all the thousands of Armbian based OMV installations run this way) and we support the same when transferring Armbian from SD card to eMMC or a disk with nand-sata-install.
Igor Posted March 4, 2018 Posted March 4, 2018 4 minutes ago, tkaiser said: When leaving out the linux-firmware package 200 MB less garbage get installed (no idea why this package is included in the first place but I stopped to join any firmware related discussions already since too frustrating, it seems it's up to @Igor to decide about anything firmware related) It looks like a bug to me or bugfix due to problems related to our dedicated (smaller) armbian-firmware package linux-firmware package is only for Xenial ... I am not sure. Will check a little.
zador.blood.stained Posted March 4, 2018 Posted March 4, 2018 IMO "linux-firmare" in Ubuntu based images should be installed by default as it contains firmware for popular wireless and multimedia devices. Solving random firmware related problems is a complicated task for our user base. 1
tkaiser Posted March 4, 2018 Author Posted March 4, 2018 20 hours ago, sgjava said: I've been using 32G for years and the price/performance is good for what I need it for. I would call the price/performance not good but simply awesome today given we get ultra performant cards like the 32GB SanDisk Ultra A1 for as low as 12 bucks currently: https://www.amazon.com/Sandisk-Ultra-Micro-UHS-I-Adapter/dp/B073JWXGNT/ (I got mine 2 weeks ago for 13€ at a local shop though). And the A1 logo is important since cards compliant to A1 performance class perform magnitudes faster with random IO and small blocksizes (which pretty much describes the majority of IO happening with Linux on our boards). As can be seen in my '2018 A1 SD card performance update' the random IO performance at small blocksizes is magnitudes better compared to an average/old/slow/bad SD card with low capacity: average 4GB card SanDisk Ultra A1 1K read 1854 3171 4K read 1595 2791 16K read 603 1777 1K write 32 456 4K write 35 843 16K write 2 548 With pretty common writes at 4K block size the A1 SanDisk shows 843 vs. 35 IOPS (IO operations per second) and with 16K writes it's 548 vs. 2 IOPS. So that's over 20 or even 250 times faster (I don't know the reason but so far all average SD cards I tested with up to 8 GB capacity show this same weird 16KB random write bottleneck -- even those normal SanDisk Ultra with just 8GB). This might be one of the reasons why 'common knowledge' amongst SBC users seems to be trying to prevent writing to SD card at all. Since the majority doesn't take care which SD cards they use, test them wrongly (looking at irrelevant sequential transfer speeds instead of random IO and IOPS) and chose therefore pretty crappy ones. BTW: the smallest A1 rated cards available start with 16GB capacity. But for obvious reasons I would better buy those with 32GB or even 64GB: price/performance ratio is much better and it should be common knowledge that buying larger cards 'than needed' leads to SD cards wearing out later. 3
Igor Posted March 4, 2018 Posted March 4, 2018 46 minutes ago, zador.blood.stained said: IMO "linux-firmare" in Ubuntu based images should be installed by default as it contains firmware for popular wireless and multimedia devices. Solving random firmware related problems is a complicated task for our user base. Unpacked linux-firmware is around 250Mb, while our custom firmware is 8Mb and it covers onboard + most popular ones. Since we have armbian-config in action: "Install full firmware package" should solve this problem.
tkaiser Posted March 4, 2018 Author Posted March 4, 2018 On 3.3.2018 at 6:30 PM, sgjava said: I've have some home made security cameras that write/read/delete 100s of movies and images a day 24/7 for years without failure Please be careful since this is an entirely different workload compared to 'using SD cards for the rootfs' since storing images and video is more ore less only sequential IO coming with a write amplification close to the optimum: 1. This means that the amount of data written at the filesystem layer, the block device layer and the flash layer are almost identical. With our use cases when running a Linux installation on the SD card this looks totally different since the majoritiy of writes are of very small sizes and write amplification with such small chunks of data is way higher which can result in 1 byte changed at the filesystem or block device layer generating a whole 8K write at the flash layer (so we have a worst case 1:8192 ratio of 'data that has changed' vs. 'real writes to flash cells') Please see here for the full details why this is important, how it matters and what it affects: https://en.wikipedia.org/wiki/Write_amplification Our most basic take on that in Armbian happens at the filesystem layer due to mount options since we use 'noatime,nodiratime,commit=600' by default. What do they do? noatime prevents the filesystem generating writes when data is only accessed (default is that access times are logged in filesystem metadata which leads to updated filesystem structures and therefore unnecessary writes all the time filesystem objects are only read) nodiratime is the same for directories (not that relevant though) commit=600 is the most important one since this tells the filesystem to flush changes back to disk/card only every 600 seconds (10 min) Increasing the commit interval from the default 5 to 600 seconds results in the majority of writes waiting in DRAM to be flushed to disk only every 10 minutes. Those changes sit in the Page Cache (see here for the basics) and add as so called 'dirty pages'. So the amount of dirty pages increases every 10 minutes to be set to 0 after flushing the changes to disk. Can be watched nicely with monitoring tools or something simple as: watch -n 5 grep Dirty /proc/meminfo While @Fourdee tries to explain 'dirty pages' would be something bad or even an indication for degraded performance it's exactly the opposite and just how Linux basics work with a tunable set for a specific use case (rootfs on SD card). To elaborate on the effect: let's think about small changes affecting only 20 byte of change every minute. With filesystem defaults (commit interval = 5 seconds) this will result in 80KB written within 10 minutes (each write affects at least a whole flash page and that's AFAIK 8K or 16K on most cards so at least 8K * 10) while with a 10 minute commit interval only 8KB will be written. Ten times less wear. But unfortunately it's even worse with installations where users run off low capacity SD cards. To my knowledge in Linux we still have no TRIM functionality with MMC storage (SD cards, eMMC) so once the total amount of data written to the card exceeds its native capacity the card controller has no clue how to distinguish between free and occupied space and has therefore to start to delete (there's no overwrite with flash storage, see for example this explanation). So all new writes now might even affect not just pages but whole so called 'Erase Blocks' that might be much larger (4MB or 16MB for example on all the cards I use). This is for example explained here. In such a case (amount of writes exceed card's native capacity) we're now talking about writes affecting Erase Blocks that might be 4MB in size. With the above example of changing 20 bytes every minute with the default commit interval of 5 seconds at the flash layer now even 40 MB would be written while with a 10 min commit interval it's 4MB (all with just 200 bytes having changed in reality). So if you really care about the longevity of your card you buy good cards with capacities much 'larger than needed', clone them from time to time to another card and now perform a TRIM operation manually by using your PC or Mac and SD Association's 'SD Formatter' to do a quick erase there. This will send ERASE (CMD38) for all flash pages to the card's controller which now treats all pages as really empty so new writes to the card from now on do NOT generate handling of whole Erase Blocks but happen at the page size level again (until the card's capacity is fully used, then you would need to repeat the process). There's a downside with an increased commit interval as usual and that affects unsafe power-offs / crashes. Everything that sits in the Page Cache and is not already flushed back to disk/card is lost in case a power loss occurs or something similar. On the bright side this higher commit interval makes it less likely that you run into filesystem corruption since filesystem structures are updated on disk also only every 10 minutes. Besides that we try to cache other stuff in RAM as much as possible (eg. browser caches and user profiles using 'profile sync daemon') and same goes for log files which are amongst those candidates that show worst write amplification possible when allowing to update logs every few seconds on 'disk' (unfortunately we can't throw logs just away as for example DietPi does it by default so we have to fiddle around with stuff like our log2ram implementation showing lots of room for improvements) 1
chwe Posted March 4, 2018 Posted March 4, 2018 In case we want to explain what armbian is.... Your post should be linked under something like 'durability'. We aren't that bad to explain where armbian differs from other 'distributions' for SBCs... we just suck to make this info 'easy accessible'. Edit: It's not about throwing dirt to others. It's about explaining why we do the things the way we do and why.
tkaiser Posted March 10, 2018 Author Posted March 10, 2018 On 4.3.2018 at 4:38 PM, chwe said: In case we want to explain what armbian is.... Your post should be linked under something like 'durability'. We aren't that bad to explain where armbian differs from other 'distributions' for SBCs... This is not that much about Armbian. This is just about flash storage basics! But no one wants to read through the above or such complex and boring stuff like this: https://forum.armbian.com/topic/6444-varlog-file-fills-up-to-100-using-pihole/?do=findComment&comment=50833 -- no one wants to deal with the differences between the affected layers (filesystem, block device, flash memory). Users usually prefer simple answers (even if wrong) over complex ones. That's why a distro should choose sane defaults (a trade-off as usual since a high commit interval also affects the amount of data lost after a crash) and that's why educating users is needed. But it seems Armbian is one of the few places where that happens (or maybe even just my essays here in the forum?). The problem with SD cards and SBC is the huge amount of ignorance and stupidity we all have to deal with. People are even today told to focus only on BS performance metrics (e.g. sequential read and write performance instead of the only thing that's really important: random IO performance, here especially writes and this especially with small to medium blocksizes). People are even today encouraged to use the most shitty SD cards possible instead of telling them the truth to better buy a new and large enough one with great random IO performance. Since an awful lot of tasks depends on sufficient (random) IO performance and you simply won't be happy with a crappy SD card unless you use it read-only.
chwe Posted March 10, 2018 Posted March 10, 2018 4 hours ago, tkaiser said: This is not that much about Armbian. This is just about flash storage basics! And that's exactly what Armbian is! For someone not familiar with this basics, Armbian take care about it - even if you don't know that this is an issue. It's not that you can't change this behavior and do 'stupid things' with armbian. But a 'stock' Armbian tries to avoid it. 4 hours ago, tkaiser said: The problem with SD cards and SBC is the huge amount of ignorance and stupidity we all have to deal with. People are even today told to focus only on BS performance metrics (e.g. sequential read and write performance instead of the only thing that's really important: random IO performance, here especially writes and this especially with small to medium blocksizes). That's why we should have your essays collected somewhere. For me the best way would be that they are written somewhere as a 'tutorial'/'education' part.. But links to the forum posts could also work. For me, Armbian is: clean Debian/Ubuntu (due to debootstrap processing of rootfs) on top of an enhanced BSP/mainline kernel (due to a bunch of patches) enhanced durability (due to log2ram & longer 'commit interval') one of the few distributions providing kernel updates on 'living systems' for arm boards (which goes sometimes wrong - but we try to fix it in case it goes wrong, and yes there is some space for improvements) an active community trying to help you when you struggle - even your issue is not Armbian related/ due to lack of (basic) linux knowledge etc. providing a similar linux experience over different SoCs/'brands' - the reason I'll never take use an image provided by the boardmaker (e.g. friendlyArm) is not because I think all of them do a bad job, it's because I don't want to waste hours of time to figure out where their behavior is different from the images provided by another boardmaker/distribution. a mighty buildscript which allows the user to fix things on his own (e.g. @TonyMac32 and me try to support CSI for the tinker, we still fail but the buildscript allows me to create different kernel configs & patches for the kernel/u-boot etc without doing the annoying diff process on my own.. I can simply do my changes inside the sources and as soon as I fixed something I can send a PR to armbian and other people can benefit from my enhancement or in case Armbian doesn't accept the PR I can provide it as an userpatch so that everyone interested in the same 'enhancement' can build it on his own). I agree that part of the people are ignorant when it comes to durability and 'proper usage' of SBCs. I also agree that user expectations are sometimes ridiculous (I want a 8$ SBC together with a 1$ PSU and a 2$ SD-card and I expect desktop performance when it's about multimedia stuff with linux! ). I'm not sure if this is the majority, I guess that there's a 'silent majority' of users which are happy with what Armbian offers to them. Normally you don't register to a Forum just to say "Awsome! Thank you for providing an reliable Debian/Ubuntu for my *random SBC*!" It's more than you register cause you something doesn't work as expected, cause it doesn't boot anymore or cause you bought a new SBC which is not supported (yet) by Armbian. As a moderator here, I get all these reminders to approve new forum postings and therefore I read a bunch of them... Some of them are funny, some of them are stupid a few of them are obviously spam but even less of them are 'impressive'. The last one I can remember was when @konsgn posted his first post where he nailed down the OPi0 thermal issue with his first post. So why did he wrote this here and not in Xunlongs forum (in case you did, mea culpa for my low google-fu , I only found it here)? Maybe he's one of the 'silent' users which are (mostly) happy with what armbian offers to them, maybe cause he thought it make sense to share it in an active community - I don't know but it seems we are a good place for sharing such informations.. We can provide those informations in a 'marketing manner' and I'm sure @zador.blood.stained wouldn't be happy when we do it this way. We can do it the way we do it at the moment, explaining the same questions again and again (and again ) to our 'customers' or we can try to collect them so that they are accessible 'as easy as possible'. 1
iamwithstupid Posted August 28, 2018 Posted August 28, 2018 On 3/10/2018 at 12:43 PM, tkaiser said: But no one wants to read through the above or such complex and boring stuff like this: https://forum.armbian.com/topic/6444-varlog-file-fills-up-to-100-using-pihole/?do=findComment&comment=50833 -- no one wants to deal with the differences between the affected layers (filesystem, block device, flash memory). Users usually prefer simple answers (even if wrong) over complex ones. Oh, I do! I'm even eating Bierstangerl while reading your post trying to grasp up some straws of knowdlege (maybe some of them will stick). Just registered because I had to add that ... have been reading your posts for some months now. Haven't flashed armbian yet tough, hench why I didn't feel the need to register yet. But oh, well, now I'll just do both ... ---- Edit: So I've tried the DietPi and the Armbian Nightly build and the idle and load temps are up to 20° higher on DietPi even when I switch the governor / frequencies in DietPi (they're correctly applied). That is for my NanoPi Neo2 - DietPi 4.14.y kernel idle after about 10 hours in the stock 3D print case with heatsink attached has an idle temp of about 60° - compared to armbian nightly with 4.18.y kernel - 40°. Both running pi-hole only so far, all updates applied. Same config.
Recommended Posts