0
tkaiser

Learning from DietPi!

Recommended Posts

DVS-VVaXcAUgbqn.png-large.png

 

The nice dashboard screenshot above is used by @Fourdee to explain why DietPi is superiour to Armbian: 'With #DietPi, logs and DietPi scripts are mounted to RAM , this reduces SD card write operations vastly' -- while I don't understand the purpose to 'mount scripts to RAM' of course the idea to cache logs into RAM is great! That's why Armbian does it since 2014 already.

 

While the above 'proof' is somewhat questionable (watching a 5 min period in a dashboard and once there's activity in one graph taking a screenshot with numbers without meaning) let's look into what makes DietPi that superiour compared to Armbian since it's always a great idea to improve even if that means taking over other project's USPs.

 

For whatever reasons DietPi dropped support for all Orange and Banana Pis recently (seems this started with a conversation between @Igor and @Fourdee on Twitter, then continued here and ended up there) so I had to take another board to do a direct comparison. The only boards that are supported by both projects are now Pine64, Rock64, Tinkerboard, some NanoPi and the ODROIDs. I chose Rock64 mostly to ensure that we use same kernel and almost same settings (Armbian's philosophy is to fix as much as possible upstream so our usual performance fixes went into ayufan's Rock64 build scripts DietPi in this case is relying on by accident so even DietPi users can continue to benefit from our work :) )

 

I took latest official DietPi image for Rock64 and the first surprise was the rootfs being pretty small and entirely full so no way to proceed:

/dev/mmcblk1p7  466M  453M     0 100% /

For whatever reasons DietPi chose to overtake ayufan's partition layout (for users new to DietPi: this is always just someone else's Debian image processed manually and by some scripts until it becames 'DietPi') but their 'dietpi-drive_manager' responsible to resize the rootfs seems not to be able to cope with this (I wanted to report it to DietPi but there's already a report that gets ignored and it seems I can't comment there).

 

Edit: Ah, it seems @Fourdee blocked me from helping them entirely. I wanted to assist DietPi folks over at https://github.com/Fourdee/DietPi/issues/1550 but can't point them to fix the thermal issues they're running into again or why it's a bit weird to reintroduce the 'rootmydevice' issue again or why the new Allwinner BSP code is not such a great idea due to non-existing dvfs/thermal support :) 

 

Fortunately our scripts below /usr/local/sbin/ were not deleted by DietPi so I simply called /usr/local/sbin/resize_rootfs.sh which instantly resized the rootfs partition and was then able to continue. For whatever reasons it took 3 whole reboots to get DietPi upgraded to their latest version v6.2 but then I was able to do so some measurements:

 

root@DietPi:/var/log# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            963M     0  963M   0% /dev
tmpfs           193M  7.6M  186M   4% /run
/dev/mmcblk1p7  466M  453M     0 100% /
tmpfs           965M     0  965M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           965M     0  965M   0% /sys/fs/cgroup
tmpfs           965M  4.0K  965M   1% /tmp
tmpfs            20M   16K   20M   1% /var/log
tmpfs            10M  1.4M  8.7M  14% /DietPi
/dev/mmcblk1p6  100M   42M   59M  42% /boot/efi

root@DietPi:/var/log# which resize_rootfs.sh 
/usr/local/sbin/resize_rootfs.sh

root@DietPi:/var/log# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            963M     0  963M   0% /dev
tmpfs           193M  5.2M  188M   3% /run
/dev/mmcblk1p7   30G  457M   28G   2% /
tmpfs           965M     0  965M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           965M     0  965M   0% /sys/fs/cgroup
tmpfs           965M     0  965M   0% /tmp
tmpfs            20M   12K   20M   1% /var/log
tmpfs            10M  1.4M  8.7M  14% /DietPi

3 Reboots later: 

root@DietPi:~# uname -a
Linux DietPi 4.4.77-rockchip-ayufan-136 #1 SMP Thu Oct 12 09:14:48 UTC 2017 aarch64 GNU/Linux

root@DietPi:~# cpu
[  OK  ] Root access verified.

 ─────────────────────────────────────────────────────
 DietPi CPU Info
 Use dietpi-config to change CPU / performance options
 ─────────────────────────────────────────────────────
 Architecture |     aarch64
 Temp         |     46'c : 114'f | Optimal temperature.
 Governor     |     ondemand
 Throttle up  |     50% CPU usage

                 Current Freq    Min Freq   Max Freq
 CPU0         |      1296 MHz      408 MHz    1296 MHz
 CPU1         |      1296 MHz      408 MHz    1296 MHz
 CPU2         |      1296 MHz      408 MHz    1296 MHz
 CPU3         |      1296 MHz      408 MHz    1296 MHz

[ INFO ] DietPi-CPU_info | CPU current frequency, may be affected by this script, due to the processing required to run it.

root@DietPi:~# dpkg -l | wc -l
229

root@DietPi:~# free
              total        used        free      shared  buff/cache   available
Mem:        1974728       39780     1782724        7348      152224     1906564
Swap:             0           0           0

root@DietPi:~# ps auxwww | wc -l
120

root@DietPi:~# lsb_release -c
-bash: lsb_release: command not found

root@DietPi:~# grep -v "^#" /etc/fstab | sed '/^\s*$/d'
proc            /proc           proc    defaults											0 0
tmpfs 			/tmp  			tmpfs 	defaults,noatime,nodev,nosuid,mode=1777				0 0
tmpfs 			/var/log 		tmpfs 	defaults,size=20m,noatime,nodev,nosuid,mode=1777	0 0
tmpfs 			/DietPi 		tmpfs 	defaults,size=10m,noatime,nodev,nosuid,mode=1777	0 0
UUID=6393-85D3    /boot/efi    auto    defaults,noatime,nofail,x-systemd.automount    0 0
UUID=75d6fd2e-96b1-4f46-a075-5f2aa3de02d3    /    auto    defaults,noatime    0 0

root@DietPi:~# iostat -y 3600
Linux 4.4.77-rockchip-ayufan-136 (DietPi) 	03/01/18 	_aarch64_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.02    0.00    0.03    0.00    0.00   99.95

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk1           0.03         0.28         0.04       1008        156

^C
root@DietPi:~# cat /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state 
408000 375774
600000 1971
816000 88
1008000 15
1200000 14
1296000 7803

root@DietPi:~# dumpe2fs /dev/mmcblk1p7 
dumpe2fs 1.43.4 (31-Jan-2017)
Filesystem volume name:   linux-root
Last mounted on:          /root
Filesystem UUID:          75d6fd2e-96b1-4f46-a075-5f2aa3de02d3
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    journal_data_writeback user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1941504
Block count:              7758971
Reserved block count:     233752
Free blocks:              7510676
Free inodes:              1928761
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      126
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Thu Oct 12 10:30:30 2017
Last mount time:          Thu Jan  1 01:00:03 1970
Last write time:          Thu Jan  1 01:00:03 1970
Mount count:              8
Maximum mount count:      -1
Last checked:             Sat Jan 20 15:04:11 2018
Check interval:           0 (<none>)
Lifetime writes:          4972 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      fc1fe5c2-59d3-479b-a585-14344b02bec8
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke
Journal size:             64M
Journal length:           16384
Journal sequence:         0x000044a8
Journal start:            1

 

 

I then downloaded our Rock64 nightly image (based on Ubuntu Xenial but that doesn't matter that much -- as we all know the userland stuff is close to irrelevant since kernel and settings matter) and did the same thing. But no reboot needed since for whatever reasons DietPi remained on pretty outdated 4.4.77 kernel so I chose to not update Armbian's kernel to our 4.4.115 but to remain at 4.4.77 too:

 

root@rock64:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            962M     0  962M   0% /dev
tmpfs           193M  2.9M  191M   2% /run
/dev/mmcblk1p1   29G  1.2G   28G   4% /
tmpfs           965M     0  965M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           965M     0  965M   0% /sys/fs/cgroup
tmpfs           965M     0  965M   0% /tmp
log2ram          50M  1.6M   49M   4% /var/log
tmpfs           193M     0  193M   0% /run/user/0

root@rock64:~# uname -a
Linux rock64 4.4.77-rk3328 #29 SMP Mon Nov 20 03:26:28 CET 2017 aarch64 aarch64 aarch64 GNU/Linux

root@rock64:~# armbianmonitor -m
Stop monitoring using [ctrl]-[c]
Time        CPU    load %cpu %sys %usr %nice %io %irq   CPU  C.St.

10:24:41: 1200MHz  0.01   0%   0%   0%   0%   0%   0% 43.6°C  0/5
10:24:46:  408MHz  0.01   0%   0%   0%   0%   0%   0% 43.2°C  0/5
10:24:51:  408MHz  0.01   0%   0%   0%   0%   0%   0% 41.4°C  0/5
10:24:56:  408MHz  0.01   0%   0%   0%   0%   0%   0% 45.5°C  0/5
10:25:01:  408MHz  0.01   0%   0%   0%   0%   0%   0% 43.6°C  0/5^C

root@rock64:~# dpkg -l | wc -l
436

root@rock64:~# free
              total        used        free      shared  buff/cache   available
Mem:        1975120       45140     1832900        7592       97080     1814712
Swap:        987552           0      987552

root@rock64:~# ps auxwww | wc -l
134

root@rock64:~# lsb_release -c
Codename:	xenial

root@rock64:~# grep -v "^#" /etc/fstab | sed '/^\s*$/d'
UUID=52ddb7ff-487f-4f7c-8606-4e27c00ab708 / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1
tmpfs /tmp tmpfs defaults,nosuid 0 0

root@rock64:~# iostat -y 3600
Linux 4.4.77-rk3328 (rock64) 	03/01/2018 	_aarch64_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.01    0.00    0.01    0.00    0.00   99.98

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk1           0.08         1.64         0.12       5912        448
zram0             0.00         0.00         0.00          0          0
zram1             0.00         0.00         0.00          0          0
zram2             0.00         0.00         0.00          0          0
zram3             0.00         0.00         0.00          0          0

root@rock64:~# sleep 3480 ; cat /sys/devices/system/cpu/cpu0/cpufreq/stats/time_in_state
408000 373420
600000 62
816000 50
1008000 172
1200000 76
1296000 452

root@rock64:~# dumpe2fs /dev/mmcblk1p1
dumpe2fs 1.42.13 (17-May-2015)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          52ddb7ff-487f-4f7c-8606-4e27c00ab708
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    journal_data_writeback user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1850240
Block count:              7709728
Reserved block count:     80976
Free blocks:              7286933
Free inodes:              1815116
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      98
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         7840
Inode blocks per group:   490
Flex block group size:    16
Filesystem created:       Mon Nov 20 11:21:03 2017
Last mount time:          Thu Feb 11 16:28:00 2016
Last write time:          Thu Jan  1 00:00:10 1970
Mount count:              3
Maximum mount count:      -1
Last checked:             Mon Nov 20 11:21:03 2017
Check interval:           0 (<none>)
Lifetime writes:          2295 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      aa162a1d-aa71-437f-a4fb-602658e3c4bb
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke
Journal size:             32M
Journal length:           8192
Journal sequence:         0x00000ffd
Journal start:            1

 

 

Let's look at the results leaving aside the various performance and security issues DietPi suffers from since not relevant if we want to look at stuff where DietPi outperforms Armbian. First 'idle behaviour':

                    DietPi         Armbian
DRAM used:         39 MB (2%)     44 MB (2%)
processes:            120            134
cpufreq lowest:      97.5%          99.8%
cpufreq highest:      2.0%           0.1%
idle temp:           46°C           43.5°C
%idle percent:      99.95%          99.98%

So we're talking more or less about identical numbers. 'Used' memory after booting is 2% of the available 2GB (anyone thinking 'free' RAM would be desirable on Linux... please try to educate yourself: https://www.linuxatemyram.com), the count of processes reported by ps is almost the same, cpufreq behaviour, %idle percentage and temperatures are also the same (DietPi temperature readout is somewhat flawed since their 'cpu' tool affects system behaviour negatively).

 

Even if Armbian ships with almost twice as much packages installed by default the process count doesn't differ that much (and idling processes really don't hurt anyway) and used memory after booting also doesn't differ significantly. But this 'boot and sit there in idle' use case isn't that relevant anyway and in situations when RAM is really needed I would assume Armbian users are in a much better position since we ship with zram active allowed to use half of the physical DRAM (see here for a brief introduction to zram).

 

So far I don't see that much advantages (none to be honest) but most probably I missed something?

 

Anyway: let's continue focussing on storage utilization and 'use':

                    DietPi             Armbian
size img.7z:        104 MB          223 MB  (x 2.1)
size img:           668 MB          1.6 GB  (x 2.5)
rootfs size:        457 MB          1.2 GB  (x 2.7)
packages:             229            436    (x 1.9)
commit interval:      5 s            600 s
kB_wrtn:            156 KB           448 KB (x 2.9)
kB_read:           1008 KB          5912 KB (x 5.9)

So both compressed and uncompressed image sizes are much larger with Armbian, same goes for used space on the rootfs which is understandable given that Armbian does not try to be as minimalistic as possible (see the count of pre-installed packages). I don't think going minimalistic is something desirable though we could think about removing development related packages from default installations as @zador.blood.stained suggested already. Maybe it's worth to adjust the rootfs partition size calculation to use slightly less so the uncompressed image size can be a little bit smaller?

 

Anyway: for people being concerned about smallest image size possible even without leaving out packages from default install simply building an own image and then switching from ext4 to btrfs does the job since reducing image size to around ~60% (one of Armbian's advantages is that our images are not hand-crafted unique 'gems' but the fully automated result of our build system so everyone on this earth can simply build his own Armbian images suiting his own needs).

 

And besides that I really see no benefit in trying to get the rootfs size smaller since we surely don't want to start to encourage users to write Armbian images to old and crappy SD cards with less than 4GB size (though I already consider 4GB cards nothing anyone should use these days since almost all those cards are insanely slow). Let's better continue to educate our users about the importance to choose good and reliable SD cards!

 

Now looking at the last 3 lines above. I executed an 'iostat -y 3600' to query the kernel about the total amount of data read and written at the block device layer. within one whole hour With DietPi/Stretch 156KB/1008KB (write/read) were reported and with Armbian/Xenial 448KB/5912KB (write/read). All numbers are too low for further investigations though something is worth a look: that's the default rootfs 'commit interval.' DietPi seems to use ext4 defaults (sync every 5 seconds to SD card) while in Armbian we choose a somewhat high 10 minute value (commit=600).

 

So while with Armbian and 448 KB written in one hour almost three times as much data has been written at the block device layer it might be possible that the 156 KB written by the DietPi installation caused more wear at the flash layer below due to a phenomenon called Write Amplification (TL;DR version: writes at the flash layer happen at 'page sizes', usually 8K, and by using a high commit interval somewhat larger data chunks will be written only every few minutes which can result in significantly less page writes at the flash layer compared to writing every few seconds smaller chunks of data. Adding to the problem once a card is 'full' now we're talking about much higher Write Amplification since now not just pages are written but usually whole Erase Blocks are affected that are much larger. So please choose your SD card wisely and always use a much larger capacity than needed since there's no TRIM with SD cards in Linux!)

 

It would need a lot of more detailled analysis about this write behaviour but IMO it's not worth the efforts and Armbian's 10 min commit interval does a great job reducing further SD card wearout (anyone with too much spare time? Grab 'iostat 5' and 'iotop -o -b -d5 -q -t -k | grep -v Total' and start to analyse what's happening at the block device and application layer forgetting about the filesystem layer in between!)

 

So where's some room for improvement when comparing our defaults with DietPi's?

 

Share this post


Link to post
Share on other sites
1 hour ago, Igor said:

 

I was more talking about bundling all this compiler stuff with default installations. Just a quick check which packages consume most space as already done a year ago:

 

root@rock64:/etc/cron.d# wajig large
Package                           Size (KB)        Status
=================================-==========-============
libc6-dev                           11,015      installed
locales                             13,684      installed
binutils                            14,288      installed
libstdc++-5-dev                     14,832      installed
cpp-5                               15,474      installed
iso-codes                           16,035      installed
g++-5                               16,580      installed
systemd                             16,872      installed
perl-modules-5.22                   17,260      installed
gcc-5                               17,622      installed
libperl5.22                         19,863      installed
git                                 20,868      installed
vim-runtime                         26,870      installed
libicu55                            30,046      installed
unicode-data                        31,681      installed
linux-image-rk3328                  55,808      installed
linux-firmware                     207,968      installed
root@rock64:/etc/cron.d# wajig large | awk -F" " '/installed$/ {print $1}' | while read ; do
> echo -e "${REPLY}: \c"
> apt-get --assume-no --purge remove "${REPLY}" | grep "be freed" | awk '{print $4, $5}'
> done
libc6-dev: 50.4 MB
locales: 14.1 MB
binutils: 51.5 MB
libstdc++-5-dev: 32.2 MB
cpp-5: 51.0 MB
iso-codes: 16.7 MB
g++-5: 17.0 MB
systemd: 23.5 MB
perl-modules-5.22: 67.5 MB
gcc-5: 35.1 MB
libperl5.22: 49.8 MB
git: 21.4 MB
vim-runtime: 29.8 MB
libicu55: 47.3 MB
unicode-data: 32.4 MB
linux-image-rk3328: 57.1 MB
linux-firmware: 213 MB

 

 

Edit: This is how it looks like with DietPi on Rock64:

 

root@DietPi:~# wajig large
Package                           Size (KB)        Status
=================================-==========-============
systemd                             11,739      installed
firmware-brcm80211                  12,605      installed
coreutils                           14,572      installed
linux-image-4.4.77-rockchip-ayufan-136   51,280      installed
linux-headers-4.4.77-rockchip-ayufan-136   69,955      installed
root@DietPi:~# wajig large | awk -F" " '/installed$/ {print $1}' | while read ; do
> echo -e "${REPLY}: \c"
> apt-get --assume-no --purge remove "${REPLY}" | grep "be freed" | awk '{print $4, $5}'
> done
systemd: 11.4 MB
firmware-brcm80211: 12.9 MB
coreutils: 16.6 MB

 

Share this post


Link to post
Share on other sites

OK. I left dev stuff in by default and only removed vim package, change apt list to compressed one and tweaked image size (overhead from +40% down to +10% for CLI images):

[ o.k. ] Preparing image file for rootfs [ tinkerboard stretch ]

[ o.k. ] Current rootfs size [ 753 MiB ]
[ o.k. ] Creating blank image for rootfs [ 961 MiB ]

I am not sure if 10% is enough ... perhaps 15% will be right number.

Share this post


Link to post
Share on other sites

Both comparisons is what I call a 'goal oriented report'. :P @Fourdee picked topics where DietPi shines and you picked up topics where armbian shines...  I avoid to decide who did it more objective, as a 'never used dietPi' guy my opinion wouldn't be objective. And I've a preference for 'not that much' customized systems cause it makes it easier to fix things with informations besides the official forum of a distribution. 

 

It's obvious that armbian outperforms dietPi on a kernel level, there's more knowledge here on a devs side and it's part of the project since the beginning whereas DietPi never touched it (correct me if I'm wrong).

Both projects have different goals:

 

DietPi is minimalistic and tries to be unique, they purge out everything from debian which they think is not important and fill it with their scripts for an easy to install things for a use case. As a result you get a minimalistic OS. 

 

To quote @zador.blood.stained:

Quote

I see it as Ubuntu/Debian + bootloader with customizations + available kernel(s) with customizations + minimal OS (userspace) customizations and some optional tools/scripts like armbian-config.

 

What we can learn (personal opinion):

- 'customer informations'

From the rumors I read, their approach seems to be more 'beginners friendly' - use the dietPi scripts to install *random feature* and it will work 'out of the box'. Their tutorials are similar to their OS - minimalistic. You want a 'DietPi - Kodi + BitTorrent Combo':

Step1 --> Step2 --> Step3 etc. 

This is for sure more user friendly that our tutorials section in the forum whereas sometimes 2-3 pages of discussions are filled. I personally appreciate that we allow our users to discuss after a tutorial, a tutorial can be improved things that are wrong can be corrected or different opinions can be discussed but the starting topic of the tutorial should be updated with the new findings, maybe this needs some stronger moderation?

DietPi.com is 'more informative' than ours or at least they tell their userbase better why you should use DietPi instead of a *random Distribution*. Whenever this informations are correct or not is out of my knowledge I don't have the time and endurance to check this statement by statement. It's also of no interest to me cause I don't use DietPi. But IMO they explain better what they are than we do. Before you insist that they don't explain well what they aren't - that wasn't part of my statement... :P

I guess and hope that our new web design allows it to improve this slightly so that beginners read through tutorials beginners guides etc. before they start the same questions again and again.  

- 'public relations' (edit: you can also call it project presentation - but I like those buzz word bs bingo.. :D )

DietPi is more visible in reviews, reports etc. It might be easier to get attention when you do things different to other distributions as when you try to be a 'boring debian/ubuntu' with just better performance than the images provided by the boardmaker. Maybe we should make our efforts more visible. Most devs/researcher hate public relations as I do too but it's 'part of the deal'. When nobody recognizes what you do better than your competitor 'the world' doesn't recognize your work. Our release history is quite minimalistic which makes it fast to read but not very informative. E.g.:

Quote

v5.37 / 23.1.2018

  • bugfix release
  • armbianmonitor -u fix
  • ...

 

What is fixed in armbianmonitor -u? I know that I get all those informations when I look through the forum and through the GitHub history, but we must accept that the average user is to lazy to do it. It's not that every small change needs a 200 words comment what we fixed but a 'predictable place' where you find those announcements could be helpful. 

E.g.:

Quote

*random SBC* reaches wip status:

what works:

-boot via SD-Card

-Ethernet

-*random feature*

what doesn't works/ not tested:

-wifi

-installation to eMMC

-*random feature*

When Bloggers, news sites etc. know where they can pick up those informations they might talk more about the project. In case we want to be more visible, we have to spend more time in summarizing our efforts, if this on minor interest the current approach is fine. When I read through @Igor website (the whole posts are not readable anymore and for sure a bit outdated), you spend more time explaining what you did than now. It's clear that the project growth a lot and your workload is for sure high enough that you don't have time to do all of this on your own. Maybe we can make a call if someone wants to fill this position?

 

 

3 hours ago, Igor said:

OK. I left dev stuff in by default and only removed vim package, change apt list to compressed one and tweaked image size (overhead from +40% down to +10% for CLI images):

Is this more a reaction to @Fourdee decision to drop 'armbian based' dietPi images or is it driven by 'rational reasons'? On SD-card driven SBCs image size is IMO not relevant.. (ok, some 8gb large images from the boardsupplier are annoying but armbian isn't that big :P) You don't find any new reliable SD-card below 16GB (and to my knowledge no device has less than 8GB eMMC which should also be enough for our images). 10% more or less aren't that important. IMO the SD-Card shouldn't be used as datastorage (but that's a personal opinion too). 

Wouldn't it make sense to explain our users how you can purge all the stuff they don't need with the buildscript - means that we explain how you manipulate this part of the buildscript with a clear reminder that we don't suggest it for the *average user* and that we don't provide support for it when you run into trouble after doing it. Packages should be removed by reasons, e.g. a driver which affects reliability of the system (talking about XR819 driver... :P ) or it forces the user to do stupid things (that's where you're the experts knowing it better).

 

huh... It got a bit long... 

Share this post


Link to post
Share on other sites

Since I was mentioned in this thread a couple times already, as much as I wanted to avoid such discussions...

 

4 minutes ago, chwe said:

- 'public relations'

please don't make me start on "public relations". I missed the Twitter arguing mentioned in the OP but I don't think that throwing crap at other distribution in the field where winning a "competition" means almost nothing is a good example of "public relations"

 

8 minutes ago, chwe said:

DietPi is minimalistic and tries to be unique, they purge out everything from debian which they think is not important and fill it with their scripts for an easy to install things for a use case. As a result you get a minimalistic OS. 

DietPi is "diet". Whether it is a healthy diet or not is a good question, but I would call it lightweight, not minimalistic. It contains too many custom scripts on top of the base distribution to be minimalistic IMO.

 

Regarding why people may choose DietPi over other images - this is what I noticed while reading Orange Pi related threads on another forum

  • Read-only rootfs with overlayfs (or whatever is used) and smaller base image size is more robust for bad and slow SD cards
  • Software installer and helper scripts are more user (or should I say "noob") friendly
  • As it was said already - better project presentation (website, tutorials)

 

6 hours ago, tkaiser said:

Maybe it's worth to adjust the rootfs partition size calculation to use slightly less so the uncompressed image size can be a little bit smaller?

 

6 hours ago, tkaiser said:

So where's some room for improvement when comparing our defaults with DietPi's?

...

Is this really worth doing right now, just for the purpose of throwing dirt at each other (I mean DietPi <-> Armbian) as I see it?

Share this post


Link to post
Share on other sites
33 minutes ago, zador.blood.stained said:

Is this really worth doing right now, just for the purpose of throwing dirt at each other (I mean DietPi <-> Armbian) as I see it?

 

Nope, you're right and apologies. While I tried hard to focus on 'doing it better' you nailed it that I was annoyed by being confrontated again with bizarre 'performance metrics' like those from their Twitter conversation.

 

Trying to calm down now and focus on our problems (the many 5.38 update issues being almost enough for me to throw the towel)

Share this post


Link to post
Share on other sites
15 hours ago, tkaiser said:

While I tried hard to focus on 'doing it better' you nailed it that I was annoyed by being confrontated again with bizarre 'performance metrics' like those from their Twitter conversation.

I have to note that dietpi.com front page claims optimizations towards disk space, memory consumption and CPU usage by the OS, not overall CPU, network and I/O performance in real world scenarios which you are interested the most. While it's easy to show that counting processes or checking RAM usage produces numbers without meaning, I don't think we should focus on that (arguing related activities).

Share this post


Link to post
Share on other sites
39 minutes ago, zador.blood.stained said:

I have to note that dietpi.com front page claims optimizations towards disk space, memory consumption and CPU usage by the OS, not overall CPU, network and I/O performance in real world scenarios which you are interested the most.

 

Oh, I'm also very much interested in reliable operation and one of the key aspects with SBC is reducing wear on flash media like SD cards since at least I (being a low consumption fetishist) want to put the rootfs on energy efficient flash storage and not more 'wasting' SSD or HDD storage.

 

@Fourdee's 'Why DietPi is better' list http://dietpi.com/phpbb/viewtopic.php?f=9&t=2794 shows at least one very dangerous misunderstanding about Linux' VM / page cache implementation but at least that explains his strange conclusions in his Tests 1) and 4). While this is only dangerous for DietPi users it's a bit tough not trying to explain why 'dirty pages' do NOT 'show wait times, before the requested memory data can be written to disk. This generally means bottle-necking has occurred and the overall system performance will be effected until the data is written'. In contrast dirty pages are exactly what we want to reduce wear on flash memory and result of our way larger commit interval at the fs / block device layer. Same with the comparison of the log attempts... once DietPi users switch to full logging (rsyslog + logrotate) their flash media is at risk while it's possible to combine rsyslog + logrotate + logs in RAM.

 

Speaking of that: Do you have an opinion on improving log2ram? https://forum.armbian.com/topic/6626-var-easily-get-full-with-log2ram/?tab=comments#comment-50286

 

Share this post


Link to post
Share on other sites
6 minutes ago, tkaiser said:

Speaking of that: Do you have an opinion on improving log2ram? https://forum.armbian.com/topic/6626-var-easily-get-full-with-log2ram/?tab=comments#comment-50286

I'm starting to regret that we added it in the past, and we need something that works reliably regardless of the daily log size. I have some thoughts, but different ideas have different requirements and expected reliability issues, and anything requires extensive tests.

Share this post


Link to post
Share on other sites

As a dev I vote to keep the dev packages in. Sure is nice to have the build tools (even git client) installed even if things like libtool and pkg-config are missing from the base. If you are going for a minimal install I understand stripping all this stuff out, but I'm not sure how important it is to have a 700K vs 1.6 G image in today's terms. As @tkaiser points out most people have moved on beyond 4G SD cards. I've been using 32G for years and the price/performance is good for what I need it for.

 

I've have some home made security cameras that write/read/delete 100s of movies and images a day 24/7 for years without failure. I realize there may be more intensive usages scenarios (more write intensive), but at the current pricing levels I'll toss the SD out every few years if I have to. As with all things the lowest common denominator or race to the bottom isn't always the best strategy. Stability and standardization are more important to me then a little extra RAM, a little more SD life, etc. Not that these are not important, but the bigger picture I think is more important to focus on.

Share this post


Link to post
Share on other sites
1 hour ago, sgjava said:

most people have moved on beyond 4G SD cards

 

I completely agree. I think, if a desire to be more minimalist exists, that we create those images specifically labelled as such, for devices that fit the case (OPi IoT, nanopi NEO, Duo, etc).  Otherwise I don't see the value.

 

My honest reaction to the discussion is that it fits perfectly with the "What is Armbian?" discussion.  I don't think playing games to save a few megabytes fits something we really care about, it isn't really our problem if an extreme subset of users wants to pull the 512 MB SD card out of their 10 year old Sandisk Sansa MP3 player and want to put Linux on it...

 

There are 2 main deliverables that I see:  A build system for people to make their flavor of Debian for their needs, and our pre-packaged images.  IMHO, if we want to make it more "lean", this can be an option available in the build system deliverable, and if we want to make questionably valuable tiny images, then we can do that in parallel with the current desktop and server images.

Share this post


Link to post
Share on other sites
16 hours ago, TonyMac32 said:

I don't think playing games to save a few megabytes fits something we really care about, it isn't really our problem if an extreme subset of users wants to pull the 512 MB SD card out of their 10 year old Sandisk Sansa MP3 player and want to put Linux on it...

 

Well, as already explained above: Since Armbian is a build system everyone really keen on using a really small installation media can do (512 MB is not a great idea but possible). Simply by switching from ext4 to btrfs and deleting the linux-firmware package from default package list.

Using btrfs works quite well (all the thousands of Armbian based OMV installations run this way) and we support the same when transferring Armbian from SD card to eMMC or a disk with nand-sata-install.

Share this post


Link to post
Share on other sites
4 minutes ago, tkaiser said:

When leaving out the linux-firmware package 200 MB less garbage get installed (no idea why this package is included in the first place but I stopped to join any firmware related discussions already since too frustrating, it seems it's up to @Igor to decide about anything firmware related)


It looks like a bug to me or bugfix due to problems related to our dedicated (smaller) armbian-firmware package 

 

linux-firmware package is only for Xenial ... I am not sure. Will check a little.

Share this post


Link to post
Share on other sites
20 hours ago, sgjava said:

I've been using 32G for years and the price/performance is good for what I need it for.

 

I would call the price/performance not good but simply awesome today given we get ultra performant cards like the 32GB SanDisk Ultra A1 for as low as 12 bucks currently: https://www.amazon.com/Sandisk-Ultra-Micro-UHS-I-Adapter/dp/B073JWXGNT/ (I got mine 2 weeks ago for 13€ at a local shop though). And the A1 logo is important since cards compliant to A1 performance class perform magnitudes faster with random IO and small blocksizes (which pretty much describes the majority of IO happening with Linux on our boards).

 

As can be seen in my '2018 A1 SD card performance update' the random IO performance at small blocksizes is magnitudes better compared to an average/old/slow/bad SD card with low capacity:

            average 4GB card   SanDisk Ultra A1
  1K read         1854              3171
  4K read         1595              2791
 16K read          603              1777
 1K write           32               456
 4K write           35               843
16K write            2               548

With pretty common writes at 4K block size the A1 SanDisk shows 843 vs. 35 IOPS (IO operations per second) and with 16K writes it's 548 vs. 2 IOPS. So that's over 20 or even 250 times faster (I don't know the reason but so far all average SD cards I tested with up to 8 GB capacity show this same weird 16KB random write bottleneck -- even those normal SanDisk Ultra with just 8GB). This might be one of the reasons why 'common knowledge' amongst SBC users seems to be trying to prevent writing to SD card at all. Since the majority doesn't take care which SD cards they use, test them wrongly (looking at irrelevant sequential transfer speeds instead of random IO and IOPS) and chose therefore pretty crappy ones.

 

BTW: the smallest A1 rated cards available start with 16GB capacity. But for obvious reasons I would better buy those with 32GB or even 64GB: price/performance ratio is much better and it should be common knowledge that buying larger cards 'than needed' leads to SD cards wearing out later.

 

Share this post


Link to post
Share on other sites
46 minutes ago, zador.blood.stained said:

IMO "linux-firmare" in Ubuntu based images should be installed by default as it contains firmware for popular wireless and multimedia devices. Solving random firmware related problems is a complicated task for our user base.


Unpacked linux-firmware is around 250Mb, while our custom firmware is 8Mb and it covers onboard + most popular ones. Since we have armbian-config in action: "Install full firmware package" should solve this problem.

Share this post


Link to post
Share on other sites
On 3.3.2018 at 6:30 PM, sgjava said:

I've have some home made security cameras that write/read/delete 100s of movies and images a day 24/7 for years without failure

 

Please be careful since this is an entirely different workload compared to 'using SD cards for the rootfs' since storing images and video is more ore less only sequential IO coming with a write amplification close to the optimum: 1. This means that the amount of data written at the filesystem layer, the block device layer and the flash layer are almost identical.

 

With our use cases when running a Linux installation on the SD card this looks totally different since the majoritiy of writes are of very small sizes and write amplification with such small chunks of data is way higher which can result in 1 byte changed at the filesystem or block device layer generating a whole 8K write at the flash layer (so we have a worst case 1:8192 ratio of 'data that has changed' vs. 'real writes to flash cells')

 

Please see here for the full details why this is important, how it matters and what it affects: https://en.wikipedia.org/wiki/Write_amplification

 

Our most basic take on that in Armbian happens at the filesystem layer due to mount options since we use 'noatime,nodiratime,commit=600' by default. What do they do?

  • noatime prevents the filesystem generating writes when data is only accessed (default is that access times are logged in filesystem metadata which leads to updated filesystem structures and therefore unnecessary writes all the time filesystem objects are only read)
  • nodiratime is the same for directories (not that relevant though)
  • commit=600 is the most important one since this tells the filesystem to flush changes back to disk/card only every 600 seconds (10 min)

Increasing the commit interval from the default 5 to 600 seconds results in the majority of writes waiting in DRAM to be flushed to disk only every 10 minutes. Those changes sit in the Page Cache (see here for the basics) and add as so called 'dirty pages'. So the amount of dirty pages increases every 10 minutes to be set to 0 after flushing the changes to disk. Can be watched nicely with monitoring tools or something simple as:

watch -n 5 grep Dirty /proc/meminfo

While @Fourdee tries to explain 'dirty pages' would be something bad or even an indication for degraded performance it's exactly the opposite and just how Linux basics work with a tunable set for a specific use case (rootfs on SD card). To elaborate on the effect: let's think about small changes affecting only 20 byte of change every minute. With filesystem defaults (commit interval = 5 seconds) this will result in 80KB written within 10 minutes (each write affects at least a whole flash page and that's AFAIK 8K or 16K on most cards so at least 8K * 10) while with a 10 minute commit interval only 8KB will be written. Ten times less wear.

 

But unfortunately it's even worse with installations where users run off low capacity SD cards. To my knowledge in Linux we still have no TRIM functionality with MMC storage (SD cards, eMMC) so once the total amount of data written to the card exceeds its native capacity the card controller has no clue how to distinguish between free and occupied space and has therefore to start to delete (there's no overwrite with flash storage, see for example this explanation). So all new writes now might even affect not just pages but whole so called 'Erase Blocks' that might be much larger (4MB or 16MB for example on all the cards I use). This is for example explained here.

 

In such a case (amount of writes exceed card's native capacity) we're now talking about writes affecting Erase Blocks that might be 4MB in size. With the above example of changing 20 bytes every minute with the default commit interval of 5 seconds at the flash layer now even 40 MB would be written while with a 10 min commit interval it's 4MB (all with just 200 bytes having changed in reality).

 

So if you really care about the longevity of your card you buy good cards with capacities much 'larger than needed', clone them from time to time to another card and now perform a TRIM operation manually by using your PC or Mac and SD Association's 'SD Formatter' to do a quick erase there. This will send ERASE (CMD38) for all flash pages to the card's controller which now treats all pages as really empty so new writes to the card from now on do NOT generate handling of whole Erase Blocks but happen at the page size level again (until the card's capacity is fully used, then you would need to repeat the process).

 

There's a downside with an increased commit interval as usual and that affects unsafe power-offs / crashes. Everything that sits in the Page Cache and is not already flushed back to disk/card is lost in case a power loss occurs or something similar. On the bright side this higher commit interval makes it less likely that you run into filesystem corruption since filesystem structures are updated on disk also only every 10 minutes.

 

Besides that we try to cache other stuff in RAM as much as possible (eg. browser caches and user profiles using 'profile sync daemon') and same goes for log files which are amongst those candidates that show worst write amplification possible when allowing to update logs every few seconds on 'disk' (unfortunately we can't throw logs just away as for example DietPi does it by default so we have to fiddle around with stuff like our log2ram implementation showing lots of room for improvements) 

Share this post


Link to post
Share on other sites

In case we want to explain what armbian is.... Your post should be linked under something like 'durability'. We aren't that bad to explain where armbian differs from other 'distributions' for SBCs... we just suck to make this info 'easy accessible'. :P 

Edit: It's not about throwing dirt to others. It's about explaining why we do the things the way we do and why. 

Share this post


Link to post
Share on other sites
On 4.3.2018 at 4:38 PM, chwe said:

In case we want to explain what armbian is.... Your post should be linked under something like 'durability'. We aren't that bad to explain where armbian differs from other 'distributions' for SBCs...

 

This is not that much about Armbian. This is just about flash storage basics!

 

But no one wants to read through the above or such complex and boring stuff like this: https://forum.armbian.com/topic/6444-varlog-file-fills-up-to-100-using-pihole/?do=findComment&comment=50833 -- no one wants to deal with the differences between the affected layers (filesystem, block device, flash memory). Users usually prefer simple answers (even if wrong) over complex ones.

 

That's why a distro should choose sane defaults (a trade-off as usual since a high commit interval also affects the amount of data lost after a crash) and that's why educating users is needed. But it seems Armbian is one of the few places where that happens (or maybe even just my essays here in the forum?).

 

The problem with SD cards and SBC is the huge amount of ignorance and stupidity we all have to deal with. People are even today told to focus only on BS performance metrics (e.g. sequential read and write performance instead of the only thing that's really important: random IO performance, here especially writes and this especially with small to medium blocksizes).

 

People are even today encouraged to use the most shitty SD cards possible instead of telling them the truth to better buy a new and large enough one with great random IO performance. Since an awful lot of tasks depends on sufficient (random) IO performance and you simply won't be happy with a crappy SD card unless you use it read-only.

 

Share this post


Link to post
Share on other sites
4 hours ago, tkaiser said:

This is not that much about Armbian. This is just about flash storage basics!

And that's exactly what Armbian is! For someone not familiar with this basics, Armbian take care about it - even if you don't know that this is an issue. It's not that you can't change this behavior and do 'stupid things' with armbian. :P But a 'stock' Armbian tries to avoid it.

4 hours ago, tkaiser said:

The problem with SD cards and SBC is the huge amount of ignorance and stupidity we all have to deal with. People are even today told to focus only on BS performance metrics (e.g. sequential read and write performance instead of the only thing that's really important: random IO performance, here especially writes and this especially with small to medium blocksizes).

That's why we should have your essays collected somewhere. :lol: For me the best way would be that they are written somewhere as a 'tutorial'/'education' part.. But links to the forum posts could also work. 

 

For me, Armbian is:

  • clean Debian/Ubuntu (due to debootstrap processing of rootfs) on top of an enhanced BSP/mainline kernel (due to a bunch of patches)
  • enhanced durability (due to log2ram & longer 'commit interval')
  • one of the few distributions providing kernel updates on 'living systems' for arm boards (which goes sometimes wrong :P - but we try to fix it in case it goes wrong, and yes there is some space for improvements)
  • an active community trying to help you when you struggle - even your issue is not Armbian related/ due to lack of (basic) linux knowledge etc. 
  • providing a similar linux experience over different SoCs/'brands' - the reason I'll never take use an image provided by the boardmaker (e.g. friendlyArm) is not because I think all of them do a bad job, it's because I don't want to waste hours of time to figure out where their behavior is different from the images provided by another boardmaker/distribution.  
  • a mighty buildscript which allows the user to fix things on his own (e.g. @TonyMac32 and me try to support CSI for the tinker, we still fail :P but the buildscript allows me to create different kernel configs & patches for the kernel/u-boot etc without doing the annoying diff process on my own.. I can simply do my changes inside the sources and as soon as I fixed something I can send a PR to armbian and other people can benefit from my enhancement or in case Armbian doesn't accept the PR I can provide it as an userpatch so that everyone interested in the same 'enhancement' can build it on his own). 

I agree that part of the people are ignorant when it comes to durability and 'proper usage' of SBCs. I also agree that user expectations are sometimes ridiculous (I want a 8$ SBC together with a 1$ PSU and a 2$ SD-card and I expect desktop performance when it's about multimedia stuff with linux! :lol:). I'm not sure if this is the majority, I guess that there's a 'silent majority' of users which are happy with what Armbian offers to them.  Normally you don't register to a Forum just to say "Awsome! Thank you for providing an reliable Debian/Ubuntu for my *random SBC*!" It's more than you register cause you something doesn't work as expected, cause it doesn't boot anymore or cause you bought a new SBC which is not supported (yet) by Armbian. 

As a moderator here, I get all these reminders to approve new forum postings and therefore I read a bunch of them... Some of them are funny, some of them are stupid a few of them are obviously spam but even less of them are 'impressive'. The last one I can remember was when @konsgn posted his first post where he nailed down the OPi0 thermal issue with his first post. So why did he wrote this here and not in Xunlongs forum (in case you did, mea culpa for my low google-fu :P, I only found it here)? Maybe he's one of the 'silent' users which are (mostly) happy with what armbian offers to them, maybe cause he thought it make sense to share it in an active community - I don't know but it seems we are a good place for sharing such informations.. :) 

 

We can provide those informations in a 'marketing manner' and I'm sure @zador.blood.stained wouldn't be happy when we do it this way. :P We can do it the way we do it at the moment, explaining the same questions again and again (and again :P ) to our 'customers'  or we can try to collect them so that they are accessible 'as easy as possible'. 

Share this post


Link to post
Share on other sites
On 3/10/2018 at 12:43 PM, tkaiser said:

But no one wants to read through the above or such complex and boring stuff like this: https://forum.armbian.com/topic/6444-varlog-file-fills-up-to-100-using-pihole/?do=findComment&comment=50833 -- no one wants to deal with the differences between the affected layers (filesystem, block device, flash memory). Users usually prefer simple answers (even if wrong) over complex ones.

 

Oh, I do! I'm even eating Bierstangerl while reading your post trying to grasp up some straws of knowdlege (maybe some of them will stick).

Just registered because I had to add that ... have been reading your posts for some months now.

 

Haven't flashed armbian yet tough, hench why I didn't feel the need to register yet. But oh, well, now I'll just do both ...

 

----

 

Edit:

 

So I've tried the DietPi and the Armbian Nightly build and the idle and load temps are up to 20° higher on DietPi even when I switch the governor / frequencies in DietPi (they're correctly applied).

 

That is for my NanoPi Neo2 - DietPi 4.14.y kernel idle after about 10 hours in the stock 3D print case with heatsink attached has an idle temp of about 60° - compared to armbian nightly with 4.18.y kernel - 40°.

 

Both running pi-hole only so far, all updates applied. Same config.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
0