zram vs swap

[tkaiser]

I added a while back 'free' output to 'armbianmonitor' and 'armhwinfo' to get some monitoring data from real-world installations about swap usage. Armbian uses an emergency swap file of 128MB by default (vm.swappiness=0) which means swapping should be avoided unless really needed. Let's have a look and extract data from +50 support URLs collected within the last months.

 

I piped the output of https://pastebin.com/raw/DjxAX6Rn through 

while read ; do curl -q "${REPLY}" | grep '^Swap: '; done | sort | uniq -c

and this is what we end with:

 175 Swap:          127M          0B        127M
   1 Swap:          127M         22M        105M
   1 Swap:          127M        256K        127M
   3 Swap:          2.0G          0B        2.0G
   1 Swap:          513M          0B        513M
   1 Swap:          617M          0B        617M

So we have 182 times swap usage monitored and get 1 single occasion when swapping occured (the 256KB case here was not swapping but zram instead and the 22MB case there is the only occurence of real swapping and also an 'Out of memory: Kill process' event has been logged with kernel 4.11, no idea what @op1tjaap did back then).

 

IMO this justifies overthinking our approach especially on boards with low memory since zram is an alternative. Currently there are a few hundred Armbian installations out there running Jessie with Ubuntu's zram-config package installed (zero swap configured, vm.swappiness=0, zram-config defaults resulting in zram area half the size of physical available memory). Not a single complaint so far. But vm.swappiness=0 is a pretty bad setting for such a configuration since as we see above swapping will be avoided anyway if possible.

 

When using vm.swappiness=100 available DRAM will be used way more efficient. But I've to admit that in my testings available memory has never been an issue anyway. So what about the following:

• add zram-config to Xenial default package list
• modify /etc/init.d/firstrun to skip swap file creation if this package is installed and branch=next|dev and set vm.swappiness=100 instead of vm.swappiness=0

 

This way we can collect within half a year some details about zram usage (count of complaints and again collecting free output through 'armbianmonitor -u' output)

 

 

Edit: Added mainline kernel requirement

[zador.blood.stained]

36 minutes ago, tkaiser said:

modify /etc/init.d/firstrun to skip swap file creation if this package is installed and set vm.swappiness=100 instead of vm.swappiness=0

We can leave this as is but check swap priorities. I remember reading about zram having higher priority than default file/partition based swap, but it needs confirming.

[tkaiser]

12 minutes ago, zador.blood.stained said:

We can leave this as is but check swap priorities.

If we leave it 'as is' then we can stop here since vm.swappiness=0 results in neither zram nor swap used unless really needed (and then there are no positive effects and we can leave everything as is).

 

No idea how vm.swappiness was set for the 'Swap: 127M 256K 127M' entry above -- maybe @dragonlost can comment on this? At least this here was me having activated zram with vm.swappiness defaults (AFAIK that's 60). As can be seen there's slight zram usage with zero swap but I can't remember exactly what I did back then (since 4 SATA devices are attached most probably ZFS/RAID tests). But with a default vm.swappiness value we might end up with swap on SD card being really used even in situations where 'cold pages' sit compressed in zram and the host is swapping to death with hot pages on SD card. This is nothing I would roll out as part of a new Armbian release. So IMO the only reasonable test scenario is dropping swap entirely, using only zram and vm.swappiness=100

 

To sum it up:

• vm.swappiness=0 --> no need to test since behaviour not different than today. Hosts limited to real physical memory, only emergency behaviour when really running out of memory would change
• vm.swappiness default (60?) --> very interesting to test with swap and zram but I fear negative behaviour once paging reaches SD card (see also 'RAM + ZRAM + SWAP (on HDD)' section here)
• vm.swappiness=100 --> only useful without swap configured. As we can see above at least the 53 'participating' parties ran really out of memory only once ( @op1tjaap might remember what happened back then)

I'm not interested in exploring better 'emergency situation' behaviour but in enabling SBC with low memory to make better use of available DRAM. I mean: that's what zram has been invented for on Android devices: Cope with low memory situations and avoid swapping to slow flash. And it works in the meantime on hundreds of millions of devices (and a few hundred -- counting Raspberries too then even a few thousand -- Armbian installations running OMV)

[tkaiser]

Another quick check: grepping for 'Out of memory' in all debug logs:

 

[ 1374.514623] Out of memory: Kill process 5386 (supertuxkart) score 523 or sacrifice child (also an occurence of real swap usage I missed before since whole system configured to use french: 'Partition d'échange')

[ 3790.835574] Out of memory: Kill process 2143 (python2.7) score 624 or sacrifice child (that's the other swapping occurence) 

 

I checked again (this time searching for '   127M  ') and we had 2 real swap scenarios in 182 cases and in both oom-killer 'resolved' the issue anyway.

[zador.blood.stained]

Just a quick check on a laptop (Ubuntu 17.10 x86_64, zram-config is installed) that I'm writing this post from

➜  ~  % swapon --show
NAME       TYPE        SIZE   USED PRIO
/dev/dm-0  partition 953,5M     0B   -1
/dev/zram0 partition 751,2M 181,5M    5
/dev/zram1 partition 751,2M 182,5M    5
➜  ~  % sysctl vm.swappiness
vm.swappiness = 60
➜  ~  % LANG=en free -h
              total        used        free      shared  buff/cache   available
Mem:           2.9G        2.1G        305M        249M        579M        555M
Swap:          2.4G        383M        2.0G
➜  ~  % 

IMO we may not need setting vm.swappiness to 100, as you can see zram will be preferred over disk/file based swap. "Once paging reaches SD card" means that zram is (almost) full and user chose a device with less RAM that is viable for the use case, at this point without zram the system would be out of memory already.

Though we need to test how different vm.swappiness values (i.e. 0, 60 and 100) interact with multiple swap devices, at which point swapping goes to a lower priority device - only when higher priority devices are 100% full or earlier?

Edited November 2, 2017 by zador.blood.stained
Replaced swapon -s (listed as deprecated in man) with swapon --show; added free -m

[tkaiser]

1 hour ago, zador.blood.stained said:

Though we need to test how different vm.swappiness values (i.e. 0, 60 and 100) interact with multiple swap devices, at which point swapping goes to a lower priority device - only when higher priority devices are 100% full or earlier?

 

Well, that might depend on a lot of other vm related stuff, most importantly few of the last ones on below list (that's a sysctl dump from XU4 4.9 kernel):

vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.compact_unevictable_allowed = 1
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
vm.drop_caches = 0
vm.extfrag_threshold = 500
vm.highmem_is_dirtyable = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 32	32
vm.max_map_count = 65530
vm.min_free_kbytes = 3454
vm.mmap_min_addr = 4096
vm.mmap_rnd_bits = 8
vm.nr_pdflush_threads = 0
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.stat_interval = 1
vm.swappiness = 100
vm.user_reserve_kbytes = 63391
vm.vfs_cache_pressure = 100
vm.watermark_scale_factor = 10

I think we all agree that swap hitting the SD card can be considered 'game over' especially when happening on 'average' SD cards (thosethe majority of users bought with 2-10 random write IOPS at 16K block size). Our strategy to deal with this in the past was to set vm.swappiness to 0 as a try to prevent swapping in any situation.

 

When we now still allow swap on SD card and increase swappiness then depending on kernel and other defaults we (or our users) might end up with situations where once the kernel is running out of zram swap devices it starts to swap to SD card (game over). As far as I understood this is not something comparable to 'tiered storage' where each storage layer is handled appropriately (and 'hot data' gets moved on demand) but this is just a couple of different swap devices with different priorities that are used based on a rather simple strategy. So if swap on SD card is allowd the system might use it for 'hot data' once the zram swap devices are full (containing partially 'cold data' then).

 

If we want to avoid 'swap on SD card' and also avoid testing through all different kernels and distros (that might tweak this or that somewhere -- see Ubuntu's ondemand service that interfered with our cpufreq settings) then IMO the easy way is to drop 'swap on SD card' entirely. We can assume this works pretty well with vm.swappiness=0 (since that's tested by OMV users in the meantime) so let's try to get some experiences with higher swappiness values.

 

 

[zador.blood.stained]

2 minutes ago, tkaiser said:

I think we all agree that swap hitting the SD card can be considered 'game over' especially when happening on 'average' SD cards (thosethe majority of users bought with 2-10 random write IOPS at 16K block size). Our strategy to deal with this in the past was to set vm.swappiness to 0 as a try to prevent swapping in any situation.

Still vm.swappiness=0 is not "preventing swapping in any situation", though the behavior was changed in kernel 3.5 (so we have slightly different logic between legacy 3.4 and all other kernels): https://github.com/torvalds/linux/commit/fe35004fbf9eaf67482b074a2e032abb9c89b1dd

This definitely doesn't say "prevents swapping in any situation"

swappiness

This control is used to define how aggressive the kernel will swap
memory pages.  Higher values will increase agressiveness, lower values
decrease the amount of swap.  A value of 0 instructs the kernel not to
initiate swap until the amount of free and file-backed pages is less
than the high water mark in a zone.

The default value is 60.

 

14 minutes ago, tkaiser said:

If we want to avoid 'swap on SD card' and also avoid testing through all different kernels and distros (that might tweak this or that somewhere -- see Ubuntu's ondemand service that interfered with our cpufreq settings) then IMO the easy way is to drop 'swap on SD card' entirely. We can assume this works pretty well with vm.swappiness=0 (since that's tested by OMV users in the meantime) so let's try to get some experiences with higher swappiness values.

I agree with dropping swap on SD for new images and using zram where it's possible. I'm just against adding any new swap related logic (except logging) to firstrun and armhwinfo - those 2 services have enough tweaks not related to each other and enough legacy and partially unmaintained code. vm.swappiness value can be tweaked by installing a file to /etc/sysctl.d/ in board support packages, though IMO for Ubuntu with zram only default value of 60 is good enough.

[tkaiser]

47 minutes ago, zador.blood.stained said:

Still vm.swappiness=0 is not "preventing swapping in any situation"

 

Yeah, so let's get rid of this rather sooner than later.

 

My personal timeline for the approach (expectations vs reality included  ):

1. Started to prepare this 7 months ago: https://github.com/armbian/build/commit/de059a12a9f8ab7f3b40ab8e2e3671030e5dd51e#diff-01333a069633e14fb9510b0dbedadab5
2. Naively thought 2 months later we would've collected enough reports to get a clue what's going on (I wanted to wait for 50 different logs) so that we can drop 'swap on SD card' with next release
3. Well, (not so) surprisingly I had to wait until now (7 months) to get those ~50 debug logs containing ~180 swap usage reports (and 2 times swapping really happened)
4. Since 2-3 months few hundred to thousand OMV users test 'no swap on SD card' successfully already but with settings that don't make better use of the physical memory

Now we know a little bit wrt 'out of memory' situations (I wouldn't call those ~50 debug logs representative but at least it's more than nothing) and that it seems to work pretty well to avoid swap on SD cards.

 

53 minutes ago, zador.blood.stained said:

I'm just against adding any new swap related logic (except logging) to firstrun and armhwinfo

 

But then any further testing is useless since how to get rid of the swap file then?

 

The reason why I want to (temporarely) implement this as proposed is that in Ubuntu Xenial it's as easy as adding 'zram-config' to a variable in our build system to get zram activated automagically. So all that's missing is not enabing the swap file and then collect real-world experiences from our users so that we can then (maybe in half a year) decide whether to use Ubuntu's zram-config package everywhere (works flawlessly with both Jessie and Stretch BTW) or we implement this as an own startup service.

 

I absolutely second the idea to move the vm.swappiness setting to a BSP contained file below /etc/sysctl.d/ but this does not solve the real problem to collect experiences now. At least in my testings (server and NAS use cases) I can not force excessive vm usage anyway. Seems the daemons I deal with are all well-behaving so we need some real-world experiences from people using SBCs as desktop and stuff like this.

 

So I still propose a simple firstrun logic change making creation of the swapfile optional based on existence of zram-config package. Once we collected new experiences this can be dropped altogether or reverted back to mandatory swapfile creation.

 

[zador.blood.stained]

5 minutes ago, tkaiser said:

But then any further testing is useless since how to get rid of the swap file then?

Collect stats from new images for now (where we are sure that we have recent enough kernel, userspace and board support package versions), maybe add a manual switch (option) to armbian-config. We can always (try to) push new settings to older images later, though I'm not a fan of silently making changes that will conflict with existing forum posts and documentation.

(I though that you want to push some logic to armhwinfo to disable/delete AD swap file on existing images.)

 

11 minutes ago, tkaiser said:

So I still propose a simple firstrun logic change making creation of the swapfile optional based on existence of zram-config package.

OK, I'm not against that (making the swap file logic and setting swappiness to 0 optional)

[zador.blood.stained]

To simplify rootfs build time caching we could add zram-config to all new images but control its startup by a systemd override file.

[tkaiser]

18 minutes ago, zador.blood.stained said:

I though that you want to push some logic to armhwinfo to disable/delete AD swap file on existing images.

 

Nah, only new images and clearly mentioning in release/changelog: 'Ubuntu Xenial images use zram instead of swap' (so anyone concerned about stuff not working as 20 years ago any more can choose Stretch for now). No modification of already existing images, the goal is to provide new images with a documented change.

 

17 minutes ago, zador.blood.stained said:

To simplify rootfs build time caching we could add zram-config to all new images but control its startup by a systemd override file.

 

Well, I wanted to activate the new behaviour on all boards by default with Xenial since it's pretty interesting to get feedback from Orange/NanoPi with just 256 MB as well as devices with 2 or even 4 GB. If vm.swappiness is controlled by a eg. /etc/sysctl.d/armbian-vm-tweaks from now on we can react quickly if we discover problems where vm.swappiness=60 doesn't fit (and since it's known that vm.swappiness=0 works even worst case scenarios can be fixed with an 'apt upgrade' on user's side)

[zador.blood.stained]

1 hour ago, tkaiser said:

If vm.swappiness is controlled by a eg. /etc/sysctl.d/armbian-vm-tweaks from now on we can react quickly if we discover problems where vm.swappiness=60 doesn't fit (and since it's known that vm.swappiness=0 works even worst case scenarios can be fixed with an 'apt upgrade' on user's side)

Adding sysctl files to the board support package will change this setting also for all previous images, so I thought about relying on kernel default value of 60. Looks like Ubuntu doesn't provide any tweaks for vm.* sysctl parameters except security related vm.mmap_min_addr.

[tkaiser]

On 2.11.2017 at 6:04 PM, zador.blood.stained said:

To simplify rootfs build time caching we could add zram-config to all new images

 

With latest commits it's now implemented like this:

• zram-config added to Xenial default package list
• firstrun checks for this package and if installed will neither create a swap file nor write vm.swappiness=0 to /etc/sysctl.conf
• nand-sata-install (hopefully) learned to deal with what's configured and will honor existing settings when transferring installation

So only new Xenial images are affected currently since there SD card based swap will be replaced by zram with default vm.swappiness. I did a lot of testing mostly with legacy kernels and to be honest find it really hard to 'measure' effects. With Pinebook I had to open 40 Chromium tabs to let Chromium eat up to 2.7 GB memory and as expected it just felt slow. But it feels slow all the time (compared to what I'm used to) so it's really hard to 'measure' any effects. I read about people adjusting vm.swappiness to much lower values happily reporting back that 'everything way better now' but based on user experience tests I wouldn't count on this and my personal tests resulted in me just thinking 'well, still slow'.

 

With my very own use cases (server tasks) I'm not able to get zram swap being used extensively, mostly since it seems I'm only using daemons that behave well. So we really need some experiences made by users out there.

 

In the meantime installation of zram-config package is not needed any more since I added everything this package does to an activate_zram function in armhwinfo (wrong place, I know -- reasons below). I took most of the functionality unaltered except a few details

• amount of memory used is the same (half of DRAM) and it's only configurable by adjusting armhwinfo code (done by intention, I really want to avoid dealing with problem reports from users adjusting everything to death while not even understanding virtual memory and paging basics)
• number of zram devices == count of CPU cores but limited to 4 (using all CPU cores works well on x86 but we don't want compression/decompression happening on A7 cores in big.LITTLE designs)
• set LZ4 as algorithm where possible (otherwise fallback to what the kernel is configured to use -- LZO today  -- will happen). This might need adjustments in the future if we switch to 'better' compression algos on a specific platform (eg. zstd -- an interesting read here)

So what's next (please keep in mind that I'm solely focused on a way to collect user experiences in the least 'expensive' (time consuming) way -- it kinda sucks how much efforts are needed to get good user feedback!

• With new Xenial images users are testing zram + vm.swappiness=60 behaviour
• Experienced users who know what they do can be encouraged by another forum thread to start experimenting with zram on their installations (it's as easy as uncommenting the 'activate_zram' line in armhwinfo, adjusting vm.swappiness in /etc/sysctl.conf and/or deleting the swap entry in /etc/fstab)
• Experienced users with interesting hardware setups (eg. running off pretty fast ODROID eMMC or a SATA SSD and not a slow SD card) can be encouraged to do even more testing (combining zram with swap on SSD for example with modified vm.swappiness)
• We can uncomment the 'activate_zram' line in armhwinfo in a few months causing no harm on existing installations since due to vm.swappiness=0 there nothing will change (only emergency swap behaviour will improve a lot)
• In half a year based on the collected experiences we can remove zram-config from Xenial package list and finally adjust firstrun to skip swapfile creation and adjusting vm.swappiness (or maybe we came up with a better value in the meantime and set this as default)

There's only a single change that affects already existing installations and that would be activate_zram enabled which only affects (improves) emergency swap behaviour. All other changes only affect new images/installations so users are safe.

 

BTW: What about renaming armhwinfo? For example armbian_hw_setup?

[zador.blood.stained]

20 minutes ago, tkaiser said:

BTW: What about renaming armhwinfo? For example armbian_hw_setup?

I would prefer to split it into multiple services: a logging service, an optimizing service, a hardware monitoring service, etc.

And instead of reinventing the wheel with zram we could use a systemd-swap service, i.e. this implementation from Arch Linux: https://github.com/Nefelim4ag/systemd-swap

 

This way it would be easy to disable individual components of Armbian that may not be needed or that may interfere with other services (i.e. your "activate_zram" function checks only for zram-config package and it would interfere with systemd-swap installed by users without an easy way to disable it (since any changes to armhwinfo will be lost on upgrade))

[tkaiser]

1 hour ago, zador.blood.stained said:

And instead of reinventing the wheel with zram we could use a systemd-swap service, i.e. this implementation from Arch Linux: https://github.com/Nefelim4ag/systemd-swap

 

Well, giving up on the approach now. It took 7 months to get at least some sparse information wrt swap usage on real-world Armbian installations and I clearly don't want to wait another few months to see any improvements.

 

Relying on systemd-swap or any other already existing 'solution' will not work that great 'on Armbian' since all those attempts are made only for x86 situation (requiring recent kernels, no big.LITTLE problem, zswap better than zram and so on). Emergency swap is fine...

 

Edit: systemd-swap's zram activation routine (which while being the only part of interest on SBC is disabled by default) needs to be rewritten entirely anyway to work properly or at all on Armbian.

[op1tjaap]

Sorry. No idea what I was doing. I know this doesn’t sound very intelligent. 

Will check if my “MacIPpi thing” caused more swapping. 

[Tido]

On 4.11.2017 at 11:57 AM, zador.blood.stained said:

I would prefer to split it into multiple services: a logging service, an optimizing service, a hardware monitoring service, etc.

Sound like a good Unix approach.

 

@tkaiser, I remember that you once explained to Wildcat_Paris that if his system is using SWAP - there is something wrong. Whereas it doesn't hurt to have that little partition - it also supports to analyze if your SBC goes crazy, right?

If you look at it from your side - I might have stumbled across it, but what is the major benefit of leaving away this SWAP partition?

[tkaiser]

26 minutes ago, Tido said:

Sound like a good Unix approach.

 

So when do you start to implement this? You won't just like no one else will anytime soon or ever and if this is the requirement to switch from swap to zram (as it seems) then it's as I already said:

On 4.11.2017 at 1:08 PM, tkaiser said:

Emergency swap is fine...

Since it seems I'm the only one interested in improving DRAM usage on small boards and wasted already way too much time on this approach I've just given up. And if you have to ask the question you asked I can only recommend to read the thread you posted to since it's explained in detail.

[zador.blood.stained]

12 minutes ago, tkaiser said:

You won't just like no one else will anytime soon or ever and if this is the requirement to switch from swap to zram (as it seems)

It's a requirement to have a maintainable code and not a 500+ lines of code in a single service. Especially outdated code, since starting with kernel 4.2 it's possible to reconfigure zram on the fly without unloading the module and new distributions (unfortunately not Jessie but Xenial, Stretch and newer) provide a "zramctl" utility that can configure zram devices and display useful statistics.

[Tido]

I found it now:

"I'm not interested in exploring better 'emergency situation' behaviour but in enabling SBC with low memory to make better use of available DRAM. I mean: that's what zram has been invented for on Android devices: Cope with low memory situations and avoid swapping to slow flash."

 

In this thread I don't understand why VM (virtual machine (I guess)) is always a topic on SBC. That said, it is hard for me to follow in your conclusion. What I can see though

 

Whereas Zador is afraid of implementing problems to armbian:

This way it would be easy to disable individual components of Armbian that may not be needed or that may interfere with other services (i.e. your "activate_zram" function checks only for zram-config package and it would interfere with systemd-swap installed by users without an easy way to disable it (since any changes to armhwinfo will be lost on upgrade)).

 

TK wants to take measure with regard to fears of Zador by:

Experienced users who know what they do, can be encouraged by another forum thread to start experimenting with zram on their installations

 

Sounds to me, like you both want the best - and while I was writing this Zador added the above posting.

[Regis Michel LeClerc]

Hi,

 

Well, I'm not sure if it's a bug or what (when gcc crashes, it says to report it as a bug), but I was trying to compile Monero on an Orange Pi PC2 (yeah, pointless, I know, but just for the PoC) and the memory (and lack of swap) came up. Here's my config:

root@orangepipc2:~/monero# uname -a
Linux orangepipc2 4.14.7-sunxi64 #38 SMP Mon Dec 18 00:49:34 CET 2017 aarch64 aarch64 aarch64 GNU/Linux
root@orangepipc2:~/monero# cat /etc/armbian-release 
# PLEASE DO NOT EDIT THIS FILE
BOARD=orangepipc2
BOARD_NAME="Orange Pi PC2"
BOARDFAMILY=sun50iw2
VERSION=5.37.171218
LINUXFAMILY=sunxi64
BRANCH=next
ARCH=arm64
IMAGE_TYPE=nightly
BOARD_TYPE=conf
INITRD_ARCH=arm64
KERNEL_IMAGE_TYPE=Image

Swapiness is configured as follows (looks good to me):

root@orangepipc2:~/monero# sysctl -A 2>&1 | grep swap
vm.swappiness = 60

and my zram is configured as follows:

root@orangepipc2:~/monero# swapon
NAME           TYPE        SIZE   USED PRIO
/dev/zram0     partition 124.3M 124.2M    5
/dev/zram1     partition 124.3M 124.2M    5
/dev/zram2     partition 124.3M 124.2M    5
/dev/zram3     partition 124.3M 124.2M    5

Ready to compile!... Huh? No.

 

Running make (even "make -j1"!), the thing froze every time and took ages before I could ssh again (I'm not fond of unplugging it and the power-off button seems software-controlled, just like the LEDs, not a NMI, just as frozen as the OPiPC2 itself). It's so frozen that there's "no route to host" when I try to ssh or ping it (I keep it headless, there's nothing plugged on the USBs).

 

The nohup.out shows that, within seconds, the situation comes to a point where cc1 is killed. Pretty ungood, isn't it?

 

Initially, I thought it was CPU over-usage, but htop showed me it actually was the memory getting completely filled, and then the swap (but is it swap or zram in that case? there's no swap configured...).

So, yeah, I know, it's not clean to configure swap on a SDCard, but:

fallocate /var/swap.file --length 8GB ;
chmod 0600 /var/swap.file ;
mkswap /var/swap.file ;
swapon /var/swap.file ;
nohup make -j3 &

Initially, I tried just 1GB, bringing total swap to 1.42GB, but htop showed it was eaten up in a matter of a couple of minutes, so I went directly to 8GB. At the moment, load average is at 4.17, swap is 2.15G of 7.94G (well beyond anything I could get without configuring a swap). It's not very reactive (to say the least), but it's working.

So, unless there's a way to tell make/cmake//gcc to do otherwise, or a "cleaner" way to get the same result, adding a big swap on the SDCard was the only way to get things done... Slowly, wrongly, but done.

 

Cheers,

Regis

[tkaiser]

Follow-Up on https://www.cnx-software.com/2018/05/13/how-to-get-started-with-opencl-on-odroid-xu4-board-with-arm-mali-t628mp6-gpu/ 

 

It's about a compile job requiring up to 3 GB memory. Jean-Luc tested on an ODROID-XU4 with 2 GB RAM so he had to add 1 GB swap since otherwise compilation failed.

 

It's 2018 and we all should know that swap on HDD is anachronistic or even stupid. But Linux users still do it since they did it already one or even two decades ago.

 

Time for a 'quick' benchmark: I test on a board that is even more limited with regard to physical RAM:  It's a freshly arrived NanoPi Fire3 with eight A53 cores but just 1 GB RAM so very likely to run in out of memory situations with compile jobs running on all 8 cores in parallel.

 

I used an Armbian Stretch image running with kernel 4.14.40 and configured 2.5 GB swap (either as /var/swap or zram). Also vm.swappiness is set to the default (60). 

When compiling the job I've seen over 2.3 GB swap being used while building with zram with an average 3.5:1 compression ratio:
 

root@nanopim3:/home/tk# zramctl 
NAME       ALGORITHM DISKSIZE   DATA COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4         495.3M 370.1M 99.3M 104.5M       8 [SWAP]
/dev/zram1 lz4         495.3M 370.2M 99.4M 104.5M       8 [SWAP]
/dev/zram2 lz4         495.3M 371.3M 99.8M   105M       8 [SWAP]
/dev/zram3 lz4         495.3M 368.8M 99.1M 104.2M       8 [SWAP]
/dev/zram4 lz4         495.3M 370.2M 99.4M 104.5M       8 [SWAP]

First tests were with zram (trying to compare lzo with lz4), then I used a swap file on ext4 formatted SD card (a fast A1 rated one and not 'the average SD card' showing magnitudes lower random IO performance). Afterwards I tested with an older 7200rpm notebook HDD and then a Samsung EVO 840 both attached via USB2 without UAS support (so both random and sequential IO are bottlenecked by USB2 -- 'typical SBC conditions'). Filesystem used was a freshly created ext4 so no performance difference between swap file and partition.

 

Benchmark execution as follows: I deleted the working directory, then untared the archive (see above) and then let the following run:

time scons Werror=1 -j8 debug=0 neon=1 opencl=1 embed_kernels=1 os=linux arch=arm64-v8a build=native

Results:

zram lzo                  46m57.427s
zram lz4                  47m18.022s
SSD                      144m40.780s
SanDisk Ultra A1 16 GB   247m56.744s
HDD                      570m14.406s

 

Using zram is a no-brainer. The job finished within 47 minutes with both lzo and lz4 compression scheme. When testing with swap on real storage for obvious reasons the SSD won (best random IO performance), then the A1 rated SD card could finish the job in 4 hours and of course swap on HDD failed miserably since spinning rust simply sucks when it's about random I/O performance. Swap on HDD is BS.

 

A device with 1 GB physical DRAM is able to perform a build job requiring 3 GB memory within 47 minutes when using modern approaches (zram), when doing it like we did it last century (swapping to storage) then it's all about random I/O performance. A great performing SSD even if bottlenecked by USB2 is able to finish the job in less than two and a half hours. An el cheapo SanDisk Ultra A1 SD card with 16 GB scores 4 hours while as expected a 7.2k notebook HDD takes almost 10 hours to do the same job. HDDs suck at random IO. Period.

 

What can we learn from this? Nothing since Armbian is only focused on adding more and more irrelevant boards to the build system instead of improving situation for all boards.

 

As a reference iozone numbers for the 3 storage scenarios:

SanDisk Ultra A1                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4     3677     3856     9344     9320     8070     2787
          102400      16     3890     7780    16398    16216    14832     6716
          102400     512     9925    13902    22494    22501    22411    13178
          102400    1024    13826    13750    22651    22649    22640    13792
          102400   16384    14031    13936    23141    23137    23163    15395

Hitachi 2.5" 7200 rpm                                         random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4     7537     9837     8016     7763      671     1119
          102400      16    17452    20618    20852    20997     2522     4250
          102400     512    33123    33403    34165    34756    25589    32262
          102400    1024    33580    33310    34331    35056    29620    33148
          102400   16384    32592    33804    34921    35591    35277    33838

Samsung EVO 840 SSD                                           random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4     7322     9481     9636     9682     7433     9496
          102400      16    16654    19453    18576    18639    18301    19370
          102400     512    31296    31489    32778    32876    32977    31337
          102400    1024    31016    31986    33334    34283    33991    31273
          102400   16384    30600    31336    33318    33678    33236    31913

 

[TonyMac32]

4 hours ago, tkaiser said:

Swap on HDD is BS

 

My 1992 Conner hard drive is weeping.     Nice write-up. 

[tkaiser]

4 hours ago, TonyMac32 said:

Nice write-up

 

And just a waste of time...

[Tido]

11 hours ago, tkaiser said:

But Linux users still do it since they did it already one or even two decades ago.

Well, it could soon be back in fashion, NVMe and other technologies just around the corner.  Not true for our SBC's of course!

 

11 hours ago, tkaiser said:

What can we learn from this? Nothing since Armbian is only focused on adding more and more irrelevant boards to the build system instead of improving situation for all boards.

while this seems to me off topic - except you want to express. "why do you spend time on new  SBCs  instead of getting the build-system to the latest and fastest technology for devices with limited HW"  :-)

 

 

@Igor, @zador.blood.stained,  any chance to expand the usage of ZRAM or are there still technical roadblocks ahead ?

[zador.blood.stained]

6 minutes ago, Tido said:

 @zador.blood.stained,  any chance to expand the usage of ZRAM

What do you mean by that exactly? zram for the build host or for built OS images?

 

7 minutes ago, Tido said:

are there still technical roadblocks ahead ?

If lack of a proper build HW counts as a technical roadblock (in addition to existing problems like a time bomb log2ram implementation and non-technical project goals/aims and resulting motivation) then yes.

[JMCC]

FWIW, I've been doing lately lots of compiling in devices with 2Gb of RAM (TB, XU4), including big projects that recommended 4Gb for compilation (e.g. Collabora CODE), and never ran out of memory with the current ZRAM implementation. Of course, the situation could be different with devices having less RAM.

 

[UPDATE:] As a matter of fact, I set up an additional swapfile in the external SSHD, just to be safe, and never got used:

Spoiler

$ cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/zram0                              partition       127612  11516   5
/dev/zram1                              partition       127612  11492   5
/dev/zram2                              partition       127612  11684   5
/dev/zram3                              partition       127612  11552   5
/dev/zram4                              partition       127612  11720   5
/dev/zram5                              partition       127612  11448   5
/dev/zram6                              partition       127612  11668   5
/dev/zram7                              partition       127612  11600   5
/mnt/externo/swapfile1                  file            2097148 0       -1

 

 

[Tido]

13 minutes ago, zador.blood.stained said:

zram for the build host or for built OS images?

I meant for the images people download.  IIRC I read from TK somewhere that ZRAM is only used on Debian or Ubuntu, why not on both?

 

15 minutes ago, zador.blood.stained said:

If lack of a proper build HW counts as a technical roadblock

A while ago I read about a big server of Igor which does compilation of the images. Is this not running properly for such tasks ?

 

17 minutes ago, zador.blood.stained said:

like a time bomb log2ram implementation

I wanted to ask you about that in April, but I couldn't find the thread where you started with an idea that needs some testing. Can you create .deb for testing and the users here can do testing for you and reporting to you ?

 

20 minutes ago, zador.blood.stained said:

non-technical project goals/aim

Can you name one or two ?   Rome wasn't built in a day..

 

[zador.blood.stained]

12 minutes ago, Tido said:

IIRC I read from TK somewhere that ZRAM is only used on Debian or Ubuntu, why not on both?

Because zram-config package exists only in Ubuntu by default, and for reasons I don't remember (probably version numbering / potential repository priority issues?) we decided to not copy it to our repository to be available for all releases.

 

12 minutes ago, Tido said:

Is this not running properly for such tasks ?

It's not set up for multi-user scenario yet (VM guests for different tasks and users instead of one shared space otherwise manual build tasks may conflict with automated ones)

 

12 minutes ago, Tido said:

Can you name one or two ?   Rome wasn't built in a day..

I meant this specific thread: https://forum.armbian.com/topic/6281-board-support-general-discussion-project-aims/

Recent examples to add to that - there is no purpose in recently added "development" branch if "master" is completely abandoned as a result and suggesting to "switch to beta" to fix any issues defeats the purpose of "beta" - fixes should be immediately pushed to the stable branch and beta should be left for non-production ready stuff.

 

[JMCC]

My two cents, as a rookie: maybe it is a good idea to finish all the work for the meant-to-be-soon release, and then discuss about necessary changes in the project structure/aims.