Is there a way to avoid killing processe when out of memory Out of memory: Killed process

[JrRockeTer]

I have a problem with  Focal on Nanopi-M4V2 is that when it is overloaded and runs out of memory, instead of getting slow it kills the process causing the problem and in the last few cases, logs out out of Ubuntu Mate ;-)

 

[Sun Jul 26 17:12:54 2020] Out of memory: Killed process 12888 (Web Content) total-vm:3097484kB, anon-rss:455520kB, file-rss:1600kB, shmem-rss:16636kB, UID:1000 pgtables:2752kB oom_score_adj:0
[Sun Jul 26 17:12:54 2020] oom_reaper: reaped process 12888 (Web Content), now anon-rss:0kB, file-rss:0kB, shmem-rss:16220kB
[Sun Jul 26 17:13:22 2020] panfrost_gem_shrinker_scan: 60 callbacks suppressed
[Sun Jul 26 17:13:22 2020] Purging 4595712 byte

 

This is at the time of running (I was in a tmux session, it was still running when I logged in again)

Time       big.LITTLE   load %cpu %sys %usr %nice %io %irq   CPU  C.St.
17:12:53: 1800/1512MHz  8.19  98%  73%  23%   0%   1%   0% 65.0°C  0/6
17:12:59: 1800/1512MHz  8.49  67%  45%  20%   0%   0%   0% 60.6°C  0/6

....

At the time, the fan was running, I have the default zram swap file.

 

swapon
NAME       TYPE       SIZE  USED PRIO
/dev/zram1 partition 1024M 86.1M    5

 

armbianmonitor -u

http://ix.io/2sme

 

The system runs off NVME

I was thinking of putting in an EMMC Module and install more swap on that but I thought I'd get your opinion .

 

I'd prefer it because slow so that I can investigate what is going on instead of logging me out when I'm testing scrpts or writing scripts or documentation.

 

Thanks in advance.

[Werner]

If you have an NVMe installed anyways you should put the swap there since it is way faster than an eMMC.

Or investigate if there is a memory leak.

[lanefu]

yeah before swapping to disk, you could look at increasing zram size..... but ultimately desktops and browsers are killers so really its probably best to focus on that front.

there's a Userspace OOM killer that's supposed to be a little smarter.. might be worth checking out https://facebookmicrosites.github.io/oomd/

[JrRockeTer]

13 minutes ago, lanefu said:

yeah before swapping to disk, you could look at increasing zram size..... but ultimately desktops and browsers are killers so really its probably best to focus on that front.

there's a Userspace OOM killer that's supposed to be a little smarter.. might be worth checking out https://facebookmicrosites.github.io/oomd/

 

I've just had a look at this, looks interesting.

I won't put it on the M4V2 at the moment as I've had enough problems without adding other potential problems ;-)

 

However, this would be interesting on my odroidN2 which is continually kill boinc processes due to out of memory

 

I was hoping there was another trick on Armbian that would kind of NOT kill stuff but just slow down..

 

Thanks for your suggestion I'll try it out on the Odroid.

 

[NicoD]

21 hours ago, JrRockeTer said:

NAME       TYPE       SIZE  USED PRIO
/dev/zram1 partition 1024M 86.1M    5

Hi @JrRockeTer 
Do :

sudo apt install zram-config

and reboot. 
That will increase your zram from 1024MB to 2048MB. There's a fault in the armbian zram-config. For 4GB ram it should create 2GB zram. But it by default creates 1GB. 

If you run on an NVMe, then create a big swap file (don't forget sudo, or use "su" before applying the commands :
 

fallocate -l 8G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo "/swapfile swap swap defaults 0 0"  >> /etc/fstab

I'll check out Focal mainline now on the M4V2. Cheers.

[JrRockeTer]

5 hours ago, NicoD said:

Hi @JrRockeTer 
Do :

and reboot. 
That will increase your zram from 1024MB to 2048MB. There's a fault in the armbian zram-config. For 4GB ram it should create 2GB zram. But it by default creates 1GB. 

If you run on an NVMe, then create a big swap file (don't forget sudo, or use "su" before applying the commands

I'll check out Focal mainline now on the M4V2. Cheers.

 

Amazing Nico, this zram info should be on the download page!

How did you know that about zram!!

 

Thanks heaps for this and the creating a large swap file!

Filename                Type        Size    Used    Priority
/dev/zram0                              partition    329960    0    5
/dev/zram1                              partition    329960    0    5
/dev/zram2                              partition    329960    0    5
/dev/zram3                              partition    329960    0    5
/dev/zram4                              partition    329960    0    5
/dev/zram5                              partition    329960    0    5
/swapfile                               file        8388604    0    -2
jerry@nano2:~/bin $

 

Thanks heaps

[NicoD]

1 hour ago, JrRockeTer said:

How did you know that about zram!!

Always been like that with 4GB models and the armbian zram script. I think back then when the script was "modified" there were no SBC's with 4GB.
Maybe Armbian should use the standard zram-config again. The maintainer has long gone. It's not my call.

You will not easily run out of memory now.

[lanefu]

pull request welcome

[lanefu]

It's the spirit of TK and persistent testing and tuning that must live on.... any help in that front is great

[JrRockeTer]

9 hours ago, NicoD said:

Always been like that with 4GB models and the armbian zram script. I think back then when the script was "modified" there were no SBC's with 4GB.
Maybe Armbian should use the standard zram-config again. The maintainer has long gone. It's not my call.

You will not easily run out of memory now.

 

Pity about the Zram config and pity this is NOT communicated when you download it so you know to run the config, but worse, that it has not been removed or repaired.

 

Yes, when I have 5 minutes, I will duplicate what I did the other day and see what happens.

 

I was trying to test my modified fancontrol.c program so was launching a few you tube videos and facebook pages ;-)

 

Thanks again for your help as always @NicoD Armbian would not be the same without you !

[JrRockeTer]

17 hours ago, NicoD said:

Always been like that with 4GB models and the armbian zram script. I think back then when the script was "modified" there were no SBC's with 4GB.
Maybe Armbian should use the standard zram-config again. The maintainer has long gone. It's not my call.

You will not easily run out of memory now.

Check this out @NicoD @Werner, beautiful and it was usable for most of the time

 

cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/zram0                              partition       329960  328920  5
/dev/zram1                              partition       329960  329020  5
/dev/zram2                              partition       329960  329076  5
/dev/zram3                              partition       329960  328840  5
/dev/zram4                              partition       329960  329008  5
/dev/zram5                              partition       329960  329000  5
/swapfile                               file            8388604 516160  -2

 

vmstat 5 100
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
14  1 2945416  82864   3580 281876   14   20    13     6   44   53  1  0 99  0  0
10  0 2965260  79092   2084 251232 43369 35211 34502 13287 10110 17175 58 34  6  2  0
 9  0 2999792  83996   4204 293888 32905 38888 31712 27475 9000 15632 61 34  4  1  0
 

I think most of my original problems were related to: Power, Memory, HEAT

1- at one time I made a mistake and was not using the factory power supply

2- the problem with zram and swapping that you fixed for me @NicoD

3- When I first installed it I had not installed fan management, this should come standard.

Thanks you, I'll cross post this to my other post

 

[Werner]

I've create an issue on jira regarding the zram thingy. Whoever wants to take a shot at it... https://armbian.atlassian.net/browse/AR-382