Guest thom_arm Posted March 4, 2017 Share Posted March 4, 2017 (edited) Hi, since one the last two kernel updates I experience random out of memory scenarious on my Cubietruck. Up until then all was fine. When Oom Killer kicks in the following is logged: ........................................ Spoiler Mar 4 06:54:23 cubietruck kernel: [80988.962548] sh invoked oom-killer: gfp_mask=0x24000c0(GFP_KERNEL), nodemask=0, order=2, oom_score_adj=0 Mar 4 06:54:23 cubietruck kernel: [80988.962563] sh cpuset=/ mems_allowed=0 Mar 4 06:54:23 cubietruck kernel: [80988.962584] CPU: 1 PID: 10543 Comm: sh Not tainted 4.9.12-sunxi #4 Mar 4 06:54:23 cubietruck kernel: [80988.962588] Hardware name: Allwinner sun7i (A20) Family Mar 4 06:54:23 cubietruck kernel: [80988.962625] [<c010c8dd>] (unwind_backtrace) from [<c01096a5>] (show_stack+0x11/0x14) Mar 4 06:54:23 cubietruck kernel: [80988.962643] [<c01096a5>] (show_stack) from [<c04fd277>] (dump_stack+0x67/0x74) Mar 4 06:54:23 cubietruck kernel: [80988.962660] [<c04fd277>] (dump_stack) from [<c0207e27>] (dump_header+0x63/0x174) Mar 4 06:54:23 cubietruck kernel: [80988.962679] [<c0207e27>] (dump_header) from [<c01c3a75>] (oom_kill_process+0x261/0x3e4) Mar 4 06:54:23 cubietruck kernel: [80988.962693] [<c01c3a75>] (oom_kill_process) from [<c01c3e19>] (out_of_memory+0xc5/0x224) Mar 4 06:54:23 cubietruck kernel: [80988.962706] [<c01c3e19>] (out_of_memory) from [<c01c75f5>] (__alloc_pages_nodemask+0x939/0x950) Mar 4 06:54:23 cubietruck kernel: [80988.962720] [<c01c75f5>] (__alloc_pages_nodemask) from [<c01c761d>] (__get_free_pages+0x11/0x20) Mar 4 06:54:23 cubietruck kernel: [80988.962734] [<c01c761d>] (__get_free_pages) from [<c01137ad>] (pgd_alloc+0x19/0xfc) Mar 4 06:54:23 cubietruck kernel: [80988.962747] [<c01137ad>] (pgd_alloc) from [<c0116b17>] (mm_init+0x97/0x118) Mar 4 06:54:23 cubietruck kernel: [80988.962760] [<c0116b17>] (mm_init) from [<c020fe6b>] (do_execveat_common+0x1db/0x540) Mar 4 06:54:23 cubietruck kernel: [80988.962773] [<c020fe6b>] (do_execveat_common) from [<c021038d>] (SyS_execve+0x25/0x28) Mar 4 06:54:23 cubietruck kernel: [80988.962786] [<c021038d>] (SyS_execve) from [<c01060c1>] (ret_fast_syscall+0x1/0x4c) Mar 4 06:54:23 cubietruck kernel: [80988.962791] Mem-Info: Mar 4 06:54:23 cubietruck kernel: [80988.962814] active_anon:12204 inactive_anon:45615 isolated_anon:0 Mar 4 06:54:23 cubietruck kernel: [80988.962814] active_file:38662 inactive_file:237639 isolated_file:31 Mar 4 06:54:23 cubietruck kernel: [80988.962814] unevictable:337 dirty:53 writeback:0 unstable:0 Mar 4 06:54:23 cubietruck kernel: [80988.962814] slab_reclaimable:125926 slab_unreclaimable:41611 Mar 4 06:54:23 cubietruck kernel: [80988.962814] mapped:19018 shmem:15015 pagetables:745 bounce:0 Mar 4 06:54:23 cubietruck kernel: [80988.962814] free:8795 free_pcp:1 free_cma:588 Mar 4 06:54:23 cubietruck kernel: [80988.962833] Node 0 active_anon:48816kB inactive_anon:182460kB active_file:154648kB inactive_file:950556kB unevictable:1348kB isolated(anon):0kB isolated(file):124kB mapped:76072kB dirty:212kB writeback:0kB shmem:60060kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no Mar 4 06:54:23 cubietruck kernel: [80988.962853] Normal free:17656kB min:3468kB low:4332kB high:5196kB active_anon:488kB inactive_anon:3588kB active_file:54324kB inactive_file:88kB unevictable:0kB writepending:212kB present:786432kB managed:753376kB mlocked:0kB slab_reclaimable:503704kB slab_unreclaimable:166444kB kernel_stack:1424kB pagetables:16kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB Mar 4 06:54:23 cubietruck kernel: lowmem_reserve[]: 0 10088 10088 Mar 4 06:54:23 cubietruck kernel: [80988.962884] HighMem free:17524kB min:512kB low:2000kB high:3488kB active_anon:48328kB inactive_anon:178872kB active_file:100356kB inactive_file:950336kB unevictable:1348kB writepending:0kB present:1307648kB managed:1307648kB mlocked:1348kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:2964kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:2352kB Mar 4 06:54:23 cubietruck kernel: lowmem_reserve[]: 0 0 0 Mar 4 06:54:23 cubietruck kernel: [80988.962898] Normal: 3077*4kB (UME) 671*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17676kB Mar 4 06:54:23 cubietruck kernel: HighMem: 2026*4kB (UMC) 615*8kB (UMC) 121*16kB (UMC) 43*32kB (UMC) 16*64kB (UM) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17616kB Mar 4 06:54:23 cubietruck kernel: 291594 total pagecache pages Mar 4 06:54:23 cubietruck kernel: [80988.962989] 0 pages in swap cache Mar 4 06:54:23 cubietruck kernel: [80988.962994] Swap cache stats: add 0, delete 0, find 0/0 Mar 4 06:54:23 cubietruck kernel: [80988.962997] Free swap = 524284kB Mar 4 06:54:23 cubietruck kernel: [80988.963000] Total swap = 524284kB Mar 4 06:54:23 cubietruck kernel: [80988.963003] 523520 pages RAM Mar 4 06:54:23 cubietruck kernel: [80988.963006] 326912 pages HighMem/MovableOnly Mar 4 06:54:23 cubietruck kernel: [80988.963009] 8264 pages reserved Mar 4 06:54:23 cubietruck kernel: [80988.963012] 4096 pages cma reserved Mar 4 06:54:23 cubietruck kernel: [80988.963016] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Mar 4 06:54:23 cubietruck kernel: [80988.963184] [ 256] 0 256 2640 569 6 0 0 -1000 udevd Mar 4 06:54:23 cubietruck kernel: [80988.963199] [ 417] 0 417 780 310 5 0 0 0 lircd Mar 4 06:54:23 cubietruck kernel: [80988.963215] [ 1538] 999 1538 4835 4368 13 0 0 0 perl Mar 4 06:54:23 cubietruck kernel: [80988.963233] [ 1554] 0 1554 7850 544 9 0 0 0 rsyslogd Mar 4 06:54:23 cubietruck kernel: [80988.963251] [ 1577] 0 1577 4438 978 12 0 0 0 nmbd Mar 4 06:54:23 cubietruck kernel: [80988.963268] [ 1590] 0 1590 543 32 6 0 0 0 atd Mar 4 06:54:23 cubietruck kernel: [80988.963286] [ 1645] 105 1645 1146 426 6 0 0 0 dbus-daemon Mar 4 06:54:23 cubietruck kernel: [80988.963317] [ 1677] 0 1677 1797 1037 7 0 0 0 haveged Mar 4 06:54:23 cubietruck kernel: [80988.963334] [ 1694] 0 1694 803 489 6 0 0 0 incrond Mar 4 06:54:23 cubietruck kernel: [80988.963354] [ 1875] 33 1875 3391 1441 10 0 0 0 lighttpd Mar 4 06:54:23 cubietruck kernel: [80988.963367] [ 1876] 33 1876 6490 4474 17 0 0 0 munin-cgi-graph Mar 4 06:54:23 cubietruck kernel: [80988.963383] [ 1886] 33 1886 6490 4443 16 0 0 0 munin-cgi-graph Mar 4 06:54:23 cubietruck kernel: [80988.963400] [ 1898] 33 1898 6490 4395 17 0 0 0 munin-cgi-graph Mar 4 06:54:23 cubietruck kernel: [80988.963417] [ 1908] 33 1908 6490 4429 18 0 0 0 munin-cgi-graph Mar 4 06:54:23 cubietruck kernel: [80988.963435] [ 1912] 33 1912 50765 5648 29 0 0 0 php-cgi Mar 4 06:54:23 cubietruck kernel: [80988.963452] [ 1914] 0 1914 360 227 5 0 0 0 mysqld_safe Mar 4 06:54:23 cubietruck kernel: [80988.963470] [ 2015] 33 2015 52050 7973 37 0 0 0 php-cgi Mar 4 06:54:23 cubietruck kernel: [80988.963487] [ 2016] 33 2016 52007 7874 37 0 0 0 php-cgi Mar 4 06:54:23 cubietruck kernel: [80988.963504] [ 2018] 33 2018 51810 7441 37 0 0 0 php-cgi Mar 4 06:54:23 cubietruck kernel: [80988.963522] [ 2019] 33 2019 51985 7447 37 0 0 0 php-cgi Mar 4 06:54:23 cubietruck kernel: [80988.963538] [ 2296] 108 2296 81749 10711 46 0 0 0 mysqld Mar 4 06:54:23 cubietruck kernel: [80988.963548] [ 2343] 106 2343 1123 734 6 0 0 0 ntpd Mar 4 06:54:23 cubietruck kernel: [80988.963564] [ 2449] 0 2449 19665 2038 19 0 0 0 fail2ban-server Mar 4 06:54:23 cubietruck kernel: [80988.963578] [ 2460] 109 2460 39311 3817 18 0 0 -900 postgres Mar 4 06:54:23 cubietruck kernel: [80988.963587] [ 2473] 0 2473 7317 2790 19 0 0 0 smbd Mar 4 06:54:23 cubietruck kernel: [80988.963598] [ 2475] 109 2475 39311 1665 21 0 0 0 postgres Mar 4 06:54:23 cubietruck kernel: [80988.963608] [ 2476] 109 2476 39311 1153 14 0 0 0 postgres Mar 4 06:54:23 cubietruck kernel: [80988.963618] [ 2477] 109 2477 39311 1952 14 0 0 0 postgres Mar 4 06:54:23 cubietruck kernel: [80988.963628] [ 2478] 109 2478 39380 1253 14 0 0 0 postgres Mar 4 06:54:23 cubietruck kernel: [80988.963642] [ 2480] 109 2480 4375 918 11 0 0 0 postgres Mar 4 06:54:23 cubietruck kernel: [80988.963654] [ 2529] 0 2529 1635 560 6 0 0 -1000 sshd Mar 4 06:54:23 cubietruck kernel: [80988.963664] [ 2707] 114 2707 4101 2100 13 0 0 0 unbound Mar 4 06:54:23 cubietruck kernel: [80988.963673] [ 2755] 0 2755 1601 438 8 0 0 0 cron Mar 4 06:54:23 cubietruck kernel: [80988.963684] [ 2769] 0 2769 693 487 6 0 0 0 dovecot Mar 4 06:54:23 cubietruck kernel: [80988.963693] [ 2774] 111 2774 1048 853 6 0 0 0 imap-login Mar 4 06:54:23 cubietruck kernel: [80988.963703] [ 2775] 110 2775 576 405 7 0 0 0 anvil Mar 4 06:54:23 cubietruck kernel: [80988.963712] [ 2777] 0 2777 610 433 6 0 0 0 log Mar 4 06:54:23 cubietruck kernel: [80988.963723] [ 2779] 111 2779 1048 782 7 0 0 0 imap-login Mar 4 06:54:23 cubietruck kernel: [80988.963732] [ 2781] 0 2781 961 677 6 0 0 0 config Mar 4 06:54:23 cubietruck kernel: [80988.963742] [ 2798] 0 2798 346 240 5 0 0 0 startpar Mar 4 06:54:23 cubietruck kernel: [80988.963752] [ 3009] 0 3009 1339 637 6 0 0 0 master Mar 4 06:54:23 cubietruck kernel: [80988.963761] [ 3016] 112 3016 1367 656 7 0 0 0 qmgr Mar 4 06:54:23 cubietruck kernel: [80988.963771] [ 3062] 0 3062 356 340 6 0 0 -1000 watchdog Mar 4 06:54:23 cubietruck kernel: [80988.963783] [29502] 112 29502 1358 635 6 0 0 0 pickup Mar 4 06:54:23 cubietruck kernel: [80988.963792] [ 1721] 0 1721 1795 463 8 0 0 0 cron Mar 4 06:54:23 cubietruck kernel: [80988.963802] [ 1724] 0 1724 360 105 4 0 0 0 sh Mar 4 06:54:23 cubietruck kernel: [80988.963811] [ 1726] 0 1726 343 228 6 0 0 0 run-parts Mar 4 06:54:23 cubietruck kernel: [80988.963822] [ 7608] 0 7608 3799 2400 11 0 0 0 munin-node Mar 4 06:54:23 cubietruck kernel: [80988.963832] [ 8055] 110 8055 782 488 7 0 0 0 auth Mar 4 06:54:23 cubietruck kernel: [80988.963843] [ 9645] 0 9645 576 124 6 0 0 0 ssl-params Mar 4 06:54:23 cubietruck kernel: [80988.963853] [ 9682] 110 9682 578 140 6 0 0 0 indexer Mar 4 06:54:23 cubietruck kernel: [80988.963863] [ 9683] 0 9683 1861 1452 7 0 0 0 indexer-worker Mar 4 06:54:23 cubietruck kernel: [80988.963873] [10255] 107 10255 4629 3009 12 0 0 0 munin-html Mar 4 06:54:23 cubietruck kernel: [80988.963885] [10264] 112 10264 1375 658 8 0 0 0 cleanup Mar 4 06:54:23 cubietruck kernel: [80988.963895] [10269] 112 10269 1360 636 6 0 0 0 trivial-rewrite Mar 4 06:54:23 cubietruck kernel: [80988.963904] [10272] 0 10272 1366 858 6 0 0 0 local Mar 4 06:54:23 cubietruck kernel: [80988.963914] [10280] 112 10280 1430 680 5 0 0 0 lmtp Mar 4 06:54:23 cubietruck kernel: [80988.963924] [10284] 0 10284 2406 1763 9 0 0 0 lmtp Mar 4 06:54:23 cubietruck kernel: [80988.963933] [10290] 0 10290 885 607 6 0 0 0 auth Mar 4 06:54:23 cubietruck kernel: [80988.963943] [10302] 1000 10302 587 446 4 0 0 0 mvmail Mar 4 06:54:23 cubietruck kernel: [80988.963952] [10368] 1000 10368 343 79 5 0 0 0 flock Mar 4 06:54:23 cubietruck kernel: [80988.963962] [10369] 1000 10369 344 87 5 0 0 0 sleep Mar 4 06:54:23 cubietruck kernel: [80988.963972] [10536] 0 10536 360 96 4 0 0 0 man-db Mar 4 06:54:23 cubietruck kernel: [80988.963982] [10542] 6 10542 360 105 4 0 0 0 sh Mar 4 06:54:23 cubietruck kernel: [80988.963990] [10543] 6 10543 360 16 4 0 0 0 sh Mar 4 06:54:23 cubietruck kernel: [80988.963999] [10544] 6 10544 360 16 4 0 0 0 sh Mar 4 06:54:23 cubietruck kernel: [80988.964006] Out of memory: Kill process 2296 (mysqld) score 16 or sacrifice child Mar 4 06:54:23 cubietruck kernel: [80988.964141] Killed process 2296 (mysqld) total-vm:326996kB, anon-rss:35548kB, file-rss:7296kB, shmem-rss:0kB ........................................ Is there a possibility to rollback the kernel and associated packages to 5.23? Where to find these and how to downgrade then? I am currently running 5.26 and am pretty sure that this started with 5.25. Yes, the system was unchanged - I also removed some services but can not nail it down. Any help to get the root cause for this is also greately appreciated - what logs / command output to provide here? Thanks! Edited March 11, 2017 by thom_arm code under spoiler Link to comment Share on other sites More sharing options...
Bernie_O Posted March 6, 2017 Share Posted March 6, 2017 Which kernel-version are you running? There was a bug in kernel 4.7/4.8: https://encrypted.google.com/search?hl=de&q=oom-killer kernel bug 4.7 There is also a memory-parameter that configures how much % of memory is used before the oom-killer is invoked. That might help you until the kernel-bug is fixed. Look for vm.overcommit_ratio in the last post here: http://serverfault.com/questions/362589/effects-of-configuring-vm-overcommit-memory Link to comment Share on other sites More sharing options...
timmr Posted March 10, 2017 Share Posted March 10, 2017 (edited) I have the same issue as thom_arm: Since an update from 5.23 to 5.25 (now on kernel 4.9.12), I get frequent oom errors resulting in oom-killer killing my unbound, system-journal etc etc daemons. I have an ssd attached and tried the following: moved /tmp from tmpfs to the ssd set vm/swappiness from 0 to 30 increased swap to 3 GByte (2 GByte physical RAM available in total on my cubietruck board) set vm/overcommit to 2 and vm/overcommit_ratio to 100 None of that helped to improve the situation. Except the armbian update mentioned above, nothing changed on the system, that I am aware of. The oom's occur during cron-daily jobs (e.g. texpire) or fetching news (leafnode). But that was working perfectly before the armbian updates Any suggestions what I should do? Sample logging: ------------------------------------------ Spoiler Mar 10 06:55:39 nari kernel: [79423.760842] texpire invoked oom-killer: gfp_mask=0x26000d0(GFP_TEMPORARY|__GFP_NOTRACK), nodemask=0, order=0, oom_score_adj=0 Mar 10 06:55:39 nari kernel: [79423.760859] texpire cpuset=/ mems_allowed=0 Mar 10 06:55:39 nari kernel: [79423.760881] CPU: 0 PID: 4368 Comm: texpire Not tainted 4.9.12-sunxi #4 Mar 10 06:55:39 nari kernel: [79423.760885] Hardware name: Allwinner sun7i (A20) Family Mar 10 06:55:39 nari kernel: [79423.760926] [<c010c8dd>] (unwind_backtrace) from [<c01096a5>] (show_stack+0x11/0x14) Mar 10 06:55:39 nari kernel: [79423.760944] [<c01096a5>] (show_stack) from [<c04fd277>] (dump_stack+0x67/0x74) Mar 10 06:55:39 nari kernel: [79423.760960] [<c04fd277>] (dump_stack) from [<c0207e27>] (dump_header+0x63/0x174) Mar 10 06:55:39 nari kernel: [79423.760981] [<c0207e27>] (dump_header) from [<c01c3a75>] (oom_kill_process+0x261/0x3e4) Mar 10 06:55:39 nari kernel: [79423.760994] [<c01c3a75>] (oom_kill_process) from [<c01c3e19>] (out_of_memory+0xc5/0x224) Mar 10 06:55:39 nari kernel: [79423.761007] [<c01c3e19>] (out_of_memory) from [<c01c75f5>] (__alloc_pages_nodemask+0x939/0x950) Mar 10 06:55:39 nari kernel: [79423.761025] [<c01c75f5>] (__alloc_pages_nodemask) from [<c01f94bf>] (new_slab+0x303/0x3f8) Mar 10 06:55:39 nari kernel: [79423.761038] [<c01f94bf>] (new_slab) from [<c01fa813>] (___slab_alloc.constprop.10+0x1a3/0x204) Mar 10 06:55:39 nari kernel: [79423.761051] [<c01fa813>] (___slab_alloc.constprop.10) from [<c01faded>] (kmem_cache_alloc+0x131/0x154) Mar 10 06:55:39 nari kernel: [79423.761065] [<c01faded>] (kmem_cache_alloc) from [<c021c7b5>] (__d_alloc+0x25/0x120) Mar 10 06:55:39 nari kernel: [79423.761076] [<c021c7b5>] (__d_alloc) from [<c021c8c1>] (d_alloc+0x11/0x58) Mar 10 06:55:39 nari kernel: [79423.761086] [<c021c8c1>] (d_alloc) from [<c021cc2f>] (d_alloc_parallel+0x3b/0x338) Mar 10 06:55:39 nari kernel: [79423.761098] [<c021cc2f>] (d_alloc_parallel) from [<c0211c51>] (lookup_slow+0x3d/0xe4) Mar 10 06:55:39 nari kernel: [79423.761110] [<c0211c51>] (lookup_slow) from [<c0213f09>] (walk_component+0x14d/0x1c4) Mar 10 06:55:39 nari kernel: [79423.761123] [<c0213f09>] (walk_component) from [<c021439b>] (path_lookupat+0x47/0xb0) Mar 10 06:55:39 nari kernel: [79423.761134] [<c021439b>] (path_lookupat) from [<c02157c7>] (filename_lookup+0x57/0xc8) Mar 10 06:55:39 nari kernel: [79423.761150] [<c02157c7>] (filename_lookup) from [<c020deff>] (vfs_fstatat+0x3f/0x78) Mar 10 06:55:39 nari kernel: [79423.761162] [<c020deff>] (vfs_fstatat) from [<c020e391>] (SyS_lstat64+0x1d/0x2c) Mar 10 06:55:39 nari kernel: [79423.761175] [<c020e391>] (SyS_lstat64) from [<c01060c1>] (ret_fast_syscall+0x1/0x4c) Mar 10 06:55:39 nari kernel: [79423.761181] Mem-Info: Mar 10 06:55:39 nari kernel: [79423.761202] active_anon:2116 inactive_anon:11199 isolated_anon:0 Mar 10 06:55:39 nari kernel: [79423.761202] active_file:108003 inactive_file:147886 isolated_file:0 Mar 10 06:55:39 nari kernel: [79423.761202] unevictable:0 dirty:0 writeback:0 unstable:0 Mar 10 06:55:39 nari kernel: [79423.761202] slab_reclaimable:182236 slab_unreclaimable:2742 Mar 10 06:55:39 nari kernel: [79423.761202] mapped:6873 shmem:4296 pagetables:349 bounce:0 Mar 10 06:55:39 nari kernel: [79423.761202] free:55967 free_pcp:523 free_cma:0 Mar 10 06:55:39 nari kernel: [79423.761220] Node 0 active_anon:8464kB inactive_anon:44796kB active_file:432012kB inactive_file:591544kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:27492kB dirty:0kB writeback:0kB shmem:17184kB writeback_tmp:0kB unstable:0kB pages_scanned:11169818 all_unreclaimable? yes Mar 10 06:55:39 nari kernel: [79423.761240] Normal free:3452kB min:3460kB low:4324kB high:5188kB active_anon:0kB inactive_anon:0kB active_file:76kB inactive_file:0kB unevictable:0kB writepending:0kB present:786432kB managed:753372kB mlocked:0kB slab_reclaimable:728944kB slab_unreclaimable:10968kB kernel_stack:992kB pagetables:0kB bounce:0kB free_pcp:1264kB local_pcp:548kB free_cma:0kB Mar 10 06:55:39 nari kernel: lowmem_reserve[]: 0 10088 10088 Mar 10 06:55:39 nari kernel: [79423.761272] HighMem free:220416kB min:512kB low:2004kB high:3496kB active_anon:8464kB inactive_anon:44796kB active_file:431912kB inactive_file:591468kB unevictable:0kB writepending:0kB present:1307648kB managed:1307648kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:1396kB bounce:0kB free_pcp:828kB local_pcp:76kB free_cma:0kB Mar 10 06:55:39 nari kernel: lowmem_reserve[]: 0 0 0 Mar 10 06:55:39 nari kernel: [79423.761285] Normal: 607*4kB (UME) 128*8kB (ME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3452kB Mar 10 06:55:39 nari kernel: HighMem: 0*4kB 468*8kB (UM) 404*16kB (UM) 289*32kB (M) 266*64kB (UM) 109*128kB (M) 68*256kB (M) 64*512kB (UM) 19*1024kB (UM) 19*2048kB (UM) 15*4096kB (M) = 220416kB Mar 10 06:55:39 nari kernel: 260204 total pagecache pages Mar 10 06:55:39 nari kernel: [79423.761387] 16 pages in swap cache Mar 10 06:55:39 nari kernel: [79423.761392] Swap cache stats: add 1980, delete 1964, find 759/1109 Mar 10 06:55:39 nari kernel: [79423.761395] Free swap = 3276696kB Mar 10 06:55:39 nari kernel: [79423.761398] Total swap = 3276792kB Mar 10 06:55:39 nari kernel: [79423.761401] 523520 pages RAM Mar 10 06:55:39 nari kernel: [79423.761404] 326912 pages HighMem/MovableOnly Mar 10 06:55:39 nari kernel: [79423.761407] 8265 pages reserved Mar 10 06:55:39 nari kernel: [79423.761410] 4096 pages cma reserved Mar 10 06:55:39 nari kernel: [79423.761414] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Mar 10 06:55:39 nari kernel: [79423.761460] [ 245] 0 245 2110 937 8 0 0 0 systemd-journal Mar 10 06:55:39 nari kernel: [79423.761470] [ 250] 0 250 2682 604 7 0 0 -1000 systemd-udevd Mar 10 06:55:39 nari kernel: [79423.761481] [ 596] 0 596 1798 1012 8 0 0 0 haveged Mar 10 06:55:39 nari kernel: [79423.761490] [ 598] 0 598 7860 633 10 0 0 0 rsyslogd Mar 10 06:55:39 nari kernel: [79423.761500] [ 599] 0 599 444 294 6 0 0 0 inetd Mar 10 06:55:39 nari kernel: [79423.761509] [ 600] 0 600 1635 897 9 0 12 -1000 sshd Mar 10 06:55:39 nari kernel: [79423.761518] [ 601] 110 601 856 532 6 0 0 0 avahi-daemon Mar 10 06:55:39 nari kernel: [79423.761528] [ 605] 0 605 1114 501 7 0 0 0 cron Mar 10 06:55:39 nari kernel: [79423.761537] [ 607] 110 607 791 54 5 0 0 0 avahi-daemon Mar 10 06:55:39 nari kernel: [79423.761546] [ 609] 0 609 774 473 6 0 6 0 systemd-logind Mar 10 06:55:39 nari kernel: [79423.761556] [ 611] 104 611 1146 607 7 0 3 -900 dbus-daemon Mar 10 06:55:39 nari kernel: [79423.761565] [ 653] 0 653 780 330 5 0 0 0 lircd Mar 10 06:55:39 nari kernel: [79423.761574] [ 691] 0 691 5026 1288 9 0 0 0 cupsd Mar 10 06:55:39 nari kernel: [79423.761584] [ 719] 105 719 1185 768 5 0 0 0 ntpd Mar 10 06:55:39 nari kernel: [79423.761594] [ 726] 0 726 13589 1052 14 0 5 0 bacula-fd Mar 10 06:55:39 nari kernel: [79423.761603] [ 730] 107 730 4153 2650 12 0 0 0 unbound Mar 10 06:55:39 nari kernel: [79423.761613] [ 739] 108 739 1357 551 8 0 0 0 dnsmasq Mar 10 06:55:39 nari kernel: [79423.761623] [ 747] 0 747 889 348 6 0 0 0 agetty Mar 10 06:55:39 nari kernel: [79423.761631] [ 753] 0 753 844 406 7 0 0 0 agetty Mar 10 06:55:39 nari kernel: [79423.761640] [ 775] 0 775 2229 1394 8 0 0 0 ddclient - slee Mar 10 06:55:39 nari kernel: [79423.761650] [ 797] 0 797 26741 4711 27 0 0 0 apache2 Mar 10 06:55:39 nari kernel: [79423.761659] [ 801] 33 801 26751 1483 21 0 0 0 apache2 Mar 10 06:55:39 nari kernel: [79423.761669] [ 802] 33 802 26751 1485 21 0 0 0 apache2 Mar 10 06:55:39 nari kernel: [79423.761678] [ 803] 33 803 26751 1448 21 0 0 0 apache2 Mar 10 06:55:39 nari kernel: [79423.761687] [ 804] 33 804 26751 1448 21 0 0 0 apache2 Mar 10 06:55:39 nari kernel: [79423.761697] [ 805] 33 805 26751 1448 21 0 0 0 apache2 Mar 10 06:55:39 nari kernel: [79423.761709] [ 4115] 0 4115 1442 488 8 0 0 0 cron Mar 10 06:55:39 nari kernel: [79423.761718] [ 4119] 0 4119 360 101 4 0 0 0 sh Mar 10 06:55:39 nari kernel: [79423.761727] [ 4120] 0 4120 343 78 4 0 0 0 run-parts Mar 10 06:55:39 nari kernel: [79423.761737] [ 4362] 0 4362 360 90 5 0 0 0 leafnode Mar 10 06:55:39 nari kernel: [79423.761747] [ 4363] 0 4363 1461 619 8 0 0 0 su Mar 10 06:55:39 nari kernel: [79423.761756] [ 4364] 9 4364 1135 767 7 0 0 0 systemd Mar 10 06:55:39 nari kernel: [79423.761765] [ 4365] 9 4365 1651 485 6 0 1 0 (sd-pam) Mar 10 06:55:39 nari kernel: [79423.761774] [ 4367] 9 4367 360 105 6 0 0 0 sh Mar 10 06:55:39 nari kernel: [79423.761783] [ 4368] 9 4368 2004 1751 10 0 0 0 texpire Mar 10 06:55:39 nari kernel: [79423.789501] rs:main Q:Reg invoked oom-killer: gfp_mask=0x2420848(GFP_NOFS|__GFP_NOFAIL|__GFP_HARDWALL|__GFP_MOVABLE), nodemask=0, order=0, oom_score_adj=0 Mar 10 06:55:39 nari kernel: [79423.789519] rs:main Q:Reg cpuset=/ mems_allowed=0 Mar 10 06:55:39 nari kernel: [79423.789541] CPU: 1 PID: 604 Comm: rs:main Q:Reg Not tainted 4.9.12-sunxi #4 Mar 10 06:55:39 nari kernel: [79423.789544] Hardware name: Allwinner sun7i (A20) Family Mar 10 06:55:39 nari kernel: [79423.789583] [<c010c8dd>] (unwind_backtrace) from [<c01096a5>] (show_stack+0x11/0x14) Mar 10 06:55:39 nari kernel: [79423.789601] [<c01096a5>] (show_stack) from [<c04fd277>] (dump_stack+0x67/0x74) Mar 10 06:55:39 nari kernel: [79423.789618] [<c04fd277>] (dump_stack) from [<c0207e27>] (dump_header+0x63/0x174) Mar 10 06:55:39 nari kernel: [79423.789640] [<c0207e27>] (dump_header) from [<c01c3a75>] (oom_kill_process+0x261/0x3e4) Mar 10 06:55:39 nari kernel: [79423.789652] [<c01c3a75>] (oom_kill_process) from [<c01c3e19>] (out_of_memory+0xc5/0x224) Mar 10 06:55:39 nari kernel: [79423.789665] [<c01c3e19>] (out_of_memory) from [<c01c758d>] (__alloc_pages_nodemask+0x8d1/0x950) Mar 10 06:55:39 nari kernel: [79423.789678] [<c01c758d>] (__alloc_pages_nodemask) from [<c01c050f>] (pagecache_get_page+0xbb/0x218) Mar 10 06:55:39 nari kernel: [79423.789695] [<c01c050f>] (pagecache_get_page) from [<c023196d>] (__getblk_gfp+0xcd/0x288) Mar 10 06:55:39 nari kernel: [79423.789714] [<c023196d>] (__getblk_gfp) from [<c026b8e7>] (__ext4_get_inode_loc+0xc3/0x374) Mar 10 06:55:39 nari kernel: [79423.789730] [<c026b8e7>] (__ext4_get_inode_loc) from [<c026ef2f>] (ext4_reserve_inode_write+0x23/0x60) Mar 10 06:55:39 nari kernel: [79423.789740] [<c026ef2f>] (ext4_reserve_inode_write) from [<c026ef93>] (ext4_mark_inode_dirty+0x27/0x15c) Mar 10 06:55:39 nari kernel: [79423.789751] [<c026ef93>] (ext4_mark_inode_dirty) from [<c027239f>] (ext4_dirty_inode+0x33/0x4c) Mar 10 06:55:39 nari kernel: [79423.789763] [<c027239f>] (ext4_dirty_inode) from [<c022b55b>] (__mark_inode_dirty+0x2b/0x2d4) Mar 10 06:55:39 nari kernel: [79423.789777] [<c022b55b>] (__mark_inode_dirty) from [<c021e16d>] (generic_update_time+0x6d/0xa0) Mar 10 06:55:39 nari kernel: [79423.789790] [<c021e16d>] (generic_update_time) from [<c021e377>] (file_update_time+0x77/0xa4) Mar 10 06:55:39 nari kernel: [79423.789803] [<c021e377>] (file_update_time) from [<c01c1a59>] (__generic_file_write_iter+0x6d/0x174) Mar 10 06:55:39 nari kernel: [79423.789816] [<c01c1a59>] (__generic_file_write_iter) from [<c02681ef>] (ext4_file_write_iter+0xa7/0x2e8) Mar 10 06:55:39 nari kernel: [79423.789832] [<c02681ef>] (ext4_file_write_iter) from [<c0209b6f>] (new_sync_write+0x7b/0xa0) Mar 10 06:55:39 nari kernel: [79423.789844] [<c0209b6f>] (new_sync_write) from [<c020a5c7>] (vfs_write+0x77/0x140) Mar 10 06:55:39 nari kernel: [79423.789854] [<c020a5c7>] (vfs_write) from [<c020b1b1>] (SyS_write+0x31/0x6c) Mar 10 06:55:39 nari kernel: [79423.789868] [<c020b1b1>] (SyS_write) from [<c01060c1>] (ret_fast_syscall+0x1/0x4c) Mar 10 06:55:39 nari kernel: [79423.789873] Mem-Info: Mar 10 06:55:39 nari kernel: [79423.789893] active_anon:2116 inactive_anon:11194 isolated_anon:0 Mar 10 06:55:39 nari kernel: [79423.789893] active_file:107982 inactive_file:147878 isolated_file:32 Mar 10 06:55:39 nari kernel: [79423.789893] unevictable:0 dirty:0 writeback:0 unstable:0 Mar 10 06:55:39 nari kernel: [79423.789893] slab_reclaimable:182227 slab_unreclaimable:2742 Mar 10 06:55:39 nari kernel: [79423.789893] mapped:6873 shmem:4296 pagetables:349 bounce:0 Mar 10 06:55:39 nari kernel: [79423.789893] free:55967 free_pcp:522 free_cma:0 Mar 10 06:55:39 nari kernel: [79423.789912] Node 0 active_anon:8464kB inactive_anon:44776kB active_file:431928kB inactive_file:591512kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:27492kB dirty:0kB writeback:0kB shmem:17184kB writeback_tmp:0kB unstable:0kB pages_scanned:11196848 all_unreclaimable? yes Mar 10 06:55:39 nari kernel: [79423.789931] Normal free:3452kB min:3460kB low:4324kB high:5188kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:44kB unevictable:0kB writepending:0kB present:786432kB managed:753372kB mlocked:0kB slab_reclaimable:728908kB slab_unreclaimable:10968kB kernel_stack:992kB pagetables:0kB bounce:0kB free_pcp:1260kB local_pcp:716kB free_cma:0kB Mar 10 06:55:39 nari kernel: lowmem_reserve[]: 0 10088 10088 Mar 10 06:55:39 nari kernel: [79423.789961] HighMem free:220416kB min:512kB low:2004kB high:3496kB active_anon:8464kB inactive_anon:44776kB active_file:431912kB inactive_file:591468kB unevictable:0kB writepending:0kB present:1307648kB managed:1307648kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:1396kB bounce:0kB free_pcp:828kB local_pcp:732kB free_cma:0kB Mar 10 06:55:39 nari kernel: lowmem_reserve[]: 0 0 0 Mar 10 06:55:39 nari kernel: [79423.789973] Normal: 607*4kB (UME) 128*8kB (ME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3452kB Mar 10 06:55:39 nari kernel: HighMem: 0*4kB 468*8kB (UM) 404*16kB (UM) 289*32kB (M) 266*64kB (UM) 109*128kB (M) 68*256kB (M) 64*512kB (UM) 19*1024kB (UM) 19*2048kB (UM) 15*4096kB (M) = 220416kB : Edited March 11, 2017 by Igor logging under spoiler Link to comment Share on other sites More sharing options...
Igor Posted March 11, 2017 Share Posted March 11, 2017 7 hours ago, timmr said: Any suggestions what I should do? In general: - going back to old kernel, where this problem does not manifest (?), - try new (developing) kernel to see if this is already fixed, - try to fix the problem and provide patch, Link to comment Share on other sites More sharing options...
Guest thom_arm Posted April 1, 2017 Share Posted April 1, 2017 Hi, meanwhile I am absolutely sure that this behaviour can not be observed at least with Kernel 4.8.4 from linux-image-next-sunxi_5.23_armhf.deb (and most probably all other version before). Something in 5.25 (did not ran this for a long time) and/or 5.26 may have introduced this. I did a rollback to 4.8.4 from linux-image-next-sunxi_5.23_armhf.deb about one week ago and found no issues so far. Nothing else changed during this time beside the kernel rollback. Dropping caches with a cronjob (0 */2 * * * sync && echo 1 > /proc/sys/vm/drop_caches) also did not helped for me (found this in some blog from a RedHat Guy) when running the 4.9.something kernel from package 5.26. Unfortunately I have no spare time to dig deeper into this - just want to leave it here for others facing this. Greetings. Thom Link to comment Share on other sites More sharing options...
timmr Posted April 2, 2017 Share Posted April 2, 2017 Hi, thanks Igor for your answer (and of course for creating/developing armbian!) and also Thom for your add-on. Unfortunately at the moment, I do not have the knowledge (and also probably not enough time) to do more analysis... so I will try to downgrade to 5.23... (as the problem still persists - I reboot nearly every day because of this). Is it enough to do a "sudo apt-get install linux-image-next-sunxi=5.23" to downgrade the kernel? (or should I at least also downgrade linux-headers? Anything else?) (Never done kernel downgrading before, sorry) Thanks in advance... Timm Link to comment Share on other sites More sharing options...
Guest thom_arm Posted April 3, 2017 Share Posted April 3, 2017 Hi, I downloaded the following files from the repository: linux-dtb-next-sunxi_5.23_armhf.deb linux-headers-next-sunxi_5.23_armhf.deb linux-jessie-root-next-cubietruck_5.23_armhf.deb linux-firmware-image-next-sunxi_5.23_armhf.deb linux-image-next-sunxi_5.23_armhf.deb and installed them manually using 'dpkg -i'. Maybe you want to check with --dry-run first to see what will happen. Additionally I added the following to /etc/apt/preferences.d/50-armbian.pref in order to prevent checking for new kernels for the moment: Package: linux-dtb-next-sunxi linux-firmware-image-next-sunxi linux-headers-next-sunxi linux-image-next-sunxi linux-jessie-root-next-cubietruck Pin: origin "apt.armbian.com" Pin-Priority: -10 Regards Thom Link to comment Share on other sites More sharing options...
Recommended Posts