Rami Khawaly Posted February 1, 2019 Share Posted February 1, 2019 (edited) Armbianmonitor: http://ix.io/1zHc I'm trying to figure out what happened in my device this morning. the process that i'm monitoring was interrupted and restarted and i'm want to know that reason, is it internal code reasone or system failure that caused to my process to crash. i'm attaching the /var/log/message piece of logs when my process crashed. can you tell me what happened? I'm using NanoPi2 NEO with latest armbian version for this specific board, Thanks alot Spoiler Feb 1 05:16:13 nanopi NetworkManager[642]: <info> [1548990973.1753] dhcp4 (eth0): address 10.0.0.5 Feb 1 05:16:13 nanopi NetworkManager[642]: <info> [1548990973.1822] dhcp4 (eth0): plen 24 (255.255.255.0) Feb 1 05:16:13 nanopi NetworkManager[642]: <info> [1548990973.1824] dhcp4 (eth0): gateway 10.0.0.138 Feb 1 05:16:13 nanopi NetworkManager[642]: <info> [1548990973.1825] dhcp4 (eth0): server identifier 10.0.0.138 Feb 1 05:16:13 nanopi NetworkManager[642]: <info> [1548990973.1827] dhcp4 (eth0): lease time 3600 Feb 1 05:16:13 nanopi NetworkManager[642]: <info> [1548990973.1828] dhcp4 (eth0): nameserver '10.0.0.138' Feb 1 05:16:13 nanopi NetworkManager[642]: <info> [1548990973.1829] dhcp4 (eth0): domain name 'mynet' Feb 1 05:16:13 nanopi NetworkManager[642]: <info> [1548990973.1831] dhcp4 (eth0): state changed bound -> bound Feb 1 05:39:37 nanopi kernel: [860647.689864] winbindd invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0 Feb 1 05:39:37 nanopi kernel: [860647.689880] winbindd cpuset=/ mems_allowed=0 Feb 1 05:39:37 nanopi kernel: [860647.689899] CPU: 3 PID: 31841 Comm: winbindd Not tainted 4.14.78-sunxi64 #416 Feb 1 05:39:37 nanopi kernel: [860647.689902] Hardware name: FriendlyARM NanoPi NEO 2 (DT) Feb 1 05:39:37 nanopi kernel: [860647.689906] Call trace: Feb 1 05:39:37 nanopi kernel: [860647.689926] [<ffff0000080893e8>] dump_backtrace+0x0/0x3f0 Feb 1 05:39:37 nanopi kernel: [860647.689934] [<ffff0000080897ec>] show_stack+0x14/0x20 Feb 1 05:39:37 nanopi kernel: [860647.689942] [<ffff000008968c08>] dump_stack+0x9c/0xbc Feb 1 05:39:37 nanopi kernel: [860647.689951] [<ffff00000816ea48>] dump_header+0x8c/0x1d4 Feb 1 05:39:37 nanopi kernel: [860647.689958] [<ffff00000816dc28>] oom_kill_process+0x280/0x538 Feb 1 05:39:37 nanopi kernel: [860647.689964] [<ffff00000816e5e0>] out_of_memory+0x1b8/0x4f0 Feb 1 05:39:37 nanopi kernel: [860647.689973] [<ffff000008173f84>] __alloc_pages_nodemask+0xb1c/0xbb8 Feb 1 05:39:37 nanopi kernel: [860647.689981] [<ffff0000081c703c>] alloc_pages_current+0x7c/0xe0 Feb 1 05:39:37 nanopi kernel: [860647.689989] [<ffff000008169cc0>] __page_cache_alloc+0x90/0xb0 Feb 1 05:39:37 nanopi kernel: [860647.689994] [<ffff00000816b650>] filemap_fault+0x378/0x598 Feb 1 05:39:37 nanopi kernel: [860647.690005] [<ffff0000082a5b38>] ext4_filemap_fault+0x30/0x50 Feb 1 05:39:37 nanopi kernel: [860647.690014] [<ffff00000819bd68>] __do_fault+0x20/0x70 Feb 1 05:39:37 nanopi kernel: [860647.690021] [<ffff0000081a1474>] __handle_mm_fault+0x7cc/0xc60 Feb 1 05:39:37 nanopi kernel: [860647.690026] [<ffff0000081a19d0>] handle_mm_fault+0xc8/0x168 Feb 1 05:39:37 nanopi kernel: [860647.690033] [<ffff000008097adc>] do_page_fault+0x1ac/0x3c8 Feb 1 05:39:37 nanopi kernel: [860647.690039] [<ffff000008097d34>] do_translation_fault+0x3c/0x48 Feb 1 05:39:37 nanopi kernel: [860647.690045] [<ffff000008080b14>] do_mem_abort+0x54/0xd0 Feb 1 05:39:37 nanopi kernel: [860647.690050] [<ffff000008080c18>] do_el0_ia_bp_hardening+0x38/0x98 Feb 1 05:39:37 nanopi kernel: [860647.690056] Exception stack(0xffff000012733ec0 to 0xffff000012734000) Feb 1 05:39:37 nanopi kernel: [860647.690063] 3ec0: 0000aaaad6c85720 0000000000000022 0000000000000022 0000000000000000 Feb 1 05:39:37 nanopi kernel: [860647.690070] 3ee0: 00000000ffffffff 0000ffffb00869c8 0000000000000000 0000000000000000 Feb 1 05:39:37 nanopi kernel: [860647.690076] 3f00: 0000000000000000 0000aaaad6c85340 0000ffffe3816760 657461766972702f Feb 1 05:39:37 nanopi kernel: [860647.690082] 3f20: 4d2d62772e676f6c 4546494c4f444e49 000000000000001e 0000000000000001 Feb 1 05:39:37 nanopi kernel: [860647.690089] 3f40: 0000ffffb015ff00 0000ffffaffb6130 0000000000000000 0000ffffb043a1d0 Feb 1 05:39:37 nanopi kernel: [860647.690095] 3f60: 0000aaaad6c87610 0000aaaad6c82ea0 0000aaaad6c85720 0000aaaac6d86810 Feb 1 05:39:37 nanopi kernel: [860647.690102] 3f80: 0000aaaac6d020a0 0000aaaac6d85000 0000aaaad6c82270 0000aaaac6d866d0 Feb 1 05:39:37 nanopi kernel: [860647.690108] 3fa0: 0000aaaad6ca0b50 0000ffffe3816a60 0000ffffb04000f0 0000ffffe3816a60 Feb 1 05:39:37 nanopi kernel: [860647.690114] 3fc0: 0000ffffb03d79a0 0000000040000000 0000000000000000 00000000ffffffff Feb 1 05:39:37 nanopi kernel: [860647.690120] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Feb 1 05:39:37 nanopi kernel: [860647.690126] [<ffff000008082e88>] el0_ia+0x18/0x1c Feb 1 05:39:37 nanopi kernel: [860647.690129] Mem-Info: Feb 1 05:39:37 nanopi kernel: [860647.690144] active_anon:44926 inactive_anon:45119 isolated_anon:0 Feb 1 05:39:37 nanopi kernel: [860647.690144] active_file:114 inactive_file:327 isolated_file:1 Feb 1 05:39:37 nanopi kernel: [860647.690144] unevictable:0 dirty:0 writeback:0 unstable:0 Feb 1 05:39:37 nanopi kernel: [860647.690144] slab_reclaimable:2812 slab_unreclaimable:5668 Feb 1 05:39:37 nanopi kernel: [860647.690144] mapped:1095 shmem:1716 pagetables:1379 bounce:0 Feb 1 05:39:37 nanopi kernel: [860647.690144] free:580 free_pcp:28 free_cma:144 Feb 1 05:39:37 nanopi kernel: [860647.690158] Node 0 active_anon:179704kB inactive_anon:180476kB active_file:456kB inactive_file:1308kB unevictable:0kB isolated(anon):0kB isolated(file):4kB mapped:4380kB dirty:0kB writeback:0kB shmem:6864kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Feb 1 05:39:37 nanopi kernel: [860647.690161] Node 0 DMA free:2320kB min:2392kB low:2988kB high:3584kB active_anon:179628kB inactive_anon:180476kB active_file:444kB inactive_file:1400kB unevictable:0kB writepending:0kB present:524288kB managed:494180kB mlocked:0kB kernel_stack:3152kB pagetables:5516kB bounce:0kB free_pcp:112kB local_pcp:52kB free_cma:576kB Feb 1 05:39:37 nanopi kernel: [860647.690177] lowmem_reserve[]: 0 0 0 Feb 1 05:39:37 nanopi kernel: [860647.690186] Node 0 DMA: 41*4kB (UEHC) 19*8kB (UEHC) 15*16kB (UEHC) 34*32kB (UEHC) 4*64kB (H) 4*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2412kB Feb 1 05:39:37 nanopi kernel: [860647.690259] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Feb 1 05:39:37 nanopi kernel: [860647.690262] 2866 total pagecache pages Feb 1 05:39:37 nanopi kernel: [860647.690267] 597 pages in swap cache Feb 1 05:39:37 nanopi kernel: [860647.690271] Swap cache stats: add 10130530, delete 10129801, find 3596714/8480500 Feb 1 05:39:37 nanopi kernel: [860647.690274] Free swap = 0kB Feb 1 05:39:37 nanopi kernel: [860647.690276] Total swap = 247088kB Feb 1 05:39:37 nanopi kernel: [860647.690279] 131072 pages RAM Feb 1 05:39:37 nanopi kernel: [860647.690281] 0 pages HighMem/MovableOnly Feb 1 05:39:37 nanopi kernel: [860647.690284] 7527 pages reserved Feb 1 05:39:37 nanopi kernel: [860647.690286] 32768 pages cma reserved Feb 1 05:39:37 nanopi kernel: [860647.690289] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Feb 1 05:39:37 nanopi kernel: [860647.690327] [ 259] 0 259 6152 123 9 4 87 0 systemd-journal Feb 1 05:39:37 nanopi kernel: [860647.690335] [ 316] 0 316 3492 15 8 3 193 -1000 systemd-udevd Feb 1 05:39:37 nanopi kernel: [860647.690344] [ 365] 0 365 1905 2 8 4 775 0 haveged Feb 1 05:39:37 nanopi kernel: [860647.690353] [ 592] 0 592 1894 32 8 4 101 0 systemd-logind Feb 1 05:39:37 nanopi kernel: [860647.690363] [ 602] 0 602 54561 29 12 4 441 0 rsyslogd Feb 1 05:39:37 nanopi kernel: [860647.690372] [ 607] 107 607 1612 82 7 4 74 -900 dbus-daemon Feb 1 05:39:37 nanopi kernel: [860647.690380] [ 642] 0 642 70151 595 28 3 1297 0 NetworkManager Feb 1 05:39:37 nanopi kernel: [860647.690388] [ 653] 0 653 1321 26 6 4 46 0 cron Feb 1 05:39:37 nanopi kernel: [860647.690397] [ 985] 109 985 50192 40 29 3 401 -900 postgres Feb 1 05:39:37 nanopi kernel: [860647.690405] [ 1010] 110 1010 2239 11 8 4 89 0 dnsmasq Feb 1 05:39:37 nanopi kernel: [860647.690414] [ 1077] 0 1077 2597 2 9 4 186 -1000 sshd Feb 1 05:39:37 nanopi kernel: [860647.690422] [ 1105] 0 1105 58483 77 16 4 189 0 polkitd Feb 1 05:39:37 nanopi kernel: [860647.690431] [ 1181] 109 1181 50223 525 33 3 405 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.690439] [ 1182] 109 1182 50192 27 24 3 414 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.690448] [ 1183] 109 1183 50192 313 24 3 420 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.690457] [ 1184] 109 1184 50293 135 26 3 437 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.690466] [ 1185] 109 1185 13976 64 20 3 402 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.690475] [ 1339] 0 1339 978 0 6 4 37 0 agetty Feb 1 05:39:37 nanopi kernel: [860647.690483] [ 1344] 0 1344 1034 0 6 3 38 0 agetty Feb 1 05:39:37 nanopi kernel: [860647.690492] [ 1349] 106 1349 18207 50 9 3 93 0 ntpd Feb 1 05:39:37 nanopi kernel: [860647.690501] [ 1355] 0 1355 6870 55 17 4 397 0 nmbd Feb 1 05:39:37 nanopi kernel: [860647.690510] [ 1370] 0 1370 8659 57 20 3 483 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.690519] [ 1390] 0 1390 10657 42 24 4 590 0 smbd Feb 1 05:39:37 nanopi kernel: [860647.690528] [ 1391] 0 1391 10126 10 23 4 595 0 smbd-notifyd Feb 1 05:39:37 nanopi kernel: [860647.690537] [ 1392] 0 1392 10126 10 22 4 595 0 cleanupd Feb 1 05:39:37 nanopi kernel: [860647.690546] [ 1397] 0 1397 8659 25 20 3 515 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.690555] [ 1399] 0 1399 8690 26 20 3 516 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.690564] [ 1403] 0 1403 10660 45 23 4 587 0 lpqd Feb 1 05:39:37 nanopi kernel: [860647.690572] [10243] 0 10243 1761 58 7 4 203 0 dhclient Feb 1 05:39:37 nanopi kernel: [860647.690581] [30651] 0 30651 932126 27866 257 8 34926 0 java Feb 1 05:39:37 nanopi kernel: [860647.690590] [30692] 109 30692 50859 939 33 3 656 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.690599] [30693] 109 30693 50719 841 33 3 567 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.690607] [30694] 109 30694 50885 824 33 3 695 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.690616] [30695] 109 30695 50739 875 33 3 556 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.690625] [30696] 109 30696 50875 942 32 3 683 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.690634] [28511] 0 28511 166639 1719 100 5 3123 0 node Feb 1 05:39:37 nanopi kernel: [860647.690643] [28527] 0 28527 185131 2884 106 5 4060 0 node Feb 1 05:39:37 nanopi kernel: [860647.690655] [31650] 0 31650 466 17 6 3 0 0 sh Feb 1 05:39:37 nanopi kernel: [860647.690664] [31654] 0 31654 261216 51653 130 5 57 0 python Feb 1 05:39:37 nanopi kernel: [860647.690671] [31828] 0 31828 440 16 4 3 0 0 sh Feb 1 05:39:37 nanopi kernel: [860647.690680] [31832] 0 31832 1321 47 6 4 25 0 cron Feb 1 05:39:37 nanopi kernel: [860647.690689] [31835] 0 31835 465 24 5 4 0 0 modprobe Feb 1 05:39:37 nanopi kernel: [860647.690698] [31838] 109 31838 50192 74 21 3 376 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.690707] [31839] 0 31839 1321 40 6 4 32 0 cron Feb 1 05:39:37 nanopi kernel: [860647.690715] [31840] 0 31840 10661 81 24 4 551 0 lpqd Feb 1 05:39:37 nanopi kernel: [860647.690723] [31841] 0 31841 8659 145 18 3 395 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.690732] [31842] 0 31842 1321 38 6 4 34 0 cron Feb 1 05:39:37 nanopi kernel: [860647.690741] [31846] 0 31846 10657 104 22 4 528 0 smbd Feb 1 05:39:37 nanopi kernel: [860647.690750] [31850] 0 31850 6870 71 16 4 381 0 nmbd Feb 1 05:39:37 nanopi kernel: [860647.695392] oom_reaper: reaped process 31650 (sh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Feb 1 05:39:37 nanopi kernel: [860647.696751] winbindd invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0 Feb 1 05:39:37 nanopi kernel: [860647.696763] winbindd cpuset=/ mems_allowed=0 Feb 1 05:39:37 nanopi kernel: [860647.696781] CPU: 3 PID: 31841 Comm: winbindd Not tainted 4.14.78-sunxi64 #416 Feb 1 05:39:37 nanopi kernel: [860647.696784] Hardware name: FriendlyARM NanoPi NEO 2 (DT) Feb 1 05:39:37 nanopi kernel: [860647.696787] Call trace: Feb 1 05:39:37 nanopi kernel: [860647.696803] [<ffff0000080893e8>] dump_backtrace+0x0/0x3f0 Feb 1 05:39:37 nanopi kernel: [860647.696811] [<ffff0000080897ec>] show_stack+0x14/0x20 Feb 1 05:39:37 nanopi kernel: [860647.696819] [<ffff000008968c08>] dump_stack+0x9c/0xbc Feb 1 05:39:37 nanopi kernel: [860647.696827] [<ffff00000816ea48>] dump_header+0x8c/0x1d4 Feb 1 05:39:37 nanopi kernel: [860647.696834] [<ffff00000816dc28>] oom_kill_process+0x280/0x538 Feb 1 05:39:37 nanopi kernel: [860647.696840] [<ffff00000816e5e0>] out_of_memory+0x1b8/0x4f0 Feb 1 05:39:37 nanopi kernel: [860647.696848] [<ffff000008173f84>] __alloc_pages_nodemask+0xb1c/0xbb8 Feb 1 05:39:37 nanopi kernel: [860647.696857] [<ffff0000081c703c>] alloc_pages_current+0x7c/0xe0 Feb 1 05:39:37 nanopi kernel: [860647.696863] [<ffff000008169cc0>] __page_cache_alloc+0x90/0xb0 Feb 1 05:39:37 nanopi kernel: [860647.696869] [<ffff00000816b650>] filemap_fault+0x378/0x598 Feb 1 05:39:37 nanopi kernel: [860647.696880] [<ffff0000082a5b38>] ext4_filemap_fault+0x30/0x50 Feb 1 05:39:37 nanopi kernel: [860647.696889] [<ffff00000819bd68>] __do_fault+0x20/0x70 Feb 1 05:39:37 nanopi kernel: [860647.696895] [<ffff0000081a1474>] __handle_mm_fault+0x7cc/0xc60 Feb 1 05:39:37 nanopi kernel: [860647.696901] [<ffff0000081a19d0>] handle_mm_fault+0xc8/0x168 Feb 1 05:39:37 nanopi kernel: [860647.696908] [<ffff000008097adc>] do_page_fault+0x1ac/0x3c8 Feb 1 05:39:37 nanopi kernel: [860647.696914] [<ffff000008097d34>] do_translation_fault+0x3c/0x48 Feb 1 05:39:37 nanopi kernel: [860647.696920] [<ffff000008080b14>] do_mem_abort+0x54/0xd0 Feb 1 05:39:37 nanopi kernel: [860647.696925] [<ffff000008080c18>] do_el0_ia_bp_hardening+0x38/0x98 Feb 1 05:39:37 nanopi kernel: [860647.696931] Exception stack(0xffff000012733ec0 to 0xffff000012734000) Feb 1 05:39:37 nanopi kernel: [860647.696938] 3ec0: 0000aaaad6c85720 0000000000000022 0000000000000022 0000000000000000 Feb 1 05:39:37 nanopi kernel: [860647.696944] 3ee0: 00000000ffffffff 0000ffffb00869c8 0000000000000000 0000000000000000 Feb 1 05:39:37 nanopi kernel: [860647.696950] 3f00: 0000000000000000 0000aaaad6c85340 0000ffffe3816760 657461766972702f Feb 1 05:39:37 nanopi kernel: [860647.696956] 3f20: 4d2d62772e676f6c 4546494c4f444e49 000000000000001e 0000000000000001 Feb 1 05:39:37 nanopi kernel: [860647.696963] 3f40: 0000ffffb015ff00 0000ffffaffb6130 0000000000000000 0000ffffb043a1d0 Feb 1 05:39:37 nanopi kernel: [860647.696969] 3f60: 0000aaaad6c87610 0000aaaad6c82ea0 0000aaaad6c85720 0000aaaac6d86810 Feb 1 05:39:37 nanopi kernel: [860647.696976] 3f80: 0000aaaac6d020a0 0000aaaac6d85000 0000aaaad6c82270 0000aaaac6d866d0 Feb 1 05:39:37 nanopi kernel: [860647.696982] 3fa0: 0000aaaad6ca0b50 0000ffffe3816a60 0000ffffb04000f0 0000ffffe3816a60 Feb 1 05:39:37 nanopi kernel: [860647.696988] 3fc0: 0000ffffb03d79a0 0000000040000000 0000000000000000 00000000ffffffff Feb 1 05:39:37 nanopi kernel: [860647.696994] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Feb 1 05:39:37 nanopi kernel: [860647.697000] [<ffff000008082e88>] el0_ia+0x18/0x1c Feb 1 05:39:37 nanopi kernel: [860647.697002] Mem-Info: Feb 1 05:39:37 nanopi kernel: [860647.697017] active_anon:44926 inactive_anon:45100 isolated_anon:0 Feb 1 05:39:37 nanopi kernel: [860647.697017] active_file:122 inactive_file:319 isolated_file:32 Feb 1 05:39:37 nanopi kernel: [860647.697017] unevictable:0 dirty:0 writeback:0 unstable:0 Feb 1 05:39:37 nanopi kernel: [860647.697017] slab_reclaimable:2812 slab_unreclaimable:5668 Feb 1 05:39:37 nanopi kernel: [860647.697017] mapped:1095 shmem:1716 pagetables:1379 bounce:0 Feb 1 05:39:37 nanopi kernel: [860647.697017] free:580 free_pcp:53 free_cma:144 Feb 1 05:39:37 nanopi kernel: [860647.697031] Node 0 active_anon:179704kB inactive_anon:180400kB active_file:488kB inactive_file:1276kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:4380kB dirty:0kB writeback:0kB shmem:6864kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Feb 1 05:39:37 nanopi kernel: [860647.697034] Node 0 DMA free:2320kB min:2392kB low:2988kB high:3584kB active_anon:179628kB inactive_anon:180476kB active_file:420kB inactive_file:1216kB unevictable:0kB writepending:0kB present:524288kB managed:494180kB mlocked:0kB kernel_stack:3152kB pagetables:5516kB bounce:0kB free_pcp:212kB local_pcp:72kB free_cma:576kB Feb 1 05:39:37 nanopi kernel: [860647.697049] lowmem_reserve[]: 0 0 0 Feb 1 05:39:37 nanopi kernel: [860647.697058] Node 0 DMA: 60*4kB (MEHC) 20*8kB (UEHC) 15*16kB (UEHC) 33*32kB (UEHC) 4*64kB (H) 4*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2464kB Feb 1 05:39:37 nanopi kernel: [860647.697106] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Feb 1 05:39:37 nanopi kernel: [860647.697108] 2828 total pagecache pages Feb 1 05:39:37 nanopi kernel: [860647.697113] 597 pages in swap cache Feb 1 05:39:37 nanopi kernel: [860647.697118] Swap cache stats: add 10130530, delete 10129801, find 3596714/8480501 Feb 1 05:39:37 nanopi kernel: [860647.697120] Free swap = 0kB Feb 1 05:39:37 nanopi kernel: [860647.697123] Total swap = 247088kB Feb 1 05:39:37 nanopi kernel: [860647.697126] 131072 pages RAM Feb 1 05:39:37 nanopi kernel: [860647.697128] 0 pages HighMem/MovableOnly Feb 1 05:39:37 nanopi kernel: [860647.697131] 7527 pages reserved Feb 1 05:39:37 nanopi kernel: [860647.697133] 32768 pages cma reserved Feb 1 05:39:37 nanopi kernel: [860647.697136] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Feb 1 05:39:37 nanopi kernel: [860647.697173] [ 259] 0 259 6152 123 9 4 87 0 systemd-journal Feb 1 05:39:37 nanopi kernel: [860647.697183] [ 316] 0 316 3492 15 8 3 193 -1000 systemd-udevd Feb 1 05:39:37 nanopi kernel: [860647.697192] [ 365] 0 365 1905 2 8 4 775 0 haveged Feb 1 05:39:37 nanopi kernel: [860647.697202] [ 592] 0 592 1894 32 8 4 101 0 systemd-logind Feb 1 05:39:37 nanopi kernel: [860647.697212] [ 602] 0 602 54561 69 12 4 403 0 rsyslogd Feb 1 05:39:37 nanopi kernel: [860647.697221] [ 607] 107 607 1612 82 7 4 74 -900 dbus-daemon Feb 1 05:39:37 nanopi kernel: [860647.697255] [ 642] 0 642 70151 595 28 3 1297 0 NetworkManager Feb 1 05:39:37 nanopi kernel: [860647.697263] [ 653] 0 653 1321 26 6 4 46 0 cron Feb 1 05:39:37 nanopi kernel: [860647.697273] [ 985] 109 985 50192 40 29 3 401 -900 postgres Feb 1 05:39:37 nanopi kernel: [860647.697281] [ 1010] 110 1010 2239 11 8 4 89 0 dnsmasq Feb 1 05:39:37 nanopi kernel: [860647.697291] [ 1077] 0 1077 2597 2 9 4 186 -1000 sshd Feb 1 05:39:37 nanopi kernel: [860647.697300] [ 1105] 0 1105 58483 77 16 4 189 0 polkitd Feb 1 05:39:37 nanopi kernel: [860647.697308] [ 1181] 109 1181 50223 525 33 3 405 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.697316] [ 1182] 109 1182 50192 27 24 3 414 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.697325] [ 1183] 109 1183 50192 313 24 3 420 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.697334] [ 1184] 109 1184 50293 135 26 3 437 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.697343] [ 1185] 109 1185 13976 64 20 3 402 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.697353] [ 1339] 0 1339 978 0 6 4 37 0 agetty Feb 1 05:39:37 nanopi kernel: [860647.697361] [ 1344] 0 1344 1034 0 6 3 38 0 agetty Feb 1 05:39:37 nanopi kernel: [860647.697370] [ 1349] 106 1349 18207 50 9 3 93 0 ntpd Feb 1 05:39:37 nanopi kernel: [860647.697379] [ 1355] 0 1355 6870 55 17 4 397 0 nmbd Feb 1 05:39:37 nanopi kernel: [860647.697388] [ 1370] 0 1370 8659 57 20 3 483 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.697397] [ 1390] 0 1390 10657 42 24 4 590 0 smbd Feb 1 05:39:37 nanopi kernel: [860647.697406] [ 1391] 0 1391 10126 10 23 4 595 0 smbd-notifyd Feb 1 05:39:37 nanopi kernel: [860647.697414] [ 1392] 0 1392 10126 10 22 4 595 0 cleanupd Feb 1 05:39:37 nanopi kernel: [860647.697423] [ 1397] 0 1397 8659 25 20 3 515 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.697432] [ 1399] 0 1399 8690 26 20 3 516 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.697441] [ 1403] 0 1403 10660 45 23 4 587 0 lpqd Feb 1 05:39:37 nanopi kernel: [860647.697450] [10243] 0 10243 1761 58 7 4 203 0 dhclient Feb 1 05:39:37 nanopi kernel: [860647.697459] [30651] 0 30651 932126 27866 257 8 34926 0 java Feb 1 05:39:37 nanopi kernel: [860647.697467] [30692] 109 30692 50859 939 33 3 656 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.697476] [30693] 109 30693 50719 841 33 3 567 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.697485] [30694] 109 30694 50885 824 33 3 695 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.697494] [30695] 109 30695 50739 875 33 3 556 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.697503] [30696] 109 30696 50875 942 32 3 683 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.697513] [28511] 0 28511 166639 1719 100 5 3123 0 node Feb 1 05:39:37 nanopi kernel: [860647.697522] [28527] 0 28527 185131 2884 106 5 4060 0 node Feb 1 05:39:37 nanopi kernel: [860647.697536] [31654] 0 31654 261216 51653 130 5 57 0 python Feb 1 05:39:37 nanopi kernel: [860647.697544] [31828] 0 31828 440 16 4 3 0 0 sh Feb 1 05:39:37 nanopi kernel: [860647.697552] [31832] 0 31832 1321 47 6 4 25 0 cron Feb 1 05:39:37 nanopi kernel: [860647.697561] [31835] 0 31835 465 24 5 4 0 0 modprobe Feb 1 05:39:37 nanopi kernel: [860647.697570] [31838] 109 31838 50192 74 21 3 376 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.697579] [31839] 0 31839 1321 40 6 4 32 0 cron Feb 1 05:39:37 nanopi kernel: [860647.697587] [31840] 0 31840 10661 81 24 4 551 0 lpqd Feb 1 05:39:37 nanopi kernel: [860647.697594] [31841] 0 31841 8659 145 18 3 395 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.697603] [31842] 0 31842 1321 38 6 4 34 0 cron Feb 1 05:39:37 nanopi kernel: [860647.697612] [31846] 0 31846 10657 104 22 4 528 0 smbd Feb 1 05:39:37 nanopi kernel: [860647.697622] [31850] 0 31850 6870 71 16 4 381 0 nmbd Feb 1 05:39:37 nanopi kernel: [860647.721670] java invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0 Feb 1 05:39:37 nanopi kernel: [860647.721686] java cpuset=/ mems_allowed=0 Feb 1 05:39:37 nanopi kernel: [860647.721705] CPU: 0 PID: 31843 Comm: java Not tainted 4.14.78-sunxi64 #416 Feb 1 05:39:37 nanopi kernel: [860647.721708] Hardware name: FriendlyARM NanoPi NEO 2 (DT) Feb 1 05:39:37 nanopi kernel: [860647.721712] Call trace: Feb 1 05:39:37 nanopi kernel: [860647.721731] [<ffff0000080893e8>] dump_backtrace+0x0/0x3f0 Feb 1 05:39:37 nanopi kernel: [860647.721739] [<ffff0000080897ec>] show_stack+0x14/0x20 Feb 1 05:39:37 nanopi kernel: [860647.721747] [<ffff000008968c08>] dump_stack+0x9c/0xbc Feb 1 05:39:37 nanopi kernel: [860647.721756] [<ffff00000816ea48>] dump_header+0x8c/0x1d4 Feb 1 05:39:37 nanopi kernel: [860647.721763] [<ffff00000816dc28>] oom_kill_process+0x280/0x538 Feb 1 05:39:37 nanopi kernel: [860647.721769] [<ffff00000816e5e0>] out_of_memory+0x1b8/0x4f0 Feb 1 05:39:37 nanopi kernel: [860647.721777] [<ffff000008173f84>] __alloc_pages_nodemask+0xb1c/0xbb8 Feb 1 05:39:37 nanopi kernel: [860647.721786] [<ffff0000081c703c>] alloc_pages_current+0x7c/0xe0 Feb 1 05:39:37 nanopi kernel: [860647.721792] [<ffff000008169cc0>] __page_cache_alloc+0x90/0xb0 Feb 1 05:39:37 nanopi kernel: [860647.721798] [<ffff00000816b650>] filemap_fault+0x378/0x598 Feb 1 05:39:37 nanopi kernel: [860647.721809] [<ffff0000082a5b38>] ext4_filemap_fault+0x30/0x50 Feb 1 05:39:37 nanopi kernel: [860647.721818] [<ffff00000819bd68>] __do_fault+0x20/0x70 Feb 1 05:39:37 nanopi kernel: [860647.721825] [<ffff0000081a1474>] __handle_mm_fault+0x7cc/0xc60 Feb 1 05:39:37 nanopi kernel: [860647.721830] [<ffff0000081a19d0>] handle_mm_fault+0xc8/0x168 Feb 1 05:39:37 nanopi kernel: [860647.721837] [<ffff000008097adc>] do_page_fault+0x1ac/0x3c8 Feb 1 05:39:37 nanopi kernel: [860647.721843] [<ffff000008097d34>] do_translation_fault+0x3c/0x48 Feb 1 05:39:37 nanopi kernel: [860647.721849] [<ffff000008080b14>] do_mem_abort+0x54/0xd0 Feb 1 05:39:37 nanopi kernel: [860647.721854] [<ffff000008080c18>] do_el0_ia_bp_hardening+0x38/0x98 Feb 1 05:39:37 nanopi kernel: [860647.721860] Exception stack(0xffff0000127a3ec0 to 0xffff0000127a4000) Feb 1 05:39:37 nanopi kernel: [860647.721867] 3ec0: 00000000f556efa0 0000ffff98055200 0000ffff98054fa0 0000ffff78dd5af8 Feb 1 05:39:37 nanopi kernel: [860647.721873] 3ee0: 000000010000d6f8 0000000000000000 0000ffff78dd5af8 0000ffff9801e7b0 Feb 1 05:39:37 nanopi kernel: [860647.721879] 3f00: 0000000000000000 0000ffff48006d40 0000ffff9d0f50b8 0000ffff9d0f4fd8 Feb 1 05:39:37 nanopi kernel: [860647.721885] 3f20: 0000000000000001 0000ffff9d0f50b0 000000010000d6f8 0000ffff9d025d20 Feb 1 05:39:37 nanopi kernel: [860647.721892] 3f40: 0000ffff9eb7d388 0000ffff9edb5050 0000ffff9d0240a0 0000ffff9d021660 Feb 1 05:39:37 nanopi kernel: [860647.721898] 3f60: 0000ffff9eb7b000 0000000000000000 000000000f56ee8e 0000ffff78dd5af8 Feb 1 05:39:37 nanopi kernel: [860647.721903] 3f80: 0000ffff9d021660 000000010000d6f8 0000ffff98054f60 000000000000004c Feb 1 05:39:37 nanopi kernel: [860647.721910] 3fa0: 000000007bf7aa0c 0000ffff12ffc100 0000ffff9e7962b4 0000ffff12ffc100 Feb 1 05:39:37 nanopi kernel: [860647.721916] 3fc0: 0000ffff9e687530 0000000060000000 0000ffff60041128 00000000ffffffff Feb 1 05:39:37 nanopi kernel: [860647.721922] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Feb 1 05:39:37 nanopi kernel: [860647.721927] [<ffff000008082e88>] el0_ia+0x18/0x1c Feb 1 05:39:37 nanopi kernel: [860647.721930] Mem-Info: Feb 1 05:39:37 nanopi kernel: [860647.721946] active_anon:44935 inactive_anon:45119 isolated_anon:0 Feb 1 05:39:37 nanopi kernel: [860647.721946] active_file:136 inactive_file:315 isolated_file:32 Feb 1 05:39:37 nanopi kernel: [860647.721946] unevictable:0 dirty:0 writeback:0 unstable:0 Feb 1 05:39:37 nanopi kernel: [860647.721946] slab_reclaimable:2812 slab_unreclaimable:5668 Feb 1 05:39:37 nanopi kernel: [860647.721946] mapped:1110 shmem:1716 pagetables:1369 bounce:0 Feb 1 05:39:37 nanopi kernel: [860647.721946] free:589 free_pcp:0 free_cma:103 Feb 1 05:39:37 nanopi kernel: [860647.721959] Node 0 active_anon:179740kB inactive_anon:180476kB active_file:544kB inactive_file:1260kB unevictable:0kB isolated(anon):0kB isolated(file):88kB mapped:4440kB dirty:0kB writeback:0kB shmem:6864kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Feb 1 05:39:37 nanopi kernel: [860647.721962] Node 0 DMA free:2356kB min:2392kB low:2988kB high:3584kB active_anon:179868kB inactive_anon:180476kB active_file:512kB inactive_file:1292kB unevictable:0kB writepending:0kB present:524288kB managed:494180kB mlocked:0kB kernel_stack:3120kB pagetables:5476kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:412kB Feb 1 05:39:37 nanopi kernel: [860647.721978] lowmem_reserve[]: 0 0 0 Feb 1 05:39:37 nanopi kernel: [860647.721987] Node 0 DMA: 60*4kB (UMEHC) 20*8kB (UEHC) 12*16kB (UEHC) 34*32kB (UEHC) 4*64kB (H) 4*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2448kB Feb 1 05:39:37 nanopi kernel: [860647.722033] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Feb 1 05:39:37 nanopi kernel: [860647.722035] 2802 total pagecache pages Feb 1 05:39:37 nanopi kernel: [860647.722041] 597 pages in swap cache Feb 1 05:39:37 nanopi kernel: [860647.722045] Swap cache stats: add 10130530, delete 10129801, find 3596714/8480501 Feb 1 05:39:37 nanopi kernel: [860647.722048] Free swap = 0kB Feb 1 05:39:37 nanopi kernel: [860647.722050] Total swap = 247088kB Feb 1 05:39:37 nanopi kernel: [860647.722053] 131072 pages RAM Feb 1 05:39:37 nanopi kernel: [860647.722055] 0 pages HighMem/MovableOnly Feb 1 05:39:37 nanopi kernel: [860647.722057] 7527 pages reserved Feb 1 05:39:37 nanopi kernel: [860647.722060] 32768 pages cma reserved Feb 1 05:39:37 nanopi kernel: [860647.722063] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Feb 1 05:39:37 nanopi kernel: [860647.722099] [ 259] 0 259 6152 123 9 4 87 0 systemd-journal Feb 1 05:39:37 nanopi kernel: [860647.722108] [ 316] 0 316 3492 15 8 3 193 -1000 systemd-udevd Feb 1 05:39:37 nanopi kernel: [860647.722117] [ 365] 0 365 1905 2 8 4 775 0 haveged Feb 1 05:39:37 nanopi kernel: [860647.722126] [ 592] 0 592 1894 32 8 4 101 0 systemd-logind Feb 1 05:39:37 nanopi kernel: [860647.722135] [ 602] 0 602 54561 69 12 4 403 0 rsyslogd Feb 1 05:39:37 nanopi kernel: [860647.722144] [ 607] 107 607 1612 82 7 4 74 -900 dbus-daemon Feb 1 05:39:37 nanopi kernel: [860647.722152] [ 642] 0 642 70151 595 28 3 1297 0 NetworkManager Feb 1 05:39:37 nanopi kernel: [860647.722160] [ 653] 0 653 1321 26 6 4 46 0 cron Feb 1 05:39:37 nanopi kernel: [860647.722168] [ 985] 109 985 50192 40 29 3 401 -900 postgres Feb 1 05:39:37 nanopi kernel: [860647.722176] [ 1010] 110 1010 2239 11 8 4 89 0 dnsmasq Feb 1 05:39:37 nanopi kernel: [860647.722185] [ 1077] 0 1077 2597 2 9 4 186 -1000 sshd Feb 1 05:39:37 nanopi kernel: [860647.722194] [ 1105] 0 1105 58483 77 16 4 189 0 polkitd Feb 1 05:39:37 nanopi kernel: [860647.722202] [ 1181] 109 1181 50223 524 33 3 405 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.722210] [ 1182] 109 1182 50192 27 24 3 414 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.722219] [ 1183] 109 1183 50192 313 24 3 420 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.722228] [ 1184] 109 1184 50293 135 26 3 437 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.722256] [ 1185] 109 1185 13976 64 20 3 402 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.722264] [ 1339] 0 1339 978 0 6 4 37 0 agetty Feb 1 05:39:37 nanopi kernel: [860647.722272] [ 1344] 0 1344 1034 0 6 3 38 0 agetty Feb 1 05:39:37 nanopi kernel: [860647.722281] [ 1349] 106 1349 18207 50 9 3 93 0 ntpd Feb 1 05:39:37 nanopi kernel: [860647.722289] [ 1355] 0 1355 6870 55 17 4 397 0 nmbd Feb 1 05:39:37 nanopi kernel: [860647.722297] [ 1370] 0 1370 8659 57 20 3 483 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.722306] [ 1390] 0 1390 10657 42 24 4 590 0 smbd Feb 1 05:39:37 nanopi kernel: [860647.722315] [ 1391] 0 1391 10126 10 23 4 595 0 smbd-notifyd Feb 1 05:39:37 nanopi kernel: [860647.722324] [ 1392] 0 1392 10126 10 22 4 595 0 cleanupd Feb 1 05:39:37 nanopi kernel: [860647.722333] [ 1397] 0 1397 8659 25 20 3 515 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.722342] [ 1399] 0 1399 8690 26 20 3 516 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.722350] [ 1403] 0 1403 10660 45 23 4 587 0 lpqd Feb 1 05:39:37 nanopi kernel: [860647.722358] [10243] 0 10243 1761 58 7 4 203 0 dhclient Feb 1 05:39:37 nanopi kernel: [860647.722367] [30651] 0 30651 932126 27866 257 8 34926 0 java Feb 1 05:39:37 nanopi kernel: [860647.722375] [30692] 109 30692 50859 939 33 3 656 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.722385] [30693] 109 30693 50719 841 33 3 567 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.722393] [30694] 109 30694 50885 824 33 3 695 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.722402] [30695] 109 30695 50739 875 33 3 556 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.722411] [30696] 109 30696 50875 942 32 3 683 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.722420] [28511] 0 28511 166639 1719 100 5 3123 0 node Feb 1 05:39:37 nanopi kernel: [860647.722429] [28527] 0 28527 185131 2884 106 5 4060 0 node Feb 1 05:39:37 nanopi kernel: [860647.722441] [31654] 0 31654 261216 51719 130 5 57 0 python Feb 1 05:39:37 nanopi kernel: [860647.722452] [31832] 0 31832 1321 47 6 4 25 0 cron Feb 1 05:39:37 nanopi kernel: [860647.722461] [31835] 0 31835 465 24 5 4 0 0 modprobe Feb 1 05:39:37 nanopi kernel: [860647.722471] [31838] 109 31838 50192 74 21 3 376 0 postgres Feb 1 05:39:37 nanopi kernel: [860647.722480] [31839] 0 31839 1321 40 6 4 32 0 cron Feb 1 05:39:37 nanopi kernel: [860647.722488] [31840] 0 31840 10661 81 24 4 551 0 lpqd Feb 1 05:39:37 nanopi kernel: [860647.722496] [31841] 0 31841 8659 145 18 3 395 0 winbindd Feb 1 05:39:37 nanopi kernel: [860647.722504] [31842] 0 31842 1321 38 6 4 34 0 cron Feb 1 05:39:37 nanopi kernel: [860647.722513] [31846] 0 31846 10657 104 22 4 528 0 smbd Feb 1 05:39:37 nanopi kernel: [860647.722523] [31850] 0 31850 6870 71 16 4 381 0 nmbd Feb 1 05:39:37 nanopi kernel: [860647.959894] oom_reaper: reaped process 30651 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:16kB Feb 1 05:39:39 nanopi NetworkManager[642]: <info> [1548992379.4651] dhcp4 (eth0): address 10.0.0.5 Feb 1 05:39:39 nanopi NetworkManager[642]: <info> [1548992379.4940] dhcp4 (eth0): plen 24 (255.255.255.0) Feb 1 05:39:39 nanopi NetworkManager[642]: <info> [1548992379.4941] dhcp4 (eth0): gateway 10.0.0.138 Feb 1 05:39:39 nanopi NetworkManager[642]: <info> [1548992379.4941] dhcp4 (eth0): server identifier 10.0.0.138 Feb 1 05:39:39 nanopi NetworkManager[642]: <info> [1548992379.4942] dhcp4 (eth0): lease time 3600 Feb 1 05:39:39 nanopi NetworkManager[642]: <info> [1548992379.4943] dhcp4 (eth0): nameserver '10.0.0.138' Feb 1 05:39:39 nanopi NetworkManager[642]: <info> [1548992379.4944] dhcp4 (eth0): domain name 'mynet' Feb 1 05:39:39 nanopi NetworkManager[642]: <info> [1548992379.4946] dhcp4 (eth0): state changed bound -> bound Edited February 1, 2019 by Igor spoilers Link to comment Share on other sites More sharing options...
Igor Posted February 1, 2019 Share Posted February 1, 2019 You are running out of memory. Link to comment Share on other sites More sharing options...
Rami Khawaly Posted February 2, 2019 Author Share Posted February 2, 2019 Right, the unclear point is that is it the Java process that is running out of memory or another process? I can't understand the mentioned pid's? What is more strange is that the same process is running on raspberrypi with exactly same params (the process is running with 256m memory) without memory problems. Is there any difference between memory management on armbian and raspbian? Link to comment Share on other sites More sharing options...
Recommended Posts