Jump to content

Understanding system logs to investigate process crash reason


Rami Khawaly

Recommended Posts

Armbianmonitor:

I'm trying to figure out what happened in my device this morning. the process that i'm monitoring was interrupted and restarted and i'm want to know that reason, is it internal code reasone or system failure that caused to my process to crash. 

i'm attaching the /var/log/message piece of logs when my process crashed.

can you tell me what happened? 

I'm using NanoPi2 NEO with latest armbian version for this specific board, Thanks alot

 

Spoiler

Feb  1 05:16:13 nanopi NetworkManager[642]: <info>  [1548990973.1753] dhcp4 (eth0):   address 10.0.0.5
Feb  1 05:16:13 nanopi NetworkManager[642]: <info>  [1548990973.1822] dhcp4 (eth0):   plen 24 (255.255.255.0)
Feb  1 05:16:13 nanopi NetworkManager[642]: <info>  [1548990973.1824] dhcp4 (eth0):   gateway 10.0.0.138
Feb  1 05:16:13 nanopi NetworkManager[642]: <info>  [1548990973.1825] dhcp4 (eth0):   server identifier 10.0.0.138
Feb  1 05:16:13 nanopi NetworkManager[642]: <info>  [1548990973.1827] dhcp4 (eth0):   lease time 3600
Feb  1 05:16:13 nanopi NetworkManager[642]: <info>  [1548990973.1828] dhcp4 (eth0):   nameserver '10.0.0.138'
Feb  1 05:16:13 nanopi NetworkManager[642]: <info>  [1548990973.1829] dhcp4 (eth0):   domain name 'mynet'
Feb  1 05:16:13 nanopi NetworkManager[642]: <info>  [1548990973.1831] dhcp4 (eth0): state changed bound -> bound
Feb  1 05:39:37 nanopi kernel: [860647.689864] winbindd invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null),  order=0, oom_score_adj=0
Feb  1 05:39:37 nanopi kernel: [860647.689880] winbindd cpuset=/ mems_allowed=0
Feb  1 05:39:37 nanopi kernel: [860647.689899] CPU: 3 PID: 31841 Comm: winbindd Not tainted 4.14.78-sunxi64 #416
Feb  1 05:39:37 nanopi kernel: [860647.689902] Hardware name: FriendlyARM NanoPi NEO 2 (DT)
Feb  1 05:39:37 nanopi kernel: [860647.689906] Call trace:
Feb  1 05:39:37 nanopi kernel: [860647.689926] [<ffff0000080893e8>] dump_backtrace+0x0/0x3f0
Feb  1 05:39:37 nanopi kernel: [860647.689934] [<ffff0000080897ec>] show_stack+0x14/0x20
Feb  1 05:39:37 nanopi kernel: [860647.689942] [<ffff000008968c08>] dump_stack+0x9c/0xbc
Feb  1 05:39:37 nanopi kernel: [860647.689951] [<ffff00000816ea48>] dump_header+0x8c/0x1d4
Feb  1 05:39:37 nanopi kernel: [860647.689958] [<ffff00000816dc28>] oom_kill_process+0x280/0x538
Feb  1 05:39:37 nanopi kernel: [860647.689964] [<ffff00000816e5e0>] out_of_memory+0x1b8/0x4f0
Feb  1 05:39:37 nanopi kernel: [860647.689973] [<ffff000008173f84>] __alloc_pages_nodemask+0xb1c/0xbb8
Feb  1 05:39:37 nanopi kernel: [860647.689981] [<ffff0000081c703c>] alloc_pages_current+0x7c/0xe0
Feb  1 05:39:37 nanopi kernel: [860647.689989] [<ffff000008169cc0>] __page_cache_alloc+0x90/0xb0
Feb  1 05:39:37 nanopi kernel: [860647.689994] [<ffff00000816b650>] filemap_fault+0x378/0x598
Feb  1 05:39:37 nanopi kernel: [860647.690005] [<ffff0000082a5b38>] ext4_filemap_fault+0x30/0x50
Feb  1 05:39:37 nanopi kernel: [860647.690014] [<ffff00000819bd68>] __do_fault+0x20/0x70
Feb  1 05:39:37 nanopi kernel: [860647.690021] [<ffff0000081a1474>] __handle_mm_fault+0x7cc/0xc60
Feb  1 05:39:37 nanopi kernel: [860647.690026] [<ffff0000081a19d0>] handle_mm_fault+0xc8/0x168
Feb  1 05:39:37 nanopi kernel: [860647.690033] [<ffff000008097adc>] do_page_fault+0x1ac/0x3c8
Feb  1 05:39:37 nanopi kernel: [860647.690039] [<ffff000008097d34>] do_translation_fault+0x3c/0x48
Feb  1 05:39:37 nanopi kernel: [860647.690045] [<ffff000008080b14>] do_mem_abort+0x54/0xd0
Feb  1 05:39:37 nanopi kernel: [860647.690050] [<ffff000008080c18>] do_el0_ia_bp_hardening+0x38/0x98
Feb  1 05:39:37 nanopi kernel: [860647.690056] Exception stack(0xffff000012733ec0 to 0xffff000012734000)
Feb  1 05:39:37 nanopi kernel: [860647.690063] 3ec0: 0000aaaad6c85720 0000000000000022 0000000000000022 0000000000000000
Feb  1 05:39:37 nanopi kernel: [860647.690070] 3ee0: 00000000ffffffff 0000ffffb00869c8 0000000000000000 0000000000000000
Feb  1 05:39:37 nanopi kernel: [860647.690076] 3f00: 0000000000000000 0000aaaad6c85340 0000ffffe3816760 657461766972702f
Feb  1 05:39:37 nanopi kernel: [860647.690082] 3f20: 4d2d62772e676f6c 4546494c4f444e49 000000000000001e 0000000000000001
Feb  1 05:39:37 nanopi kernel: [860647.690089] 3f40: 0000ffffb015ff00 0000ffffaffb6130 0000000000000000 0000ffffb043a1d0
Feb  1 05:39:37 nanopi kernel: [860647.690095] 3f60: 0000aaaad6c87610 0000aaaad6c82ea0 0000aaaad6c85720 0000aaaac6d86810
Feb  1 05:39:37 nanopi kernel: [860647.690102] 3f80: 0000aaaac6d020a0 0000aaaac6d85000 0000aaaad6c82270 0000aaaac6d866d0
Feb  1 05:39:37 nanopi kernel: [860647.690108] 3fa0: 0000aaaad6ca0b50 0000ffffe3816a60 0000ffffb04000f0 0000ffffe3816a60
Feb  1 05:39:37 nanopi kernel: [860647.690114] 3fc0: 0000ffffb03d79a0 0000000040000000 0000000000000000 00000000ffffffff
Feb  1 05:39:37 nanopi kernel: [860647.690120] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Feb  1 05:39:37 nanopi kernel: [860647.690126] [<ffff000008082e88>] el0_ia+0x18/0x1c
Feb  1 05:39:37 nanopi kernel: [860647.690129] Mem-Info:
Feb  1 05:39:37 nanopi kernel: [860647.690144] active_anon:44926 inactive_anon:45119 isolated_anon:0
Feb  1 05:39:37 nanopi kernel: [860647.690144]  active_file:114 inactive_file:327 isolated_file:1
Feb  1 05:39:37 nanopi kernel: [860647.690144]  unevictable:0 dirty:0 writeback:0 unstable:0
Feb  1 05:39:37 nanopi kernel: [860647.690144]  slab_reclaimable:2812 slab_unreclaimable:5668
Feb  1 05:39:37 nanopi kernel: [860647.690144]  mapped:1095 shmem:1716 pagetables:1379 bounce:0
Feb  1 05:39:37 nanopi kernel: [860647.690144]  free:580 free_pcp:28 free_cma:144
Feb  1 05:39:37 nanopi kernel: [860647.690158] Node 0 active_anon:179704kB inactive_anon:180476kB active_file:456kB inactive_file:1308kB unevictable:0kB isolated(anon):0kB isolated(file):4kB mapped:4380kB dirty:0kB writeback:0kB shmem:6864kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Feb  1 05:39:37 nanopi kernel: [860647.690161] Node 0 DMA free:2320kB min:2392kB low:2988kB high:3584kB active_anon:179628kB inactive_anon:180476kB active_file:444kB inactive_file:1400kB unevictable:0kB writepending:0kB present:524288kB managed:494180kB mlocked:0kB kernel_stack:3152kB pagetables:5516kB bounce:0kB free_pcp:112kB local_pcp:52kB free_cma:576kB
Feb  1 05:39:37 nanopi kernel: [860647.690177] lowmem_reserve[]: 0 0 0
Feb  1 05:39:37 nanopi kernel: [860647.690186] Node 0 DMA: 41*4kB (UEHC) 19*8kB (UEHC) 15*16kB (UEHC) 34*32kB (UEHC) 4*64kB (H) 4*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2412kB
Feb  1 05:39:37 nanopi kernel: [860647.690259] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb  1 05:39:37 nanopi kernel: [860647.690262] 2866 total pagecache pages
Feb  1 05:39:37 nanopi kernel: [860647.690267] 597 pages in swap cache
Feb  1 05:39:37 nanopi kernel: [860647.690271] Swap cache stats: add 10130530, delete 10129801, find 3596714/8480500
Feb  1 05:39:37 nanopi kernel: [860647.690274] Free swap  = 0kB
Feb  1 05:39:37 nanopi kernel: [860647.690276] Total swap = 247088kB
Feb  1 05:39:37 nanopi kernel: [860647.690279] 131072 pages RAM
Feb  1 05:39:37 nanopi kernel: [860647.690281] 0 pages HighMem/MovableOnly
Feb  1 05:39:37 nanopi kernel: [860647.690284] 7527 pages reserved
Feb  1 05:39:37 nanopi kernel: [860647.690286] 32768 pages cma reserved
Feb  1 05:39:37 nanopi kernel: [860647.690289] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Feb  1 05:39:37 nanopi kernel: [860647.690327] [  259]     0   259     6152      123       9       4       87             0 systemd-journal
Feb  1 05:39:37 nanopi kernel: [860647.690335] [  316]     0   316     3492       15       8       3      193         -1000 systemd-udevd
Feb  1 05:39:37 nanopi kernel: [860647.690344] [  365]     0   365     1905        2       8       4      775             0 haveged
Feb  1 05:39:37 nanopi kernel: [860647.690353] [  592]     0   592     1894       32       8       4      101             0 systemd-logind
Feb  1 05:39:37 nanopi kernel: [860647.690363] [  602]     0   602    54561       29      12       4      441             0 rsyslogd
Feb  1 05:39:37 nanopi kernel: [860647.690372] [  607]   107   607     1612       82       7       4       74          -900 dbus-daemon
Feb  1 05:39:37 nanopi kernel: [860647.690380] [  642]     0   642    70151      595      28       3     1297             0 NetworkManager
Feb  1 05:39:37 nanopi kernel: [860647.690388] [  653]     0   653     1321       26       6       4       46             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.690397] [  985]   109   985    50192       40      29       3      401          -900 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690405] [ 1010]   110  1010     2239       11       8       4       89             0 dnsmasq
Feb  1 05:39:37 nanopi kernel: [860647.690414] [ 1077]     0  1077     2597        2       9       4      186         -1000 sshd
Feb  1 05:39:37 nanopi kernel: [860647.690422] [ 1105]     0  1105    58483       77      16       4      189             0 polkitd
Feb  1 05:39:37 nanopi kernel: [860647.690431] [ 1181]   109  1181    50223      525      33       3      405             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690439] [ 1182]   109  1182    50192       27      24       3      414             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690448] [ 1183]   109  1183    50192      313      24       3      420             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690457] [ 1184]   109  1184    50293      135      26       3      437             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690466] [ 1185]   109  1185    13976       64      20       3      402             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690475] [ 1339]     0  1339      978        0       6       4       37             0 agetty
Feb  1 05:39:37 nanopi kernel: [860647.690483] [ 1344]     0  1344     1034        0       6       3       38             0 agetty
Feb  1 05:39:37 nanopi kernel: [860647.690492] [ 1349]   106  1349    18207       50       9       3       93             0 ntpd
Feb  1 05:39:37 nanopi kernel: [860647.690501] [ 1355]     0  1355     6870       55      17       4      397             0 nmbd
Feb  1 05:39:37 nanopi kernel: [860647.690510] [ 1370]     0  1370     8659       57      20       3      483             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.690519] [ 1390]     0  1390    10657       42      24       4      590             0 smbd
Feb  1 05:39:37 nanopi kernel: [860647.690528] [ 1391]     0  1391    10126       10      23       4      595             0 smbd-notifyd
Feb  1 05:39:37 nanopi kernel: [860647.690537] [ 1392]     0  1392    10126       10      22       4      595             0 cleanupd
Feb  1 05:39:37 nanopi kernel: [860647.690546] [ 1397]     0  1397     8659       25      20       3      515             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.690555] [ 1399]     0  1399     8690       26      20       3      516             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.690564] [ 1403]     0  1403    10660       45      23       4      587             0 lpqd
Feb  1 05:39:37 nanopi kernel: [860647.690572] [10243]     0 10243     1761       58       7       4      203             0 dhclient
Feb  1 05:39:37 nanopi kernel: [860647.690581] [30651]     0 30651   932126    27866     257       8    34926             0 java
Feb  1 05:39:37 nanopi kernel: [860647.690590] [30692]   109 30692    50859      939      33       3      656             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690599] [30693]   109 30693    50719      841      33       3      567             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690607] [30694]   109 30694    50885      824      33       3      695             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690616] [30695]   109 30695    50739      875      33       3      556             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690625] [30696]   109 30696    50875      942      32       3      683             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690634] [28511]     0 28511   166639     1719     100       5     3123             0 node
Feb  1 05:39:37 nanopi kernel: [860647.690643] [28527]     0 28527   185131     2884     106       5     4060             0 node
Feb  1 05:39:37 nanopi kernel: [860647.690655] [31650]     0 31650      466       17       6       3        0             0 sh
Feb  1 05:39:37 nanopi kernel: [860647.690664] [31654]     0 31654   261216    51653     130       5       57             0 python
Feb  1 05:39:37 nanopi kernel: [860647.690671] [31828]     0 31828      440       16       4       3        0             0 sh
Feb  1 05:39:37 nanopi kernel: [860647.690680] [31832]     0 31832     1321       47       6       4       25             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.690689] [31835]     0 31835      465       24       5       4        0             0 modprobe
Feb  1 05:39:37 nanopi kernel: [860647.690698] [31838]   109 31838    50192       74      21       3      376             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.690707] [31839]     0 31839     1321       40       6       4       32             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.690715] [31840]     0 31840    10661       81      24       4      551             0 lpqd
Feb  1 05:39:37 nanopi kernel: [860647.690723] [31841]     0 31841     8659      145      18       3      395             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.690732] [31842]     0 31842     1321       38       6       4       34             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.690741] [31846]     0 31846    10657      104      22       4      528             0 smbd
Feb  1 05:39:37 nanopi kernel: [860647.690750] [31850]     0 31850     6870       71      16       4      381             0 nmbd
Feb  1 05:39:37 nanopi kernel: [860647.695392] oom_reaper: reaped process 31650 (sh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Feb  1 05:39:37 nanopi kernel: [860647.696751] winbindd invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null),  order=0, oom_score_adj=0
Feb  1 05:39:37 nanopi kernel: [860647.696763] winbindd cpuset=/ mems_allowed=0
Feb  1 05:39:37 nanopi kernel: [860647.696781] CPU: 3 PID: 31841 Comm: winbindd Not tainted 4.14.78-sunxi64 #416
Feb  1 05:39:37 nanopi kernel: [860647.696784] Hardware name: FriendlyARM NanoPi NEO 2 (DT)
Feb  1 05:39:37 nanopi kernel: [860647.696787] Call trace:
Feb  1 05:39:37 nanopi kernel: [860647.696803] [<ffff0000080893e8>] dump_backtrace+0x0/0x3f0
Feb  1 05:39:37 nanopi kernel: [860647.696811] [<ffff0000080897ec>] show_stack+0x14/0x20
Feb  1 05:39:37 nanopi kernel: [860647.696819] [<ffff000008968c08>] dump_stack+0x9c/0xbc
Feb  1 05:39:37 nanopi kernel: [860647.696827] [<ffff00000816ea48>] dump_header+0x8c/0x1d4
Feb  1 05:39:37 nanopi kernel: [860647.696834] [<ffff00000816dc28>] oom_kill_process+0x280/0x538
Feb  1 05:39:37 nanopi kernel: [860647.696840] [<ffff00000816e5e0>] out_of_memory+0x1b8/0x4f0
Feb  1 05:39:37 nanopi kernel: [860647.696848] [<ffff000008173f84>] __alloc_pages_nodemask+0xb1c/0xbb8
Feb  1 05:39:37 nanopi kernel: [860647.696857] [<ffff0000081c703c>] alloc_pages_current+0x7c/0xe0
Feb  1 05:39:37 nanopi kernel: [860647.696863] [<ffff000008169cc0>] __page_cache_alloc+0x90/0xb0
Feb  1 05:39:37 nanopi kernel: [860647.696869] [<ffff00000816b650>] filemap_fault+0x378/0x598
Feb  1 05:39:37 nanopi kernel: [860647.696880] [<ffff0000082a5b38>] ext4_filemap_fault+0x30/0x50
Feb  1 05:39:37 nanopi kernel: [860647.696889] [<ffff00000819bd68>] __do_fault+0x20/0x70
Feb  1 05:39:37 nanopi kernel: [860647.696895] [<ffff0000081a1474>] __handle_mm_fault+0x7cc/0xc60
Feb  1 05:39:37 nanopi kernel: [860647.696901] [<ffff0000081a19d0>] handle_mm_fault+0xc8/0x168
Feb  1 05:39:37 nanopi kernel: [860647.696908] [<ffff000008097adc>] do_page_fault+0x1ac/0x3c8
Feb  1 05:39:37 nanopi kernel: [860647.696914] [<ffff000008097d34>] do_translation_fault+0x3c/0x48
Feb  1 05:39:37 nanopi kernel: [860647.696920] [<ffff000008080b14>] do_mem_abort+0x54/0xd0
Feb  1 05:39:37 nanopi kernel: [860647.696925] [<ffff000008080c18>] do_el0_ia_bp_hardening+0x38/0x98
Feb  1 05:39:37 nanopi kernel: [860647.696931] Exception stack(0xffff000012733ec0 to 0xffff000012734000)
Feb  1 05:39:37 nanopi kernel: [860647.696938] 3ec0: 0000aaaad6c85720 0000000000000022 0000000000000022 0000000000000000
Feb  1 05:39:37 nanopi kernel: [860647.696944] 3ee0: 00000000ffffffff 0000ffffb00869c8 0000000000000000 0000000000000000
Feb  1 05:39:37 nanopi kernel: [860647.696950] 3f00: 0000000000000000 0000aaaad6c85340 0000ffffe3816760 657461766972702f
Feb  1 05:39:37 nanopi kernel: [860647.696956] 3f20: 4d2d62772e676f6c 4546494c4f444e49 000000000000001e 0000000000000001
Feb  1 05:39:37 nanopi kernel: [860647.696963] 3f40: 0000ffffb015ff00 0000ffffaffb6130 0000000000000000 0000ffffb043a1d0
Feb  1 05:39:37 nanopi kernel: [860647.696969] 3f60: 0000aaaad6c87610 0000aaaad6c82ea0 0000aaaad6c85720 0000aaaac6d86810
Feb  1 05:39:37 nanopi kernel: [860647.696976] 3f80: 0000aaaac6d020a0 0000aaaac6d85000 0000aaaad6c82270 0000aaaac6d866d0
Feb  1 05:39:37 nanopi kernel: [860647.696982] 3fa0: 0000aaaad6ca0b50 0000ffffe3816a60 0000ffffb04000f0 0000ffffe3816a60
Feb  1 05:39:37 nanopi kernel: [860647.696988] 3fc0: 0000ffffb03d79a0 0000000040000000 0000000000000000 00000000ffffffff
Feb  1 05:39:37 nanopi kernel: [860647.696994] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Feb  1 05:39:37 nanopi kernel: [860647.697000] [<ffff000008082e88>] el0_ia+0x18/0x1c
Feb  1 05:39:37 nanopi kernel: [860647.697002] Mem-Info:
Feb  1 05:39:37 nanopi kernel: [860647.697017] active_anon:44926 inactive_anon:45100 isolated_anon:0
Feb  1 05:39:37 nanopi kernel: [860647.697017]  active_file:122 inactive_file:319 isolated_file:32
Feb  1 05:39:37 nanopi kernel: [860647.697017]  unevictable:0 dirty:0 writeback:0 unstable:0
Feb  1 05:39:37 nanopi kernel: [860647.697017]  slab_reclaimable:2812 slab_unreclaimable:5668
Feb  1 05:39:37 nanopi kernel: [860647.697017]  mapped:1095 shmem:1716 pagetables:1379 bounce:0
Feb  1 05:39:37 nanopi kernel: [860647.697017]  free:580 free_pcp:53 free_cma:144
Feb  1 05:39:37 nanopi kernel: [860647.697031] Node 0 active_anon:179704kB inactive_anon:180400kB active_file:488kB inactive_file:1276kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:4380kB dirty:0kB writeback:0kB shmem:6864kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Feb  1 05:39:37 nanopi kernel: [860647.697034] Node 0 DMA free:2320kB min:2392kB low:2988kB high:3584kB active_anon:179628kB inactive_anon:180476kB active_file:420kB inactive_file:1216kB unevictable:0kB writepending:0kB present:524288kB managed:494180kB mlocked:0kB kernel_stack:3152kB pagetables:5516kB bounce:0kB free_pcp:212kB local_pcp:72kB free_cma:576kB
Feb  1 05:39:37 nanopi kernel: [860647.697049] lowmem_reserve[]: 0 0 0
Feb  1 05:39:37 nanopi kernel: [860647.697058] Node 0 DMA: 60*4kB (MEHC) 20*8kB (UEHC) 15*16kB (UEHC) 33*32kB (UEHC) 4*64kB (H) 4*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2464kB
Feb  1 05:39:37 nanopi kernel: [860647.697106] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb  1 05:39:37 nanopi kernel: [860647.697108] 2828 total pagecache pages
Feb  1 05:39:37 nanopi kernel: [860647.697113] 597 pages in swap cache
Feb  1 05:39:37 nanopi kernel: [860647.697118] Swap cache stats: add 10130530, delete 10129801, find 3596714/8480501
Feb  1 05:39:37 nanopi kernel: [860647.697120] Free swap  = 0kB
Feb  1 05:39:37 nanopi kernel: [860647.697123] Total swap = 247088kB
Feb  1 05:39:37 nanopi kernel: [860647.697126] 131072 pages RAM
Feb  1 05:39:37 nanopi kernel: [860647.697128] 0 pages HighMem/MovableOnly
Feb  1 05:39:37 nanopi kernel: [860647.697131] 7527 pages reserved
Feb  1 05:39:37 nanopi kernel: [860647.697133] 32768 pages cma reserved
Feb  1 05:39:37 nanopi kernel: [860647.697136] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Feb  1 05:39:37 nanopi kernel: [860647.697173] [  259]     0   259     6152      123       9       4       87             0 systemd-journal
Feb  1 05:39:37 nanopi kernel: [860647.697183] [  316]     0   316     3492       15       8       3      193         -1000 systemd-udevd
Feb  1 05:39:37 nanopi kernel: [860647.697192] [  365]     0   365     1905        2       8       4      775             0 haveged
Feb  1 05:39:37 nanopi kernel: [860647.697202] [  592]     0   592     1894       32       8       4      101             0 systemd-logind
Feb  1 05:39:37 nanopi kernel: [860647.697212] [  602]     0   602    54561       69      12       4      403             0 rsyslogd
Feb  1 05:39:37 nanopi kernel: [860647.697221] [  607]   107   607     1612       82       7       4       74          -900 dbus-daemon
Feb  1 05:39:37 nanopi kernel: [860647.697255] [  642]     0   642    70151      595      28       3     1297             0 NetworkManager
Feb  1 05:39:37 nanopi kernel: [860647.697263] [  653]     0   653     1321       26       6       4       46             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.697273] [  985]   109   985    50192       40      29       3      401          -900 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697281] [ 1010]   110  1010     2239       11       8       4       89             0 dnsmasq
Feb  1 05:39:37 nanopi kernel: [860647.697291] [ 1077]     0  1077     2597        2       9       4      186         -1000 sshd
Feb  1 05:39:37 nanopi kernel: [860647.697300] [ 1105]     0  1105    58483       77      16       4      189             0 polkitd
Feb  1 05:39:37 nanopi kernel: [860647.697308] [ 1181]   109  1181    50223      525      33       3      405             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697316] [ 1182]   109  1182    50192       27      24       3      414             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697325] [ 1183]   109  1183    50192      313      24       3      420             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697334] [ 1184]   109  1184    50293      135      26       3      437             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697343] [ 1185]   109  1185    13976       64      20       3      402             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697353] [ 1339]     0  1339      978        0       6       4       37             0 agetty
Feb  1 05:39:37 nanopi kernel: [860647.697361] [ 1344]     0  1344     1034        0       6       3       38             0 agetty
Feb  1 05:39:37 nanopi kernel: [860647.697370] [ 1349]   106  1349    18207       50       9       3       93             0 ntpd
Feb  1 05:39:37 nanopi kernel: [860647.697379] [ 1355]     0  1355     6870       55      17       4      397             0 nmbd
Feb  1 05:39:37 nanopi kernel: [860647.697388] [ 1370]     0  1370     8659       57      20       3      483             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.697397] [ 1390]     0  1390    10657       42      24       4      590             0 smbd
Feb  1 05:39:37 nanopi kernel: [860647.697406] [ 1391]     0  1391    10126       10      23       4      595             0 smbd-notifyd
Feb  1 05:39:37 nanopi kernel: [860647.697414] [ 1392]     0  1392    10126       10      22       4      595             0 cleanupd
Feb  1 05:39:37 nanopi kernel: [860647.697423] [ 1397]     0  1397     8659       25      20       3      515             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.697432] [ 1399]     0  1399     8690       26      20       3      516             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.697441] [ 1403]     0  1403    10660       45      23       4      587             0 lpqd
Feb  1 05:39:37 nanopi kernel: [860647.697450] [10243]     0 10243     1761       58       7       4      203             0 dhclient
Feb  1 05:39:37 nanopi kernel: [860647.697459] [30651]     0 30651   932126    27866     257       8    34926             0 java
Feb  1 05:39:37 nanopi kernel: [860647.697467] [30692]   109 30692    50859      939      33       3      656             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697476] [30693]   109 30693    50719      841      33       3      567             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697485] [30694]   109 30694    50885      824      33       3      695             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697494] [30695]   109 30695    50739      875      33       3      556             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697503] [30696]   109 30696    50875      942      32       3      683             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697513] [28511]     0 28511   166639     1719     100       5     3123             0 node
Feb  1 05:39:37 nanopi kernel: [860647.697522] [28527]     0 28527   185131     2884     106       5     4060             0 node
Feb  1 05:39:37 nanopi kernel: [860647.697536] [31654]     0 31654   261216    51653     130       5       57             0 python
Feb  1 05:39:37 nanopi kernel: [860647.697544] [31828]     0 31828      440       16       4       3        0             0 sh
Feb  1 05:39:37 nanopi kernel: [860647.697552] [31832]     0 31832     1321       47       6       4       25             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.697561] [31835]     0 31835      465       24       5       4        0             0 modprobe
Feb  1 05:39:37 nanopi kernel: [860647.697570] [31838]   109 31838    50192       74      21       3      376             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.697579] [31839]     0 31839     1321       40       6       4       32             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.697587] [31840]     0 31840    10661       81      24       4      551             0 lpqd
Feb  1 05:39:37 nanopi kernel: [860647.697594] [31841]     0 31841     8659      145      18       3      395             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.697603] [31842]     0 31842     1321       38       6       4       34             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.697612] [31846]     0 31846    10657      104      22       4      528             0 smbd
Feb  1 05:39:37 nanopi kernel: [860647.697622] [31850]     0 31850     6870       71      16       4      381             0 nmbd
Feb  1 05:39:37 nanopi kernel: [860647.721670] java invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null),  order=0, oom_score_adj=0
Feb  1 05:39:37 nanopi kernel: [860647.721686] java cpuset=/ mems_allowed=0
Feb  1 05:39:37 nanopi kernel: [860647.721705] CPU: 0 PID: 31843 Comm: java Not tainted 4.14.78-sunxi64 #416
Feb  1 05:39:37 nanopi kernel: [860647.721708] Hardware name: FriendlyARM NanoPi NEO 2 (DT)
Feb  1 05:39:37 nanopi kernel: [860647.721712] Call trace:
Feb  1 05:39:37 nanopi kernel: [860647.721731] [<ffff0000080893e8>] dump_backtrace+0x0/0x3f0
Feb  1 05:39:37 nanopi kernel: [860647.721739] [<ffff0000080897ec>] show_stack+0x14/0x20
Feb  1 05:39:37 nanopi kernel: [860647.721747] [<ffff000008968c08>] dump_stack+0x9c/0xbc
Feb  1 05:39:37 nanopi kernel: [860647.721756] [<ffff00000816ea48>] dump_header+0x8c/0x1d4
Feb  1 05:39:37 nanopi kernel: [860647.721763] [<ffff00000816dc28>] oom_kill_process+0x280/0x538
Feb  1 05:39:37 nanopi kernel: [860647.721769] [<ffff00000816e5e0>] out_of_memory+0x1b8/0x4f0
Feb  1 05:39:37 nanopi kernel: [860647.721777] [<ffff000008173f84>] __alloc_pages_nodemask+0xb1c/0xbb8
Feb  1 05:39:37 nanopi kernel: [860647.721786] [<ffff0000081c703c>] alloc_pages_current+0x7c/0xe0
Feb  1 05:39:37 nanopi kernel: [860647.721792] [<ffff000008169cc0>] __page_cache_alloc+0x90/0xb0
Feb  1 05:39:37 nanopi kernel: [860647.721798] [<ffff00000816b650>] filemap_fault+0x378/0x598
Feb  1 05:39:37 nanopi kernel: [860647.721809] [<ffff0000082a5b38>] ext4_filemap_fault+0x30/0x50
Feb  1 05:39:37 nanopi kernel: [860647.721818] [<ffff00000819bd68>] __do_fault+0x20/0x70
Feb  1 05:39:37 nanopi kernel: [860647.721825] [<ffff0000081a1474>] __handle_mm_fault+0x7cc/0xc60
Feb  1 05:39:37 nanopi kernel: [860647.721830] [<ffff0000081a19d0>] handle_mm_fault+0xc8/0x168
Feb  1 05:39:37 nanopi kernel: [860647.721837] [<ffff000008097adc>] do_page_fault+0x1ac/0x3c8
Feb  1 05:39:37 nanopi kernel: [860647.721843] [<ffff000008097d34>] do_translation_fault+0x3c/0x48
Feb  1 05:39:37 nanopi kernel: [860647.721849] [<ffff000008080b14>] do_mem_abort+0x54/0xd0
Feb  1 05:39:37 nanopi kernel: [860647.721854] [<ffff000008080c18>] do_el0_ia_bp_hardening+0x38/0x98
Feb  1 05:39:37 nanopi kernel: [860647.721860] Exception stack(0xffff0000127a3ec0 to 0xffff0000127a4000)
Feb  1 05:39:37 nanopi kernel: [860647.721867] 3ec0: 00000000f556efa0 0000ffff98055200 0000ffff98054fa0 0000ffff78dd5af8
Feb  1 05:39:37 nanopi kernel: [860647.721873] 3ee0: 000000010000d6f8 0000000000000000 0000ffff78dd5af8 0000ffff9801e7b0
Feb  1 05:39:37 nanopi kernel: [860647.721879] 3f00: 0000000000000000 0000ffff48006d40 0000ffff9d0f50b8 0000ffff9d0f4fd8
Feb  1 05:39:37 nanopi kernel: [860647.721885] 3f20: 0000000000000001 0000ffff9d0f50b0 000000010000d6f8 0000ffff9d025d20
Feb  1 05:39:37 nanopi kernel: [860647.721892] 3f40: 0000ffff9eb7d388 0000ffff9edb5050 0000ffff9d0240a0 0000ffff9d021660
Feb  1 05:39:37 nanopi kernel: [860647.721898] 3f60: 0000ffff9eb7b000 0000000000000000 000000000f56ee8e 0000ffff78dd5af8
Feb  1 05:39:37 nanopi kernel: [860647.721903] 3f80: 0000ffff9d021660 000000010000d6f8 0000ffff98054f60 000000000000004c
Feb  1 05:39:37 nanopi kernel: [860647.721910] 3fa0: 000000007bf7aa0c 0000ffff12ffc100 0000ffff9e7962b4 0000ffff12ffc100
Feb  1 05:39:37 nanopi kernel: [860647.721916] 3fc0: 0000ffff9e687530 0000000060000000 0000ffff60041128 00000000ffffffff
Feb  1 05:39:37 nanopi kernel: [860647.721922] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Feb  1 05:39:37 nanopi kernel: [860647.721927] [<ffff000008082e88>] el0_ia+0x18/0x1c
Feb  1 05:39:37 nanopi kernel: [860647.721930] Mem-Info:
Feb  1 05:39:37 nanopi kernel: [860647.721946] active_anon:44935 inactive_anon:45119 isolated_anon:0
Feb  1 05:39:37 nanopi kernel: [860647.721946]  active_file:136 inactive_file:315 isolated_file:32
Feb  1 05:39:37 nanopi kernel: [860647.721946]  unevictable:0 dirty:0 writeback:0 unstable:0
Feb  1 05:39:37 nanopi kernel: [860647.721946]  slab_reclaimable:2812 slab_unreclaimable:5668
Feb  1 05:39:37 nanopi kernel: [860647.721946]  mapped:1110 shmem:1716 pagetables:1369 bounce:0
Feb  1 05:39:37 nanopi kernel: [860647.721946]  free:589 free_pcp:0 free_cma:103
Feb  1 05:39:37 nanopi kernel: [860647.721959] Node 0 active_anon:179740kB inactive_anon:180476kB active_file:544kB inactive_file:1260kB unevictable:0kB isolated(anon):0kB isolated(file):88kB mapped:4440kB dirty:0kB writeback:0kB shmem:6864kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Feb  1 05:39:37 nanopi kernel: [860647.721962] Node 0 DMA free:2356kB min:2392kB low:2988kB high:3584kB active_anon:179868kB inactive_anon:180476kB active_file:512kB inactive_file:1292kB unevictable:0kB writepending:0kB present:524288kB managed:494180kB mlocked:0kB kernel_stack:3120kB pagetables:5476kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:412kB
Feb  1 05:39:37 nanopi kernel: [860647.721978] lowmem_reserve[]: 0 0 0
Feb  1 05:39:37 nanopi kernel: [860647.721987] Node 0 DMA: 60*4kB (UMEHC) 20*8kB (UEHC) 12*16kB (UEHC) 34*32kB (UEHC) 4*64kB (H) 4*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2448kB
Feb  1 05:39:37 nanopi kernel: [860647.722033] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb  1 05:39:37 nanopi kernel: [860647.722035] 2802 total pagecache pages
Feb  1 05:39:37 nanopi kernel: [860647.722041] 597 pages in swap cache
Feb  1 05:39:37 nanopi kernel: [860647.722045] Swap cache stats: add 10130530, delete 10129801, find 3596714/8480501
Feb  1 05:39:37 nanopi kernel: [860647.722048] Free swap  = 0kB
Feb  1 05:39:37 nanopi kernel: [860647.722050] Total swap = 247088kB
Feb  1 05:39:37 nanopi kernel: [860647.722053] 131072 pages RAM
Feb  1 05:39:37 nanopi kernel: [860647.722055] 0 pages HighMem/MovableOnly
Feb  1 05:39:37 nanopi kernel: [860647.722057] 7527 pages reserved
Feb  1 05:39:37 nanopi kernel: [860647.722060] 32768 pages cma reserved
Feb  1 05:39:37 nanopi kernel: [860647.722063] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Feb  1 05:39:37 nanopi kernel: [860647.722099] [  259]     0   259     6152      123       9       4       87             0 systemd-journal
Feb  1 05:39:37 nanopi kernel: [860647.722108] [  316]     0   316     3492       15       8       3      193         -1000 systemd-udevd
Feb  1 05:39:37 nanopi kernel: [860647.722117] [  365]     0   365     1905        2       8       4      775             0 haveged
Feb  1 05:39:37 nanopi kernel: [860647.722126] [  592]     0   592     1894       32       8       4      101             0 systemd-logind
Feb  1 05:39:37 nanopi kernel: [860647.722135] [  602]     0   602    54561       69      12       4      403             0 rsyslogd
Feb  1 05:39:37 nanopi kernel: [860647.722144] [  607]   107   607     1612       82       7       4       74          -900 dbus-daemon
Feb  1 05:39:37 nanopi kernel: [860647.722152] [  642]     0   642    70151      595      28       3     1297             0 NetworkManager
Feb  1 05:39:37 nanopi kernel: [860647.722160] [  653]     0   653     1321       26       6       4       46             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.722168] [  985]   109   985    50192       40      29       3      401          -900 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722176] [ 1010]   110  1010     2239       11       8       4       89             0 dnsmasq
Feb  1 05:39:37 nanopi kernel: [860647.722185] [ 1077]     0  1077     2597        2       9       4      186         -1000 sshd
Feb  1 05:39:37 nanopi kernel: [860647.722194] [ 1105]     0  1105    58483       77      16       4      189             0 polkitd
Feb  1 05:39:37 nanopi kernel: [860647.722202] [ 1181]   109  1181    50223      524      33       3      405             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722210] [ 1182]   109  1182    50192       27      24       3      414             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722219] [ 1183]   109  1183    50192      313      24       3      420             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722228] [ 1184]   109  1184    50293      135      26       3      437             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722256] [ 1185]   109  1185    13976       64      20       3      402             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722264] [ 1339]     0  1339      978        0       6       4       37             0 agetty
Feb  1 05:39:37 nanopi kernel: [860647.722272] [ 1344]     0  1344     1034        0       6       3       38             0 agetty
Feb  1 05:39:37 nanopi kernel: [860647.722281] [ 1349]   106  1349    18207       50       9       3       93             0 ntpd
Feb  1 05:39:37 nanopi kernel: [860647.722289] [ 1355]     0  1355     6870       55      17       4      397             0 nmbd
Feb  1 05:39:37 nanopi kernel: [860647.722297] [ 1370]     0  1370     8659       57      20       3      483             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.722306] [ 1390]     0  1390    10657       42      24       4      590             0 smbd
Feb  1 05:39:37 nanopi kernel: [860647.722315] [ 1391]     0  1391    10126       10      23       4      595             0 smbd-notifyd
Feb  1 05:39:37 nanopi kernel: [860647.722324] [ 1392]     0  1392    10126       10      22       4      595             0 cleanupd
Feb  1 05:39:37 nanopi kernel: [860647.722333] [ 1397]     0  1397     8659       25      20       3      515             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.722342] [ 1399]     0  1399     8690       26      20       3      516             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.722350] [ 1403]     0  1403    10660       45      23       4      587             0 lpqd
Feb  1 05:39:37 nanopi kernel: [860647.722358] [10243]     0 10243     1761       58       7       4      203             0 dhclient
Feb  1 05:39:37 nanopi kernel: [860647.722367] [30651]     0 30651   932126    27866     257       8    34926             0 java
Feb  1 05:39:37 nanopi kernel: [860647.722375] [30692]   109 30692    50859      939      33       3      656             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722385] [30693]   109 30693    50719      841      33       3      567             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722393] [30694]   109 30694    50885      824      33       3      695             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722402] [30695]   109 30695    50739      875      33       3      556             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722411] [30696]   109 30696    50875      942      32       3      683             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722420] [28511]     0 28511   166639     1719     100       5     3123             0 node
Feb  1 05:39:37 nanopi kernel: [860647.722429] [28527]     0 28527   185131     2884     106       5     4060             0 node
Feb  1 05:39:37 nanopi kernel: [860647.722441] [31654]     0 31654   261216    51719     130       5       57             0 python
Feb  1 05:39:37 nanopi kernel: [860647.722452] [31832]     0 31832     1321       47       6       4       25             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.722461] [31835]     0 31835      465       24       5       4        0             0 modprobe
Feb  1 05:39:37 nanopi kernel: [860647.722471] [31838]   109 31838    50192       74      21       3      376             0 postgres
Feb  1 05:39:37 nanopi kernel: [860647.722480] [31839]     0 31839     1321       40       6       4       32             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.722488] [31840]     0 31840    10661       81      24       4      551             0 lpqd
Feb  1 05:39:37 nanopi kernel: [860647.722496] [31841]     0 31841     8659      145      18       3      395             0 winbindd
Feb  1 05:39:37 nanopi kernel: [860647.722504] [31842]     0 31842     1321       38       6       4       34             0 cron
Feb  1 05:39:37 nanopi kernel: [860647.722513] [31846]     0 31846    10657      104      22       4      528             0 smbd
Feb  1 05:39:37 nanopi kernel: [860647.722523] [31850]     0 31850     6870       71      16       4      381             0 nmbd
Feb  1 05:39:37 nanopi kernel: [860647.959894] oom_reaper: reaped process 30651 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:16kB
Feb  1 05:39:39 nanopi NetworkManager[642]: <info>  [1548992379.4651] dhcp4 (eth0):   address 10.0.0.5
Feb  1 05:39:39 nanopi NetworkManager[642]: <info>  [1548992379.4940] dhcp4 (eth0):   plen 24 (255.255.255.0)
Feb  1 05:39:39 nanopi NetworkManager[642]: <info>  [1548992379.4941] dhcp4 (eth0):   gateway 10.0.0.138
Feb  1 05:39:39 nanopi NetworkManager[642]: <info>  [1548992379.4941] dhcp4 (eth0):   server identifier 10.0.0.138
Feb  1 05:39:39 nanopi NetworkManager[642]: <info>  [1548992379.4942] dhcp4 (eth0):   lease time 3600
Feb  1 05:39:39 nanopi NetworkManager[642]: <info>  [1548992379.4943] dhcp4 (eth0):   nameserver '10.0.0.138'
Feb  1 05:39:39 nanopi NetworkManager[642]: <info>  [1548992379.4944] dhcp4 (eth0):   domain name 'mynet'
Feb  1 05:39:39 nanopi NetworkManager[642]: <info>  [1548992379.4946] dhcp4 (eth0): state changed bound -> bound

 

 

Edited by Igor
spoilers
Link to comment
Share on other sites

Right, the unclear point is that is it the Java process that is running out of memory or another process? I can't understand the mentioned pid's? 

What is more strange is that the same process is running on raspberrypi with exactly same params (the process is running with 256m memory) without memory problems. 

Is there any difference between memory management on armbian and raspbian? 

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines