edupv Posted February 6, 2018 Posted February 6, 2018 11 hours ago, lukasb said: Does this issue also affecting the idle clocks? Today I reinstalled my OrangePi PC2 with 5.40 Ubuntu Xenial Next desktop, and the cpu clocks rarely back to 408Mhz even cpu loads are low ( 0.05 at 1152Mhz ) Tried changing governor from ondemand to interactive, but it seems no effect current /etc/default/cpufrequtils: # WARNING: this file will be replaced on board support package (linux-root-...) upgrade ENABLE=true MIN_SPEED=240000 MAX_SPEED=1296000 GOVERNOR=interactive My previous OS is 5.27 Ubuntu Xenial, which works awesome for almost a year. Cpu clocks idle at 240Mhz (I lowered it myself from 408Mhz) You can adjust these 2 parameters, for example : echo 65 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold echo 50000 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate Then CPU clock will go to 408MHz when idle. 0 Quote
lukasb Posted February 6, 2018 Posted February 6, 2018 On 2/6/2018 at 12:24 AM, Igor said: DVFS part will receive some changes (AFAIK refactoring) since it is not working as it should yet. Nevertheless, a kernel is good enough for testing, perhaps even for use. Remember, it is a first "stable to be" build ... (not so big) problems are expected. alright, thank you so much! I wish I could donate to you for your amazing job maintaining the armbian - sadly i don't have any credit cards, but maybe I'll send you in bitcoins after I mine some. (is it okay?) 18 hours ago, edupv said: You can adjust these 2 parameters, for example : echo 65 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold echo 50000 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate Then CPU clock will go to 408MHz when idle. AFAIK, this isn't retained after reboot right? can I just simply pasting these commands on /etc/rc.local ? 0 Quote
edupv Posted February 7, 2018 Posted February 7, 2018 4 hours ago, lukasb said: AFAIK, this isn't retained after reboot right? can I just simply pasting these commands on /etc/rc.local ? Yes, you can run these commands in /etc/rc.local. Moreover, the above may not be the best values for up_threshold/sampling_rate. You can test different values yourself. 0 Quote
Loic Quertenmont Posted March 24, 2018 Posted March 24, 2018 "Hello, I am trying to setup my orange pi pc2 to play video on the tv (among other things), but I must say that I am having a real hard time. Today, I restarted from this clean image: Armbian_5.40_Orangepipc2_Ubuntu_xenial_next_4.14.15_desktop using startx+mpv I am able to play my video out of the box using software decoding... Unfortunately, if I use the full screen mode, the video is slow like hell. (without full screen it plays fine). So I suspect I need to use hardware decoding. startx /usr/bin/mpv --fs MyVideo.avi X.Org X Server 1.18.4 Release Date: 2016-07-19 X Protocol Version 11, Revision 0 Build Operating System: Linux 4.4.0-97-generic aarch64 Ubuntu Current Operating System: Linux orangepipc2 4.14.15-sunxi64 #30 SMP Tue Jan 30 17:40:12 CET 2018 aarch64 Kernel command line: root=UUID=6342048c-723e-476d-a785-c345ff4c7099 rootwait rootfstype=ext4 console=tty1 console=ttyS0,115200 panic=10 consoleblank=0 loglevel=1 ubootpart=3bd93a3c-01 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u cgroup_enable=memory swapaccount=1 Build Date: 13 October 2017 01:58:44PM xorg-server 2:1.18.4-0ubuntu0.7 (For technical support please see http://www.ubuntu.com/support) Current version of pixman: 0.33.6 Before reporting problems, check http://wiki.x.org to make sure that you have the latest version. Markers: (--) probed, (**) from config file, (==) default setting, (++) from command line, (!!) notice, (II) informational, (WW) warning, (EE) error, (NI) not implemented, (??) unknown. (==) Log file: "/var/log/Xorg.1.log", Time: Sat Mar 24 09:17:18 2018 (==) Using config directory: "/etc/X11/xorg.conf.d" (==) Using system config directory "/usr/share/X11/xorg.conf.d" gbm: failed to open any driver (search paths /usr/lib/aarch64-linux-gnu/dri:${ORIGIN}/dri:/usr/lib/dri) gbm: Last dlopen error: /usr/lib/dri/sun4i-drm_dri.so: cannot open shared object file: No such file or directory failed to load driver: sun4i-drm EGL_MESA_drm_image required. Playing: MyVideo.avi (+) Video --vid=1 (mpeg4) (+) Audio --aid=1 (mp3) AO: [alsa] 48000Hz stereo 2ch s16 VO: [x11] 720x404 yuv420p Should I worry about the "failed to load driver: sun4i-drm" error message ? I tried to install vdpau, but I couldn't find instructions that seem to be working fine for armbian for H5. See bellow the output of the lsmod command: >lsmod Module Size Used by fuse 98304 3 rpcsec_gss_krb5 32768 0 zram 28672 4 sun8i_codec_analog 28672 1 sun4i_codec 49152 3 snd_soc_hdmi_codec 16384 1 sun4i_i2s 20480 2 snd_soc_simple_card 16384 1 snd_soc_simple_card_utils 16384 1 snd_soc_simple_card snd_soc_core 155648 6 sun4i_codec,sun4i_i2s,sun8i_codec_analog,snd_soc_hdmi_codec,snd_soc_simple_card_utils,snd_soc_simple_card ir_lirc_codec 16384 0 snd_pcm_dmaengine 16384 1 snd_soc_core snd_pcm 106496 4 sun4i_i2s,snd_pcm_dmaengine,snd_soc_hdmi_codec,snd_soc_core lirc_dev 16384 1 ir_lirc_codec snd_timer 32768 1 snd_pcm sunxi_cir 16384 0 sunxi 20480 0 musb_hdrc 98304 1 sunxi sun4i_gpadc_iio 16384 0 iio_hwmon 16384 0 industrialio 65536 2 iio_hwmon,sun4i_gpadc_iio dw_hdmi_i2s_audio 16384 0 dw_hdmi_cec 16384 0 sy8106a_regulator 16384 1 sun8i_dw_hdmi 16384 0 dw_hdmi 28672 2 dw_hdmi_i2s_audio,sun8i_dw_hdmi cec 53248 2 dw_hdmi_cec,dw_hdmi rc_core 32768 6 ir_lirc_codec,cec,lirc_dev,sunxi_cir sun4i_tcon 28672 1 sun8i_dw_hdmi sun8i_mixer 16384 0 sun4i_drm 16384 2 If I anyway install vdpau and try to see vdpauinfo I get: vdpauinfo says: display: :0 screen: 0 Error creating VDPAU device: 25 So I am a bit disappointed and running out of ideas to find a solution to this problem. Did anybody have some advice (or even better, a recipe) for me ? Thanks in advance for your help, Loic 0 Quote
Igor Posted March 24, 2018 Posted March 24, 2018 48 minutes ago, Loic Quertenmont said: So I am a bit disappointed and running out of ideas to find a solution to this problem. This feature cost 30.000 EUR and will be implemented free of charge when it is delivered You can find detail info on that page. 2 Quote
Loic Quertenmont Posted March 24, 2018 Posted March 24, 2018 Good to lnow, but it doesn't sound like a short term solution 0 Quote
chwe Posted March 24, 2018 Posted March 24, 2018 2 hours ago, Loic Quertenmont said: Good to lnow, but it doesn't sound like a short term solution you need a short term solution? Buy a random TV box for 50-100$ with android... Okay, maybe not the cheapest you get, but I'm sure you'll find one which will perform well... This is a nice example how much efforts it needs to bring up such tasks.. And I don't think that they make much profit out of this campaign (just a guess, I don't know how they calculate the price). 0 Quote
Loic Quertenmont Posted March 24, 2018 Posted March 24, 2018 I should have by a rpi, I had a rpi1 doing video playing since 2011, until it died 2weeks ago. It's a pity that the opi pc2 can't do something that simple while it is way more powerful on paper. Thanks anyway 0 Quote
chwe Posted March 24, 2018 Posted March 24, 2018 welcome to reality.. H5 is for sure more powerful than a RPi for a bunch of applications.. But (at the moment) not for the whole video stuff. RPi focused relatively fast on multimedia stuff cause 90% of the RPis end as a TV-Box replacement (just another guess, I've no statistics and hand to prove that ).. Multimedia isn't an easy task. If you look for such a board do your paperwork first and look which boards perform well.. And since H5 is mainline only, you have to wait until bootlin is ready or you look if someone provides a good android for it (I never paid attention to android images for SBCs)... A decision should be made by use-case --> research which board performs well. 0 Quote
tkaiser Posted March 24, 2018 Posted March 24, 2018 23 minutes ago, chwe said: If you look for such a board do your paperwork first and look which boards perform well.. If you want a KODI box why not asking the KODI guys? 24 minutes ago, chwe said: since H5 is mainline only, you have to wait until bootlin is ready Bootlin is today working on Allwinner SoCs that are horribly outdated (A33 being the only expection somehow, the others are A10, A13 and A20 from 5 to 8 years ago). They're all ARMv7 and support for Allwinner SoCs that are just outdated (A64 and H5) will come later. According to their Kickstarter page at the end of Dec 2018 if they succeed. Welcome to the Allwinner reality. Get the hardware, be able to use only a shitty Android or crappy Android/Linux hybrids, wait for software support becoming mature (100% relying on community) and use the hardware as intended when it's already obsolete. 1 Quote
jernej Posted March 24, 2018 Posted March 24, 2018 15 minutes ago, tkaiser said: Bootlin is today working on Allwinner SoCs that are horribly outdated Fortunately, VPU driver is around 95% usable also for newer SoCs. 18 minutes ago, tkaiser said: wait for software support becoming mature (100% relying on community) and use the hardware as intended when it's already obsolete. I have a feeling that H6 support will become useful a bit quicker. 1 Quote
tkaiser Posted March 24, 2018 Posted March 24, 2018 22 minutes ago, jernej said: I have a feeling that H6 support will become useful a bit quicker. Yeah, but that's all just due to some of my personal superheroes doing such a great work (talking about especially icenowy and you right now). Thank you BTW But still it's a huge problem with Allwinner SoCs that we can't use what board or chip makers provide (at least not without wasting insane amounts of time to turn Allwinner's BSP crap into something useful like longsleep did two years ago). And then there's the insanely time consuming mainline upstreaming process... 1 Quote
chwe Posted March 24, 2018 Posted March 24, 2018 Why should they? Seems that outsourcing to the community works quite well... The stuff they stick together for their BSP seems to be 'good enough' for cheap tv boxes.. No matter what they deliver, there are experienced and patient people around which will do their work and I think hiring them would be expensive.. It's not a open-source friendly way and they could do it much better, but it works... As long as the SoC is 'exciting' enough (e.g. H6) mainlineing is fast, in case the SoC is 'boring' it's slow(er).. The H3 is still fast enough for a lot of stuff and cheap as hell and probably one of the best mainlined SoC in its price-range compared to performance... 0 Quote
hojnikb Posted March 24, 2018 Posted March 24, 2018 4 hours ago, Loic Quertenmont said: I should have by a rpi, I had a rpi1 doing video playing since 2011, until it died 2weeks ago. It's a pity that the opi pc2 can't do something that simple while it is way more powerful on paper. Thanks anyway You can install android on PC2 and it will do all the decoding you need. 0 Quote
tkaiser Posted March 24, 2018 Posted March 24, 2018 1 hour ago, chwe said: Why should they? Seems that outsourcing to the community works quite well... Which 'outsourcing'? Allwinner sells hardware. Cheap hardware for special markets. Android tablets back then, $something in between, now smart speakers, dashcams, retro gaming stuff, again tablets and TV boxes. Nintendo's NES Classic sold 2.3 million units in no time. Using a boring A33 SoC with technology from 5 years ago running a smelly 3.4.39 kernel from 5 years ago. Their customers (that's not us) do not care so why should Allwinner care so far? They enable device manufacturers to throw out cheap hardware with somewhat working software with ok-ish margins (their main market) or sometimes enable their customers like Nintendo to sell insanely overpriced/overhyped products where again no-one cares about kernel, software or anything else we would be interested in. For Allwinner there's still no 'Linux market'. Though things might change in the future. But unless there's an incentive to mainline their own hardware and submit code upstream (good luck given the BSP code quality) I doubt anything will change soon or at all. But of course I highly appreciate that they now contribute and react in a very responsive way. As @jernej pointed out in the meantime many requests will be answered positively (and I always chuckle seeing wink, the Allwinner guy, directly contributing to linux-sunxi wiki now -- I never thought this would happen) 2 Quote
guidol Posted March 31, 2018 Posted March 31, 2018 a few short questions: whats the difference between Armbian_5.40_Orangepipc2_Debian_stretch_next_4.14.15.7z and Armbian_5.37.171217_Orangepipc2_Debian_stretch_next_4.14.6.7z ? I did install Armbian_5.40_Orangepipc2_Debian_stretch_next_4.14.15.7z and got a armbian 5.40 stable. The Armbian_5.37.171217_Orangepipc2_Debian_stretch_next_4.14.6.7z is a nightly and wont be as stable as the 5.40, even when some nightlys - on other supported sbcs - via update are at 5.42? Its that the only difference? Is there any way to see at which actual armbian version-level a nightly is when there is in the downloads only a lower version-number? How often is a install-image for the nightly version normalyl generated? 0 Quote
Igor Posted April 21, 2018 Posted April 21, 2018 On 31. 3. 2018 at 10:42 AM, guidol said: Armbian_5.40_Orangepipc2_Debian_stretch_next_4.14.15.7z and Armbian_5.37.171217_Orangepipc2_Debian_stretch_next_4.14.6.7z ? The only real difference is that nightly images are (usually) not tested. They are just built from upstream, normally every day ... in reality building can be on hold for a week or more until things, which break the compilation, gets fixed or are removed. And most of the images don't get daily images rebuild, but only packages. This means any existing image, stable or nightly, can be upgraded to the latest nightly. (armbian-config -> switch to nightly) 171217 = 17. December, 2017 1 Quote
Андрей Исаев Posted May 23, 2018 Posted May 23, 2018 I have a problem - my Orange PC2 is connected to the TV via HDMI and when I turn off the TV, and then turn on the signal in the HDMI does not appear. A box connected through the SSH continues to work. Helps only REBOOT. 0 Quote
Igor Posted May 23, 2018 Posted May 23, 2018 9 minutes ago, Андрей Исаев said: I have a problem - my Orange PC2 is connected to the TV via HDMI and when I turn off the TV, and then turn on the signal in the HDMI does not appear. A box connected through the SSH continues to work. Helps only REBOOT. Video drivers are not yet fully developed and stable. We manage to put things together to have at least some HDMI output ... while full DRM capable and bug fixed video driver is coming to 4.17 - 4.18 http://linux-sunxi.org/Linux_mainlining_effort#Merged_into_4.17 Current support is as is. H5 images are experimental and we have pretty good functioning for such state. 0 Quote
whats up skip Posted June 25, 2018 Posted June 25, 2018 I noticed just be for my Orange Pi PC2 became unresponsive that the KswapD command was using the CPU heavily and had used a massive amount of the total CPU time. From the research I have done I am not entirely sure what the cause is. About two years ago there were similar sounding problems caused by a bug in kernel. Given I am running the latest non-stretch image, would I be safe to assume that this problem has been resolved? Other possibilites include problems with the configuration of the swap file, a lack of physical RAM memory or a program going rogue taking up too much RAM. Currently with the system running on a relatively light load it has the following conditions: Welcome to ARMBIAN 5.40 stable Ubuntu 16.04.4 LTS 4.14.48-sunxi64 System load: 0.05 0.14 0.17 Up time: 4:48 hours Local users: 4 Memory usage: 56 % of 994MB Swap usage: 88 % of 497Mb IP: 192.168.1.150 CPU temp: 40°C Usage of /: 87% of 29G top - 15:14:35 up 4:49, 5 users, load average: 0.28, 0.20, 0.19 Tasks: 180 total, 1 running, 140 sleeping, 0 stopped, 0 zombie %Cpu(s): 4.6 us, 1.3 sy, 0.2 ni, 93.6 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st KiB Mem : 1018416 total, 164388 free, 568404 used, 285624 buff/cache KiB Swap: 509200 total, 59004 free, 450196 used. 329512 avail Mem Any suggestions or ideas? 0 Quote
Seasalt Posted August 26, 2018 Posted August 26, 2018 On 5/23/2018 at 9:09 PM, Андрей Исаев said: I have a problem - my Orange PC2 is connected to the TV via HDMI and when I turn off the TV, and then turn on the signal in the HDMI does not appear. I have the same problem. It is really annoying. 0 Quote
Seasalt Posted August 26, 2018 Posted August 26, 2018 I have just update my OrangePi PC2 and now I can play 720p video really well. It appears to use 80 - 90 CPU to do it . So I am wondering is it playing the HEVC 720p file in software or is it using Hardware VPU decoding. I am also totally impressed now that Armbian has fixed the screen resolution option box. Huge improvement. This is a really big set of improvements to the Armbian PC2. Well done. 1 Quote
svts Posted December 9, 2018 Posted December 9, 2018 Hello everybody! I've bought recently OrangePi PC2 to develop a small data gathering machine but got stuck with some difficulties with Armbian. Board is OrangePi PC2 v 1.1. The problem is random crashes (kernel oops and kernel panics then sometimes). What I have done: made SD card with Armbian "next" 5.65 distro (kernel 4.14.78-sunxi64). No changes, no overlays etc. What happens: the system crashes randomly. It happens never during initial kernel boot but it starts happening after systemd load. Sometimes (like 1-2 of 10) system cannot boot at all without crash. When system boots okay I use a script to test: Spoiler #!/bin/sh while true; do date for I in `seq 1 4`; do timeout $I cat /dev/zero > /dev/null & cat /var/log/syslog | grep 123 > /dev/null & done wait sleep 1 done The script runs well for a while (like 5-15 mins) and then crash appears in system console and dmesg. Sometimes system hangs immediately but sometimes it works then. I check then the state of the system using ambianmonitor -m (temp and cpu freq) and also top (f -> P to watch CPU id of the process) When there's no direct oops crash in system log, some local crashes can occur, i.e. it can be segfault of some tool (ssh, top, sudo etc). What I have already tried: updated kernel to dev-version (4.19) but got the same random crashes. Downgraded to 4.10.0 and got the same things. Tried to upgrade u-boot to dev version, tried to downgrade u-boot. Uploaded u-boot to NOR flash and used USB for armbian (no SD card at all) - random crashes appear. "Official" distro (Debian 8, kernel 3.10) works with no issue. But it's too old to use with 1-wire, I2C and other sensors. Then I spent hours to check PC and LR values, also checked processes which was involved to crash issue but nothing common. I tried to downclock CPU (to 480 MHz min and max) - no luck. Checked voltages (1.1 Vcore, 1.2Vdram) - everything is okay. Changed PSU - no effect. Changed the board to another OPI PC2 one - no effect, same crashes. Then I tried to run the system on a single core by kernel parameter maxcpus=1. And everything was FINE. No crashes for 24h. Then I made CPU1 online (echo 1 > /sys/bus/cpu/devices/cpu1/online) and got crash after like 5 minutes of running the script (non stop). Then I tried to play with kernel parameter isolcpus=1-3. I saw then some interrupts (cat /proc/interrupts) got served by all CPU cores, but all software were running at CPU0 core. Then no crash for like 12h. Then I started the same script on CPU2 using taskset -c 4 ./test.sh, saw it running on CPU2 (using top) and then got a crash again in 15 minutes. Then I tried isolcpus=0,2-3, saw systemd and all services running at CPU1, run test.sh for like 4 hours on CPU1 (as well) with no problem, then run it on CPU2 taskset -c 4 ./test.sh and got crash in like 10 minutes. UPDATED: nope, that was wrong. Crashes still appear but very rarely. Also I tried memtester on both boards I have - everything is fine. Conclusion: it seems it's not hardware issue, something prevents apps work normally when there's high I/O load when multiple cores are used. And it seems it's not kernel itself as kernel processes work and serve interrupts during isolcpus parameter used. Could it be an issue with systemd? There's no option in repo to update it and check. Please help me to figure out what's happening. Thank you so much. UPDATED2: u-boot causes kernel crashes of OPI PC2 Some infos is in my post below UPDATED3: new u-boot set DRAM PLL to 624MHz which seems to high for some boards (especially "cheapest" from aliexpress). So I set DRAM PLL to 600MHz using u-boot mw.l command and I have no crash anymore. Some infos is in my post after Some crash oops logs are below. Spoiler [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Set console scheme. [ OK ] Created slice system-getty.slice. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenBSD Secure Shell server. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Ubuntu 18.04.1 LTS orangepipc2 ttyS0 orangepipc2 login: [ 38.155840] Unable to handle kernel NULL pointer dereference at virtual address 00000077 [ 38.163950] pgd = ffff000008bef000 [ 38.167350] [00000077] *pgd=000000007effe003, *pud=000000007effd003, *pmd=0000000000000000 [ 38.175639] Internal error: Oops: 96000044 [#1] SMP [ 38.180518] Modules linked in: tun sun8i_codec_analog sun4i_codec snd_soc_core snd_pcm_dmaengine snd_pcm snd_timer sch_fq_codel [ 38.192001] CPU: 1 PID: 7 Comm: rcu_sched Not tainted 4.10.0-sun50iw2 #7 [ 38.198692] Hardware name: Xunlong Orange Pi PC 2 (DT) [ 38.203824] task: ffff80003d864b00 task.stack: ffff80003d898000 [ 38.209741] PC is at rcu_gp_kthread+0x3c8/0x7c8 [ 38.214263] LR is at rcu_gp_kthread+0x394/0x7c8 [ 38.218792] pc : [<ffff0000080ff3c8>] lr : [<ffff0000080ff394>] pstate: 20000145 [ 38.226175] sp : ffff80003d89bdd0 [ 38.229483] x29: 0000000000000000 x28: 0000000000000000 [ 38.234786] x27: 0000000000000000 x26: ffff000008b32000 [ 38.240092] x25: 0000000000000003 x24: ffff80003d89be50 [ 38.245407] x23: 0000000000000001 x22: 0000000000000004 [ 38.250713] x21: ffff000008b16000 x20: ffff000008b32b20 [ 38.256015] x19: ffff000008b32800 x18: ffff000008ba7a00 [ 38.261330] x17: 0000ffff9d099010 x16: ffff0000080ad170 [ 38.266639] x15: fffffffffffffc68 x14: 000000000000024d [ 38.271947] x13: ffff000008a3c000 x12: dead000000000100 [ 38.277252] x11: ffff7e0000000000 x10: 0000000000000860 [ 38.282564] x9 : ffff80003d89bcb0 x8 : ffff80003d8653c0 [ 38.287871] x7 : ffff80003ef99c80 x6 : 00000008e2435045 [ 38.293174] x5 : 0000000000000000 x4 : 0000000000000000 [ 38.298476] x3 : 0000000000000078 x2 : 0000000000000077 [ 38.303778] x1 : 00000000ffff004b x0 : 00000000000005f2 [ 38.309088] [ 38.310581] Process rcu_sched (pid: 7, stack limit = 0xffff80003d898000) [ 38.317269] Stack: (0xffff80003d89bdd0 to 0xffff80003d89c000) [ 38.323015] bdc0: ffff80003d89be60 ffff0000080c5a28 [ 38.330847] bde0: ffff80003d831300 ffff80003d831200 ffff000008ba5860 ffff80003d864b00 [ 38.338669] be00: ffff0000089ca578 ffff000008b32800 ffff0000080ff000 ffff80003d831338 [ 38.346484] be20: ffff80003d857ce0 0000000000000000 ffff000008b37d0d ffff80003d864b00 [ 38.354301] be40: 00ff80003d831300 ffff80003d864b00 ffff80003d89be50 ffff80003d89be50 [ 38.362118] be60: 0000000000000000 ffff0000080826c0 ffff0000080c5938 ffff80003d831200 [ 38.369934] be80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.377751] bea0: 0000000000000000 0000000000000000 0000000000000000 ffffffffffffffff [ 38.385566] bec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.393381] bee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.401196] bf00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.409011] bf20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.416826] bf40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.424641] bf60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.432457] bf80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.440272] bfa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.448087] bfc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000 [ 38.455901] bfe0: 0000000000000000 0000000000000000 ffffffffffffffff ffffffffffffffff [ 38.463715] Call trace: [ 38.466156] Exception stack(0xffff80003d89bc00 to 0xffff80003d89bd30) [ 38.472585] bc00: ffff000008b32800 0001000000000000 ffff80003d89bdd0 ffff0000080ff3c8 [ 38.480400] bc20: ffff80003d89bc30 ffff0000080df768 ffff80003d89bcd0 ffff000008811d7c [ 38.488214] bc40: ffff80003ef9f900 ffff000008b18000 ffff000008af1000 ffff80003d864b00 [ 38.496031] bc60: ffff80003d89bc90 ffff0000080db368 ffff80003d864b80 ffff000008b18000 [ 38.503845] bc80: 0000000000000001 0000000000000000 ffff80003d89bcd0 ffff000008811e30 [ 38.511660] bca0: 00000000000005f2 00000000ffff004b 0000000000000077 0000000000000078 [ 38.519475] bcc0: 0000000000000000 0000000000000000 00000008e2435045 ffff80003ef99c80 [ 38.527291] bce0: ffff80003d8653c0 ffff80003d89bcb0 0000000000000860 ffff7e0000000000 [ 38.535106] bd00: dead000000000100 ffff000008a3c000 000000000000024d fffffffffffffc68 [ 38.542920] bd20: ffff0000080ad170 0000ffff9d099010 [ 38.547795] [<ffff0000080ff3c8>] rcu_gp_kthread+0x3c8/0x7c8 [ 38.553359] Code: 9101e3a3 f94542a1 9101dfa2 91000400 (3901dfbf) [ 38.559577] ---[ end trace f49e754a16b6761c ]--- [ 43.200819] fuse init (API version 7.26) Sun Dec 9 08:21:06 MSK 2018 Sun Dec 9 08:21:11 MSK 2018 Sun Dec 9 08:21:16 MSK 2018 [ 59.156035] INFO: rcu_sched detected stalls on CPUs/tasks: [ 59.161537] (detected by 2, t=5253 jiffies, g=1185, c=1184, q=6363) [ 59.167896] All QSes seen, last rcu_sched kthread activity 5254 (4294907087-4294901833), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 59.179697] cat R running task 0 1638 1637 0x00000000 [ 59.186736] Call trace: [ 59.189189] [<ffff000008087a20>] dump_backtrace+0x0/0x220 [ 59.194579] [<ffff000008087c54>] show_stack+0x14/0x20 [ 59.199624] [<ffff0000080d0c84>] sched_show_task+0x10c/0x170 [ 59.205279] [<ffff0000081002e4>] rcu_check_callbacks+0x854/0x858 [ 59.211277] [<ffff000008102c74>] update_process_times+0x2c/0x58 [ 59.217189] [<ffff00000811205c>] tick_sched_handle.isra.7+0x24/0x70 [ 59.223444] [<ffff0000081120ec>] tick_sched_timer+0x44/0x90 [ 59.229029] [<ffff0000081037a8>] __hrtimer_run_queues+0xf0/0x178 [ 59.235034] [<ffff000008103ae8>] hrtimer_interrupt+0xa0/0x220 [ 59.240774] [<ffff00000869b970>] arch_timer_handler_phys+0x30/0x40 [ 59.246946] [<ffff0000080f61b8>] handle_percpu_devid_irq+0x78/0x130 [ 59.253204] [<ffff0000080f0eac>] generic_handle_irq+0x24/0x38 [ 59.258943] [<ffff0000080f152c>] __handle_domain_irq+0x5c/0xb8 [ 59.264768] [<ffff000008080d24>] gic_handle_irq+0x54/0xa8 [ 59.270157] Exception stack(0xffff80003c423be0 to 0xffff80003c423d10) [ 59.276586] 3be0: 0000ffffa7556d40 00000000000002c0 0000000000001000 0000000000000001 [ 59.284418] 3c00: 0000000000020000 0001000000000000 0000000000000020 0000ffffa753d000 [ 59.292240] 3c20: 000000000000003f 0000ffffa78742a0 0000000000000840 0000000000000000 [ 59.300057] 3c40: 0000000000000001 000000000000270f 0000000000000002 0000000000000000 [ 59.307874] 3c60: ffff0000081cb7b0 0000ffffa77bba78 0000ffffa784ca70 0000000000001000 [ 59.315689] 3c80: 0000000000019000 ffff80003c423de0 ffff80003c8a0000 ffff80003c423eb8 [ 59.323503] 3ca0: 0000000000000015 0000000000000123 000000000000003f ffff80003c423e18 [ 59.331320] 3cc0: 0000000000001000 ffff80003c423d10 ffff0000084c7fa0 ffff80003c423d10 [ 59.339136] 3ce0: ffff0000084b2388 0000000020000145 ffff80003c0f5780 ffff80003c8a0000 [ 59.346951] 3d00: 0001000000000000 ffff000008812500 [ 59.351825] [<ffff000008081ff4>] el1_irq+0xb4/0x140 [ 59.356700] [<ffff0000084b2388>] __clear_user+0x18/0x90 [ 59.361919] [<ffff00000856882c>] read_iter_zero+0x4c/0xc0 [ 59.367312] [<ffff0000081c9064>] __vfs_read+0xac/0xf8 [ 59.372354] [<ffff0000081ca2fc>] vfs_read+0x84/0x140 [ 59.377310] [<ffff0000081cb7f4>] SyS_read+0x44/0xa0 [ 59.382179] [<ffff000008082730>] el0_svc_naked+0x24/0x28 [ 59.387484] rcu_sched kthread starved for 5309 jiffies! g1185 c1184 f0x2 RCU_GP_DOING_FQS(4) ->state=0x40 Spoiler [ 100.848748] Unable to handle kernel NULL pointer dereference at virtual address 000000d0 [ 100.856945] pgd = ffff80003c641000 [ 100.860346] [000000d0] *pgd=0000000000000000 [ 100.864667] Internal error: Oops: 96000044 [#1] SMP [ 100.869541] Modules linked in: fuse tun sun4i_codec sun8i_codec_analog snd_soc_core snd_pcm_dmaengine snd_pcm snd_timer sch_fq_codel [ 100.881456] CPU: 2 PID: 1439 Comm: test.sh Not tainted 4.10.0-sun50iw2 #7 [ 100.888232] Hardware name: Xunlong Orange Pi PC 2 (DT) [ 100.893362] task: ffff80003ca0d780 task.stack: ffff80003c614000 [ 100.899281] PC is at copy_process.isra.5.part.6+0x150/0x1470 [ 100.904934] LR is at copy_process.isra.5.part.6+0x134/0x1470 [ 100.910584] pc : [<ffff0000080a6bd8>] lr : [<ffff0000080a6bbc>] pstate: 40000145 [ 100.917965] sp : ffff80003c617d50 [ 100.921273] x29: 0000000000000000 x28: 0000000001200011 [ 100.926579] x27: ffff80003c880000 x26: ffff80003c810c80 [ 100.931883] x25: 0000000000000000 x24: 0000000000000000 [ 100.937186] x23: 0000000000000000 x22: 0000000000000000 [ 100.942490] x21: 0000000000000000 x20: 0000ffff8d4cac80 [ 100.947793] x19: 0000000000000000 x18: 0000000000000000 [ 100.953098] x17: 0000ffff8d3f27c0 x16: ffff0000080a8370 [ 100.958402] x15: 0000000000000000 x14: 0000000000000000 [ 100.963705] x13: 0000000000000000 x12: 0000000000000000 [ 100.969009] x11: 0000000000000000 x10: 0000000000000000 [ 100.974312] x9 : 0000000000000000 x8 : 0000000000000000 [ 100.979614] x7 : 0000000000000000 x6 : ffff80003c811900 [ 100.984916] x5 : 0000000000000000 x4 : 0000000000000000 [ 100.990221] x3 : ffff000008af5739 x2 : 0000000000000001 [ 100.995524] x1 : ffff80003c810c80 x0 : 0000000000000001 [ 101.000826] [ 101.002315] Process test.sh (pid: 1439, stack limit = 0xffff80003c614000) [ 101.009093] Stack: (0xffff80003c617d50 to 0xffff80003c618000) [ 101.014831] 7d40: ffff80003c617e40 ffff0000080a8058 [ 101.022648] 7d60: 0000000001200011 0000000000000000 ffffffffffffffff 0000000000000000 [ 101.030465] 7d80: 0000000040000000 0000000000000015 0000000000000123 00000000000000dc [ 101.038281] 7da0: ffff000008822000 ffff80003ca0d780 ffff000008828178 000000009200004f [ 101.046098] 7dc0: 0000aaaab2f5c908 ffff80003c617ec0 0000000000000000 0000000000000024 [ 101.053914] 7de0: 000000009200004f 0000aaaab2f5c908 0000aaaac955f8b0 ffff80003ca0d780 [ 101.061730] 7e00: 0000000001200011 0000000000000002 0000000000000000 ffff00000808236c [ 101.069547] 7e20: 0000000000000000 ffff000008ba4000 ffffffffffffffff 0000aaaab2f39ac4 [ 101.077363] 7e40: ffff80003c617eb0 ffff0000080a8388 0000000000000000 00008000364c5000 [ 101.085180] 7e60: ffffffffffffffff 0000ffff8d3f28a4 0000000040000000 0000000000000015 [ 101.092998] 7e80: 0000000000000123 00000000000000dc ffff80003c617eb0 ffff0000081c75d0 [ 101.100817] 7ea0: 0000000000000000 00008000364c5000 0000000000000000 ffff000008082730 [ 101.108634] 7ec0: 0000000001200011 0000000000000000 0000000000000000 0000000000000000 [ 101.116451] 7ee0: 0000ffff8d4cac80 0000ffff8d4cabb0 0000aaaac9561120 000000000003e9a3 [ 101.124269] 7f00: 00000000000000dc 0000aaaab2f5c000 0101010101010101 0000000000000000 [ 101.132085] 7f20: 0101010101010101 000000000000270f 0000000000000001 0000000000000020 [ 101.139901] 7f40: 0000aaaab2f5baf0 0000ffff8d3f27c0 0000ffff8d4a3a70 0000000000000000 [ 101.147716] 7f60: 0000000000000000 0000ffff8d4a2000 0000aaaac955f8b0 0000ffff8d4caaa0 [ 101.155532] 7f80: 0000ffff8d4cb2a0 0000000000000000 0000aaaab2f5c0d8 0000aaaac955f8b0 [ 101.163348] 7fa0: 0000ffffc0c044a0 0000ffffc0c04390 0000aaaab2f3f480 0000ffffc0c04390 [ 101.171165] 7fc0: 0000ffff8d3f28a4 0000000040000000 0000000001200011 00000000000000dc [ 101.178981] 7fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 101.186796] Call trace: [ 101.189239] Exception stack(0xffff80003c617b80 to 0xffff80003c617cb0) [ 101.195672] 7b80: 0000000000000000 0001000000000000 ffff80003c617d50 ffff0000080a6bd8 [ 101.203488] 7ba0: ffff80003c617c20 ffff000008184af8 ffff7e00017000c0 ffff7e000074fb80 [ 101.211305] 7bc0: ffff80003c617d08 0000000000000180 000000003ca07840 ffff000008b9b600 [ 101.219122] 7be0: 0000000000000000 0000000000000140 ffff800000000001 ffff000008184b6c [ 101.226937] 7c00: ffff80003c617d50 ffff0000080a6ba4 0000000000000000 0000ffff8d4cac80 [ 101.234754] 7c20: 0000000000000001 ffff80003c810c80 0000000000000001 ffff000008af5739 [ 101.242569] 7c40: 0000000000000000 0000000000000000 ffff80003c811900 0000000000000000 [ 101.250385] 7c60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 101.258201] 7c80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 101.266016] 7ca0: ffff0000080a8370 0000ffff8d3f27c0 [ 101.270890] [<ffff0000080a6bd8>] copy_process.isra.5.part.6+0x150/0x1470 [ 101.277583] Code: 350019a0 2a0203e0 aa1a03e1 f903435f (b900d3a2) [ 101.283806] ---[ end trace 2625318557f33a56 ]--- Spoiler [ 3.319363] sun8i-emac 1c30000.ethernet: PHY regulator found [ 3.325754] input: r_gpio_keys as /devices/platform/r_gpio_keys/input/input0 [ 3.333011] of_cfs_init [ 3.335489] of_cfs_init: OK [ 3.338387] gmac-3v3: disabling [ 3.341539] ALSA device list: [ 3.344505] No soundcards found. [ 3.348283] Freeing unused kernel memory: 448K Loading, please wait... starting version 237 [ 3.417575] systemd-udevd[203]: unhandled level 0 translation fault (11) at 0x00000038, esr 0x92000004 [ 3.426960] pgd = ffff80003be82000 [ 3.430373] [00000038] *pgd=0000000000000000 [ 3.434643] [ 3.436140] CPU: 1 PID: 203 Comm: systemd-udevd Not tainted 4.10.0-sun50iw2 #7 [ 3.443356] Hardware name: Xunlong Orange Pi PC 2 (DT) [ 3.448490] task: ffff80003c19f080 task.stack: ffff80003bec4000 [ 3.454409] PC is at 0xaaaad4fce1c8 [ 3.457896] LR is at 0xaaaad4fce1b0 [ 3.461382] pc : [<0000aaaad4fce1c8>] lr : [<0000aaaad4fce1b0>] pstate: 60000000 [ 3.468768] sp : 0000ffffeba07510 [ 3.472080] x29: 0000000000000000 x28: 000000000000000e [ 3.477389] x27: 0000aaaaeca4a140 x26: 00000000000002ea [ 3.482697] x25: 0000aaaaeca6a180 x24: 0000000000000054 [ 3.488006] x23: 0000aaaaeca6a4a0 x22: 0000aaaaeca66b80 [ 3.493340] x21: 0000aaaaeca6b2d0 x20: 000000000000007c [ 3.498655] x19: 0000aaaad5036000 x18: 0000ffff9d690a70 [ 3.503963] x17: 0000ffff9d60e5f8 x16: 0000aaaad5036a58 [ 3.509269] x15: 0000000000000000 x14: 0000000000000002 [ 3.514574] x13: 000000000000270f x12: 0000000000000001 [ 3.519880] x11: 0000000000000000 x10: 0000000000000001 [ 3.525186] x9 : 0000ffff9d6b9c90 x8 : 00000000000000d3 [ 3.530492] x7 : 7f7fffffff7f7fff x6 : fefffefe9c6809cf Spoiler [ OK ] Started LSB: Load kernel modules needed to enable cpufreq scaling. Starting LSB: set CPUFreq kernel parameters... [ 11.379399] Unable to handle kernel NULL pointer dereference at virtual address 00000038 [ 11.379403] Unable to handle kernel NULL pointer dereference at virtual address 00000038 [ 11.379406] Unable to handle kernel NULL pointer dereference at virtual address 00000038 [ 11.379409] Unable to handle kernel NULL pointer dereference at virtual address 00000038 [ 11.379417] Mem abort info: [ 11.379418] Mem abort info: [ 11.379419] Mem abort info: [ 11.379422] Exception class = DABT (current EL), IL = 32 bits [ 11.379424] Exception class = DABT (current EL), IL = 32 bits [ 11.379426] Exception class = DABT (current EL), IL = 32 bits [ 11.379428] SET = 0, FnV = 0 [ 11.379429] SET = 0, FnV = 0 [ 11.379431] SET = 0, FnV = 0 [ 11.379432] EA = 0, S1PTW = 0 [ 11.379433] EA = 0, S1PTW = 0 [ 11.379434] EA = 0, S1PTW = 0 [ 11.379435] Data abort info: [ 11.379436] Data abort info: [ 11.379437] Data abort info: [ 11.379438] ISV = 0, ISS = 0x00000006 [ 11.379439] ISV = 0, ISS = 0x00000004 [ 11.379440] ISV = 0, ISS = 0x00000004 [ 11.379442] CM = 0, WnR = 0 [ 11.379443] CM = 0, WnR = 0 [ 11.379444] CM = 0, WnR = 0 [ 11.379448] user pgtable: 4k pages, 48-bit VAs, pgd = ffff80003508c000 [ 11.379451] user pgtable: 4k pages, 48-bit VAs, pgd = ffff8000351b8000 [ 11.379454] user pgtable: 4k pages, 48-bit VAs, pgd = ffff800034572000 [ 11.379456] [0000000000000038] *pgd=000000007568b003 [ 11.379458] [0000000000000038] *pgd=0000000000000000 [ 11.379460] [0000000000000038] *pgd=0000000000000000 [ 11.379461] , *pud=0000000075699003 [ 11.379464] , *pmd=0000000000000000 [ 11.379467] Internal error: Oops: 96000004 [#1] SMP [ 11.379469] Modules linked in: sun8i_codec_analog sun4i_i2s ir_lirc_codec lirc_dev snd_soc_hdmi_codec sun4i_codec snd_soc_simple_card snd_soc_simple_card_utils snd_soc_core sunxi_cir sun4i_gpadc_iio snd_pcm_dmaengine snd_pcm snd_timer iio_hwmon industrialio zram sch_fq_codel dw_hdmi_cec dw_hdmi_i2s_audio sy8106a_regulator sun8i_dw_hdmi dw_hdmi cec rc_core uas sunxi musb_hdrc sun4i_tcon sun8i_mixer sun4i_drm [ 11.379524] CPU: 3 PID: 724 Comm: armbian-hardwar Not tainted 4.14.78-sunxi64 #416 [ 11.379526] Hardware name: Xunlong Orange Pi PC 2 (DT) [ 11.379528] task: ffff80003539db00 task.stack: ffff00000a5d8000 [ 11.379541] PC is at cpu_load_update_active+0x4c/0xa8 [ 11.379546] LR is at cpu_load_update_active+0x9c/0xa8 [ 11.379549] pc : [<ffff0000080e3064>] lr : [<ffff0000080e30b4>] pstate: 800001c5 [ 11.379550] sp : ffff00000801bd90 [ 11.379551] x29: 0000000000000000 x28: ffff80003539db00 [ 11.379556] x27: ffff800037fb6dc0 x26: ffff800037fb6dc0 [ 11.379560] x25: ffff800037fb6dd0 x24: 00000002a600d616 [ 11.379563] x23: ffff80003539db00 x22: 0000000000000003 [ 11.379567] x21: 0000000000000404 x20: ffff800037fbb000 [ 11.379571] x19: ffff000008c68d88 x18: 0000ffffbc77fa70 [ 11.379575] x17: 0000ffffbc6ceb60 x16: ffff0000081ff200 [ 11.379579] x15: 0000000000000406 x14: 0000000000000000 [ 11.379582] x13: ffff000008998918 x12: 0000000000000001 [ 11.379586] x11: ffff000008998920 x10: 0000000000000406 [ 11.379590] x9 : 0000000000000004 x8 : ffff800037fbb030 [ 11.379594] x7 : 0000000000000020 x6 : 0000000000000404 [ 11.379597] x5 : 00000002a6438871 x4 : 000000000000000f [ 11.379601] x3 : 000000001dcd6500 x2 : 0000000016349bec [ 11.379604] x1 : 00000002900eec85 x0 : ffff800037fbb000 [ 11.379610] Process armbian-hardwar (pid: 724, stack limit = 0xffff00000a5d8000) [ 11.379612] Call trace: [ 11.379616] Exception stack(0xffff00000801bc50 to 0xffff00000801bd90) [ 11.379620] bc40: ffff800037fbb000 00000002900eec85 [ 11.379624] bc60: 0000000016349bec 000000001dcd6500 000000000000000f 00000002a6438871 [ 11.379628] bc80: 0000000000000404 0000000000000020 ffff800037fbb030 0000000000000004 [ 11.379633] bca0: 0000000000000406 ffff000008998920 0000000000000001 ffff000008998918 [ 11.379637] bcc0: 0000000000000000 0000000000000406 ffff0000081ff200 0000ffffbc6ceb60 [ 11.379641] bce0: 0000ffffbc77fa70 ffff000008c68d88 ffff800037fbb000 0000000000000404 [ 11.379645] bd00: 0000000000000003 ffff80003539db00 00000002a600d616 ffff800037fb6dd0 [ 11.379649] bd20: ffff800037fb6dc0 ffff800037fb6dc0 ffff80003539db00 0000000000000000 [ 11.379653] bd40: ffff0000080e30b4 ffff00000801bd90 ffff0000080e3064 00000000800001c5 [ 11.379657] bd60: ffff800037fbb000 ffff80003539db80 0000ffffffffffff ffff0000080e1a84 [ 11.379660] bd80: 0000000000000000 ffff0000080e3064 [ 11.379666] [<ffff0000080e3064>] cpu_load_update_active+0x4c/0xa8 [ 11.379672] Code: f9454000 eb020002 54000141 91362273 (f9401fa1) [ 11.379676] ---[ end trace a5890dadb94a76a7 ]--- [ 11.379679] Kernel panic - not syncing: Fatal exception in interrupt [ 11.379684] SMP: stopping secondary CPUs [ 11.812830] Mem abort info: [ 11.815620] Exception class = DABT (current EL), IL = 32 bits [ 11.821531] SET = 0, FnV = 0 [ 11.824581] EA = 0, S1PTW = 0 [ 11.827716] Data abort info: [ 11.830592] ISV = 0, ISS = 0x00000006 [ 11.834422] CM = 0, WnR = 0 [ 11.837386] user pgtable: 4k pages, 48-bit VAs, pgd = ffff8000350b9000 [ 11.843904] [0000000000000038] *pgd=0000000075795003, *pud=00000000757a4003, *pmd=0000000000000000 f[ 12.422302] SMP: failed to stop secondary CPUs 0-3 [ 12.427089] Kernel Offset: disabled [ 12.430574] CPU features: 0x0802004 [ 12.434058] Memory Limit: none [ 12.437110] Rebooting in 10 seconds.. [ 22.441187] SMP: stopping secondary CPUs [ 23.487718] SMP: failed to stop secondary CPUs 0-3 0 Quote
Menion Posted December 9, 2018 Posted December 9, 2018 16 minutes ago, svts said: Hello everybody! I've bought recently OrangePi PC2 to develop a small data gathering machine but got stuck with some difficulties with Armbian. Board is OrangePi PC2 v 1.1. The problem is random crashes (kernel oops and kernel panics then sometimes). What I have done: made SD card with Armbian "next" 5.65 distro (kernel 4.14.78-sunxi64). No changes, no overlays etc. What happens: the system crashes randomly. It happens never during initial kernel boot but it starts happening after systemd load. Sometimes (like 1-2 of 10) system cannot boot at all without crash. When system boots okay I use a script to test: Reveal hidden contents #!/bin/sh while true; do date for I in `seq 1 4`; do timeout $I cat /dev/zero > /dev/null & cat /var/log/syslog | grep 123 > /dev/null & done wait sleep 1 done The script runs well for a while (like 5-15 mins) and then crash appears in system console and dmesg. Sometimes system hangs immediately but sometimes it works then. I check then the state of the system using ambianmonitor -m (temp and cpu freq) and also top (f -> P to watch CPU id of the process) When there's no direct oops crash in system log, some local crashes can occur, i.e. it can be segfault of some tool (ssh, top, sudo etc). What I have already tried: updated kernel to dev-version (4.19) but got the same random crashes. Downgraded to 4.10.0 and got the same things. Tried to upgrade u-boot to dev version, tried to downgrade u-boot. Uploaded u-boot to NOR flash and used USB for armbian (no SD card at all) - random crashes appear. "Official" distro (Debian 8, kernel 3.10) works with no issue. But it's too old to use with 1-wire, I2C and other sensors. Then I spent hours to check PC and LR values, also checked processes which was involved to crash issue but nothing common. I tried to downclock CPU (to 480 MHz min and max) - no luck. Checked voltages (1.1 Vcore, 1.2Vdram) - everything is okay. Changed PSU - no effect. Changed the board to another OPI PC2 one - no effect, same crashes. Then I tried to run the system on a single core by kernel parameter maxcpus=1. And everything was FINE. No crashes for 24h. Then I made CPU1 online (echo 1 > /sys/bus/cpu/devices/cpu1/online) and got crash after like 5 minutes of running the script (non stop). Then I tried to play with kernel parameter isolcpus=1-3. I saw then some interrupts (cat /proc/interrupts) got served by all CPU cores, but all software were running at CPU0 core. Then no crash for like 12h. Then I started the same script on CPU2 using taskset -c 4 ./test.sh, saw it running on CPU2 (using top) and then got a crash again in 15 minutes. Then I tried isolcpus=0,2-3, saw systemd and all services running at CPU1, run test.sh for like 4 hours on CPU1 (as well) with no problem, then run it on CPU2 taskset -c 4 ./test.sh and got crash in like 10 minutes. Also I tried memtester on both boards I have - everything is fine. Conclusion: it seems it's not hardware issue, something prevents apps work normally when there's high I/O load when multiple cores are used. And it seems it's not kernel itself as kernel processes work and serve interrupts during isolcpus parameter used. Could it be an issue with systemd? There's no option in repo to update it and check. Please help me to figure out what's happening. Thank you so much. Some crash oops logs are below. Reveal hidden contents [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Set console scheme. [ OK ] Created slice system-getty.slice. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenBSD Secure Shell server. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Ubuntu 18.04.1 LTS orangepipc2 ttyS0 orangepipc2 login: [ 38.155840] Unable to handle kernel NULL pointer dereference at virtual address 00000077 [ 38.163950] pgd = ffff000008bef000 [ 38.167350] [00000077] *pgd=000000007effe003, *pud=000000007effd003, *pmd=0000000000000000 [ 38.175639] Internal error: Oops: 96000044 [#1] SMP [ 38.180518] Modules linked in: tun sun8i_codec_analog sun4i_codec snd_soc_core snd_pcm_dmaengine snd_pcm snd_timer sch_fq_codel [ 38.192001] CPU: 1 PID: 7 Comm: rcu_sched Not tainted 4.10.0-sun50iw2 #7 [ 38.198692] Hardware name: Xunlong Orange Pi PC 2 (DT) [ 38.203824] task: ffff80003d864b00 task.stack: ffff80003d898000 [ 38.209741] PC is at rcu_gp_kthread+0x3c8/0x7c8 [ 38.214263] LR is at rcu_gp_kthread+0x394/0x7c8 [ 38.218792] pc : [<ffff0000080ff3c8>] lr : [<ffff0000080ff394>] pstate: 20000145 [ 38.226175] sp : ffff80003d89bdd0 [ 38.229483] x29: 0000000000000000 x28: 0000000000000000 [ 38.234786] x27: 0000000000000000 x26: ffff000008b32000 [ 38.240092] x25: 0000000000000003 x24: ffff80003d89be50 [ 38.245407] x23: 0000000000000001 x22: 0000000000000004 [ 38.250713] x21: ffff000008b16000 x20: ffff000008b32b20 [ 38.256015] x19: ffff000008b32800 x18: ffff000008ba7a00 [ 38.261330] x17: 0000ffff9d099010 x16: ffff0000080ad170 [ 38.266639] x15: fffffffffffffc68 x14: 000000000000024d [ 38.271947] x13: ffff000008a3c000 x12: dead000000000100 [ 38.277252] x11: ffff7e0000000000 x10: 0000000000000860 [ 38.282564] x9 : ffff80003d89bcb0 x8 : ffff80003d8653c0 [ 38.287871] x7 : ffff80003ef99c80 x6 : 00000008e2435045 [ 38.293174] x5 : 0000000000000000 x4 : 0000000000000000 [ 38.298476] x3 : 0000000000000078 x2 : 0000000000000077 [ 38.303778] x1 : 00000000ffff004b x0 : 00000000000005f2 [ 38.309088] [ 38.310581] Process rcu_sched (pid: 7, stack limit = 0xffff80003d898000) [ 38.317269] Stack: (0xffff80003d89bdd0 to 0xffff80003d89c000) [ 38.323015] bdc0: ffff80003d89be60 ffff0000080c5a28 [ 38.330847] bde0: ffff80003d831300 ffff80003d831200 ffff000008ba5860 ffff80003d864b00 [ 38.338669] be00: ffff0000089ca578 ffff000008b32800 ffff0000080ff000 ffff80003d831338 [ 38.346484] be20: ffff80003d857ce0 0000000000000000 ffff000008b37d0d ffff80003d864b00 [ 38.354301] be40: 00ff80003d831300 ffff80003d864b00 ffff80003d89be50 ffff80003d89be50 [ 38.362118] be60: 0000000000000000 ffff0000080826c0 ffff0000080c5938 ffff80003d831200 [ 38.369934] be80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.377751] bea0: 0000000000000000 0000000000000000 0000000000000000 ffffffffffffffff [ 38.385566] bec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.393381] bee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.401196] bf00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.409011] bf20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.416826] bf40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.424641] bf60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.432457] bf80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.440272] bfa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 38.448087] bfc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000 [ 38.455901] bfe0: 0000000000000000 0000000000000000 ffffffffffffffff ffffffffffffffff [ 38.463715] Call trace: [ 38.466156] Exception stack(0xffff80003d89bc00 to 0xffff80003d89bd30) [ 38.472585] bc00: ffff000008b32800 0001000000000000 ffff80003d89bdd0 ffff0000080ff3c8 [ 38.480400] bc20: ffff80003d89bc30 ffff0000080df768 ffff80003d89bcd0 ffff000008811d7c [ 38.488214] bc40: ffff80003ef9f900 ffff000008b18000 ffff000008af1000 ffff80003d864b00 [ 38.496031] bc60: ffff80003d89bc90 ffff0000080db368 ffff80003d864b80 ffff000008b18000 [ 38.503845] bc80: 0000000000000001 0000000000000000 ffff80003d89bcd0 ffff000008811e30 [ 38.511660] bca0: 00000000000005f2 00000000ffff004b 0000000000000077 0000000000000078 [ 38.519475] bcc0: 0000000000000000 0000000000000000 00000008e2435045 ffff80003ef99c80 [ 38.527291] bce0: ffff80003d8653c0 ffff80003d89bcb0 0000000000000860 ffff7e0000000000 [ 38.535106] bd00: dead000000000100 ffff000008a3c000 000000000000024d fffffffffffffc68 [ 38.542920] bd20: ffff0000080ad170 0000ffff9d099010 [ 38.547795] [<ffff0000080ff3c8>] rcu_gp_kthread+0x3c8/0x7c8 [ 38.553359] Code: 9101e3a3 f94542a1 9101dfa2 91000400 (3901dfbf) [ 38.559577] ---[ end trace f49e754a16b6761c ]--- [ 43.200819] fuse init (API version 7.26) Sun Dec 9 08:21:06 MSK 2018 Sun Dec 9 08:21:11 MSK 2018 Sun Dec 9 08:21:16 MSK 2018 [ 59.156035] INFO: rcu_sched detected stalls on CPUs/tasks: [ 59.161537] (detected by 2, t=5253 jiffies, g=1185, c=1184, q=6363) [ 59.167896] All QSes seen, last rcu_sched kthread activity 5254 (4294907087-4294901833), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 59.179697] cat R running task 0 1638 1637 0x00000000 [ 59.186736] Call trace: [ 59.189189] [<ffff000008087a20>] dump_backtrace+0x0/0x220 [ 59.194579] [<ffff000008087c54>] show_stack+0x14/0x20 [ 59.199624] [<ffff0000080d0c84>] sched_show_task+0x10c/0x170 [ 59.205279] [<ffff0000081002e4>] rcu_check_callbacks+0x854/0x858 [ 59.211277] [<ffff000008102c74>] update_process_times+0x2c/0x58 [ 59.217189] [<ffff00000811205c>] tick_sched_handle.isra.7+0x24/0x70 [ 59.223444] [<ffff0000081120ec>] tick_sched_timer+0x44/0x90 [ 59.229029] [<ffff0000081037a8>] __hrtimer_run_queues+0xf0/0x178 [ 59.235034] [<ffff000008103ae8>] hrtimer_interrupt+0xa0/0x220 [ 59.240774] [<ffff00000869b970>] arch_timer_handler_phys+0x30/0x40 [ 59.246946] [<ffff0000080f61b8>] handle_percpu_devid_irq+0x78/0x130 [ 59.253204] [<ffff0000080f0eac>] generic_handle_irq+0x24/0x38 [ 59.258943] [<ffff0000080f152c>] __handle_domain_irq+0x5c/0xb8 [ 59.264768] [<ffff000008080d24>] gic_handle_irq+0x54/0xa8 [ 59.270157] Exception stack(0xffff80003c423be0 to 0xffff80003c423d10) [ 59.276586] 3be0: 0000ffffa7556d40 00000000000002c0 0000000000001000 0000000000000001 [ 59.284418] 3c00: 0000000000020000 0001000000000000 0000000000000020 0000ffffa753d000 [ 59.292240] 3c20: 000000000000003f 0000ffffa78742a0 0000000000000840 0000000000000000 [ 59.300057] 3c40: 0000000000000001 000000000000270f 0000000000000002 0000000000000000 [ 59.307874] 3c60: ffff0000081cb7b0 0000ffffa77bba78 0000ffffa784ca70 0000000000001000 [ 59.315689] 3c80: 0000000000019000 ffff80003c423de0 ffff80003c8a0000 ffff80003c423eb8 [ 59.323503] 3ca0: 0000000000000015 0000000000000123 000000000000003f ffff80003c423e18 [ 59.331320] 3cc0: 0000000000001000 ffff80003c423d10 ffff0000084c7fa0 ffff80003c423d10 [ 59.339136] 3ce0: ffff0000084b2388 0000000020000145 ffff80003c0f5780 ffff80003c8a0000 [ 59.346951] 3d00: 0001000000000000 ffff000008812500 [ 59.351825] [<ffff000008081ff4>] el1_irq+0xb4/0x140 [ 59.356700] [<ffff0000084b2388>] __clear_user+0x18/0x90 [ 59.361919] [<ffff00000856882c>] read_iter_zero+0x4c/0xc0 [ 59.367312] [<ffff0000081c9064>] __vfs_read+0xac/0xf8 [ 59.372354] [<ffff0000081ca2fc>] vfs_read+0x84/0x140 [ 59.377310] [<ffff0000081cb7f4>] SyS_read+0x44/0xa0 [ 59.382179] [<ffff000008082730>] el0_svc_naked+0x24/0x28 [ 59.387484] rcu_sched kthread starved for 5309 jiffies! g1185 c1184 f0x2 RCU_GP_DOING_FQS(4) ->state=0x40 Reveal hidden contents [ 100.848748] Unable to handle kernel NULL pointer dereference at virtual address 000000d0 [ 100.856945] pgd = ffff80003c641000 [ 100.860346] [000000d0] *pgd=0000000000000000 [ 100.864667] Internal error: Oops: 96000044 [#1] SMP [ 100.869541] Modules linked in: fuse tun sun4i_codec sun8i_codec_analog snd_soc_core snd_pcm_dmaengine snd_pcm snd_timer sch_fq_codel [ 100.881456] CPU: 2 PID: 1439 Comm: test.sh Not tainted 4.10.0-sun50iw2 #7 [ 100.888232] Hardware name: Xunlong Orange Pi PC 2 (DT) [ 100.893362] task: ffff80003ca0d780 task.stack: ffff80003c614000 [ 100.899281] PC is at copy_process.isra.5.part.6+0x150/0x1470 [ 100.904934] LR is at copy_process.isra.5.part.6+0x134/0x1470 [ 100.910584] pc : [<ffff0000080a6bd8>] lr : [<ffff0000080a6bbc>] pstate: 40000145 [ 100.917965] sp : ffff80003c617d50 [ 100.921273] x29: 0000000000000000 x28: 0000000001200011 [ 100.926579] x27: ffff80003c880000 x26: ffff80003c810c80 [ 100.931883] x25: 0000000000000000 x24: 0000000000000000 [ 100.937186] x23: 0000000000000000 x22: 0000000000000000 [ 100.942490] x21: 0000000000000000 x20: 0000ffff8d4cac80 [ 100.947793] x19: 0000000000000000 x18: 0000000000000000 [ 100.953098] x17: 0000ffff8d3f27c0 x16: ffff0000080a8370 [ 100.958402] x15: 0000000000000000 x14: 0000000000000000 [ 100.963705] x13: 0000000000000000 x12: 0000000000000000 [ 100.969009] x11: 0000000000000000 x10: 0000000000000000 [ 100.974312] x9 : 0000000000000000 x8 : 0000000000000000 [ 100.979614] x7 : 0000000000000000 x6 : ffff80003c811900 [ 100.984916] x5 : 0000000000000000 x4 : 0000000000000000 [ 100.990221] x3 : ffff000008af5739 x2 : 0000000000000001 [ 100.995524] x1 : ffff80003c810c80 x0 : 0000000000000001 [ 101.000826] [ 101.002315] Process test.sh (pid: 1439, stack limit = 0xffff80003c614000) [ 101.009093] Stack: (0xffff80003c617d50 to 0xffff80003c618000) [ 101.014831] 7d40: ffff80003c617e40 ffff0000080a8058 [ 101.022648] 7d60: 0000000001200011 0000000000000000 ffffffffffffffff 0000000000000000 [ 101.030465] 7d80: 0000000040000000 0000000000000015 0000000000000123 00000000000000dc [ 101.038281] 7da0: ffff000008822000 ffff80003ca0d780 ffff000008828178 000000009200004f [ 101.046098] 7dc0: 0000aaaab2f5c908 ffff80003c617ec0 0000000000000000 0000000000000024 [ 101.053914] 7de0: 000000009200004f 0000aaaab2f5c908 0000aaaac955f8b0 ffff80003ca0d780 [ 101.061730] 7e00: 0000000001200011 0000000000000002 0000000000000000 ffff00000808236c [ 101.069547] 7e20: 0000000000000000 ffff000008ba4000 ffffffffffffffff 0000aaaab2f39ac4 [ 101.077363] 7e40: ffff80003c617eb0 ffff0000080a8388 0000000000000000 00008000364c5000 [ 101.085180] 7e60: ffffffffffffffff 0000ffff8d3f28a4 0000000040000000 0000000000000015 [ 101.092998] 7e80: 0000000000000123 00000000000000dc ffff80003c617eb0 ffff0000081c75d0 [ 101.100817] 7ea0: 0000000000000000 00008000364c5000 0000000000000000 ffff000008082730 [ 101.108634] 7ec0: 0000000001200011 0000000000000000 0000000000000000 0000000000000000 [ 101.116451] 7ee0: 0000ffff8d4cac80 0000ffff8d4cabb0 0000aaaac9561120 000000000003e9a3 [ 101.124269] 7f00: 00000000000000dc 0000aaaab2f5c000 0101010101010101 0000000000000000 [ 101.132085] 7f20: 0101010101010101 000000000000270f 0000000000000001 0000000000000020 [ 101.139901] 7f40: 0000aaaab2f5baf0 0000ffff8d3f27c0 0000ffff8d4a3a70 0000000000000000 [ 101.147716] 7f60: 0000000000000000 0000ffff8d4a2000 0000aaaac955f8b0 0000ffff8d4caaa0 [ 101.155532] 7f80: 0000ffff8d4cb2a0 0000000000000000 0000aaaab2f5c0d8 0000aaaac955f8b0 [ 101.163348] 7fa0: 0000ffffc0c044a0 0000ffffc0c04390 0000aaaab2f3f480 0000ffffc0c04390 [ 101.171165] 7fc0: 0000ffff8d3f28a4 0000000040000000 0000000001200011 00000000000000dc [ 101.178981] 7fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 101.186796] Call trace: [ 101.189239] Exception stack(0xffff80003c617b80 to 0xffff80003c617cb0) [ 101.195672] 7b80: 0000000000000000 0001000000000000 ffff80003c617d50 ffff0000080a6bd8 [ 101.203488] 7ba0: ffff80003c617c20 ffff000008184af8 ffff7e00017000c0 ffff7e000074fb80 [ 101.211305] 7bc0: ffff80003c617d08 0000000000000180 000000003ca07840 ffff000008b9b600 [ 101.219122] 7be0: 0000000000000000 0000000000000140 ffff800000000001 ffff000008184b6c [ 101.226937] 7c00: ffff80003c617d50 ffff0000080a6ba4 0000000000000000 0000ffff8d4cac80 [ 101.234754] 7c20: 0000000000000001 ffff80003c810c80 0000000000000001 ffff000008af5739 [ 101.242569] 7c40: 0000000000000000 0000000000000000 ffff80003c811900 0000000000000000 [ 101.250385] 7c60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 101.258201] 7c80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 101.266016] 7ca0: ffff0000080a8370 0000ffff8d3f27c0 [ 101.270890] [<ffff0000080a6bd8>] copy_process.isra.5.part.6+0x150/0x1470 [ 101.277583] Code: 350019a0 2a0203e0 aa1a03e1 f903435f (b900d3a2) [ 101.283806] ---[ end trace 2625318557f33a56 ]--- Reveal hidden contents [ 3.319363] sun8i-emac 1c30000.ethernet: PHY regulator found [ 3.325754] input: r_gpio_keys as /devices/platform/r_gpio_keys/input/input0 [ 3.333011] of_cfs_init [ 3.335489] of_cfs_init: OK [ 3.338387] gmac-3v3: disabling [ 3.341539] ALSA device list: [ 3.344505] No soundcards found. [ 3.348283] Freeing unused kernel memory: 448K Loading, please wait... starting version 237 [ 3.417575] systemd-udevd[203]: unhandled level 0 translation fault (11) at 0x00000038, esr 0x92000004 [ 3.426960] pgd = ffff80003be82000 [ 3.430373] [00000038] *pgd=0000000000000000 [ 3.434643] [ 3.436140] CPU: 1 PID: 203 Comm: systemd-udevd Not tainted 4.10.0-sun50iw2 #7 [ 3.443356] Hardware name: Xunlong Orange Pi PC 2 (DT) [ 3.448490] task: ffff80003c19f080 task.stack: ffff80003bec4000 [ 3.454409] PC is at 0xaaaad4fce1c8 [ 3.457896] LR is at 0xaaaad4fce1b0 [ 3.461382] pc : [<0000aaaad4fce1c8>] lr : [<0000aaaad4fce1b0>] pstate: 60000000 [ 3.468768] sp : 0000ffffeba07510 [ 3.472080] x29: 0000000000000000 x28: 000000000000000e [ 3.477389] x27: 0000aaaaeca4a140 x26: 00000000000002ea [ 3.482697] x25: 0000aaaaeca6a180 x24: 0000000000000054 [ 3.488006] x23: 0000aaaaeca6a4a0 x22: 0000aaaaeca66b80 [ 3.493340] x21: 0000aaaaeca6b2d0 x20: 000000000000007c [ 3.498655] x19: 0000aaaad5036000 x18: 0000ffff9d690a70 [ 3.503963] x17: 0000ffff9d60e5f8 x16: 0000aaaad5036a58 [ 3.509269] x15: 0000000000000000 x14: 0000000000000002 [ 3.514574] x13: 000000000000270f x12: 0000000000000001 [ 3.519880] x11: 0000000000000000 x10: 0000000000000001 [ 3.525186] x9 : 0000ffff9d6b9c90 x8 : 00000000000000d3 [ 3.530492] x7 : 7f7fffffff7f7fff x6 : fefffefe9c6809cf Reveal hidden contents [ OK ] Started LSB: Load kernel modules needed to enable cpufreq scaling. Starting LSB: set CPUFreq kernel parameters... [ 11.379399] Unable to handle kernel NULL pointer dereference at virtual address 00000038 [ 11.379403] Unable to handle kernel NULL pointer dereference at virtual address 00000038 [ 11.379406] Unable to handle kernel NULL pointer dereference at virtual address 00000038 [ 11.379409] Unable to handle kernel NULL pointer dereference at virtual address 00000038 [ 11.379417] Mem abort info: [ 11.379418] Mem abort info: [ 11.379419] Mem abort info: [ 11.379422] Exception class = DABT (current EL), IL = 32 bits [ 11.379424] Exception class = DABT (current EL), IL = 32 bits [ 11.379426] Exception class = DABT (current EL), IL = 32 bits [ 11.379428] SET = 0, FnV = 0 [ 11.379429] SET = 0, FnV = 0 [ 11.379431] SET = 0, FnV = 0 [ 11.379432] EA = 0, S1PTW = 0 [ 11.379433] EA = 0, S1PTW = 0 [ 11.379434] EA = 0, S1PTW = 0 [ 11.379435] Data abort info: [ 11.379436] Data abort info: [ 11.379437] Data abort info: [ 11.379438] ISV = 0, ISS = 0x00000006 [ 11.379439] ISV = 0, ISS = 0x00000004 [ 11.379440] ISV = 0, ISS = 0x00000004 [ 11.379442] CM = 0, WnR = 0 [ 11.379443] CM = 0, WnR = 0 [ 11.379444] CM = 0, WnR = 0 [ 11.379448] user pgtable: 4k pages, 48-bit VAs, pgd = ffff80003508c000 [ 11.379451] user pgtable: 4k pages, 48-bit VAs, pgd = ffff8000351b8000 [ 11.379454] user pgtable: 4k pages, 48-bit VAs, pgd = ffff800034572000 [ 11.379456] [0000000000000038] *pgd=000000007568b003 [ 11.379458] [0000000000000038] *pgd=0000000000000000 [ 11.379460] [0000000000000038] *pgd=0000000000000000 [ 11.379461] , *pud=0000000075699003 [ 11.379464] , *pmd=0000000000000000 [ 11.379467] Internal error: Oops: 96000004 [#1] SMP [ 11.379469] Modules linked in: sun8i_codec_analog sun4i_i2s ir_lirc_codec lirc_dev snd_soc_hdmi_codec sun4i_codec snd_soc_simple_card snd_soc_simple_card_utils snd_soc_core sunxi_cir sun4i_gpadc_iio snd_pcm_dmaengine snd_pcm snd_timer iio_hwmon industrialio zram sch_fq_codel dw_hdmi_cec dw_hdmi_i2s_audio sy8106a_regulator sun8i_dw_hdmi dw_hdmi cec rc_core uas sunxi musb_hdrc sun4i_tcon sun8i_mixer sun4i_drm [ 11.379524] CPU: 3 PID: 724 Comm: armbian-hardwar Not tainted 4.14.78-sunxi64 #416 [ 11.379526] Hardware name: Xunlong Orange Pi PC 2 (DT) [ 11.379528] task: ffff80003539db00 task.stack: ffff00000a5d8000 [ 11.379541] PC is at cpu_load_update_active+0x4c/0xa8 [ 11.379546] LR is at cpu_load_update_active+0x9c/0xa8 [ 11.379549] pc : [<ffff0000080e3064>] lr : [<ffff0000080e30b4>] pstate: 800001c5 [ 11.379550] sp : ffff00000801bd90 [ 11.379551] x29: 0000000000000000 x28: ffff80003539db00 [ 11.379556] x27: ffff800037fb6dc0 x26: ffff800037fb6dc0 [ 11.379560] x25: ffff800037fb6dd0 x24: 00000002a600d616 [ 11.379563] x23: ffff80003539db00 x22: 0000000000000003 [ 11.379567] x21: 0000000000000404 x20: ffff800037fbb000 [ 11.379571] x19: ffff000008c68d88 x18: 0000ffffbc77fa70 [ 11.379575] x17: 0000ffffbc6ceb60 x16: ffff0000081ff200 [ 11.379579] x15: 0000000000000406 x14: 0000000000000000 [ 11.379582] x13: ffff000008998918 x12: 0000000000000001 [ 11.379586] x11: ffff000008998920 x10: 0000000000000406 [ 11.379590] x9 : 0000000000000004 x8 : ffff800037fbb030 [ 11.379594] x7 : 0000000000000020 x6 : 0000000000000404 [ 11.379597] x5 : 00000002a6438871 x4 : 000000000000000f [ 11.379601] x3 : 000000001dcd6500 x2 : 0000000016349bec [ 11.379604] x1 : 00000002900eec85 x0 : ffff800037fbb000 [ 11.379610] Process armbian-hardwar (pid: 724, stack limit = 0xffff00000a5d8000) [ 11.379612] Call trace: [ 11.379616] Exception stack(0xffff00000801bc50 to 0xffff00000801bd90) [ 11.379620] bc40: ffff800037fbb000 00000002900eec85 [ 11.379624] bc60: 0000000016349bec 000000001dcd6500 000000000000000f 00000002a6438871 [ 11.379628] bc80: 0000000000000404 0000000000000020 ffff800037fbb030 0000000000000004 [ 11.379633] bca0: 0000000000000406 ffff000008998920 0000000000000001 ffff000008998918 [ 11.379637] bcc0: 0000000000000000 0000000000000406 ffff0000081ff200 0000ffffbc6ceb60 [ 11.379641] bce0: 0000ffffbc77fa70 ffff000008c68d88 ffff800037fbb000 0000000000000404 [ 11.379645] bd00: 0000000000000003 ffff80003539db00 00000002a600d616 ffff800037fb6dd0 [ 11.379649] bd20: ffff800037fb6dc0 ffff800037fb6dc0 ffff80003539db00 0000000000000000 [ 11.379653] bd40: ffff0000080e30b4 ffff00000801bd90 ffff0000080e3064 00000000800001c5 [ 11.379657] bd60: ffff800037fbb000 ffff80003539db80 0000ffffffffffff ffff0000080e1a84 [ 11.379660] bd80: 0000000000000000 ffff0000080e3064 [ 11.379666] [<ffff0000080e3064>] cpu_load_update_active+0x4c/0xa8 [ 11.379672] Code: f9454000 eb020002 54000141 91362273 (f9401fa1) [ 11.379676] ---[ end trace a5890dadb94a76a7 ]--- [ 11.379679] Kernel panic - not syncing: Fatal exception in interrupt [ 11.379684] SMP: stopping secondary CPUs [ 11.812830] Mem abort info: [ 11.815620] Exception class = DABT (current EL), IL = 32 bits [ 11.821531] SET = 0, FnV = 0 [ 11.824581] EA = 0, S1PTW = 0 [ 11.827716] Data abort info: [ 11.830592] ISV = 0, ISS = 0x00000006 [ 11.834422] CM = 0, WnR = 0 [ 11.837386] user pgtable: 4k pages, 48-bit VAs, pgd = ffff8000350b9000 [ 11.843904] [0000000000000038] *pgd=0000000075795003, *pud=00000000757a4003, *pmd=0000000000000000 f[ 12.422302] SMP: failed to stop secondary CPUs 0-3 [ 12.427089] Kernel Offset: disabled [ 12.430574] CPU features: 0x0802004 [ 12.434058] Memory Limit: none [ 12.437110] Rebooting in 10 seconds.. [ 22.441187] SMP: stopping secondary CPUs [ 23.487718] SMP: failed to stop secondary CPUs 0-3 This looks like to be a flaw in kernel internals. Please report it in linux-sunxi mainlining effort newsgroup 0 Quote
Seasalt Posted December 10, 2018 Posted December 10, 2018 5 hours ago, svts said: Sometimes system hangs immediately but sometimes it works then. What is the 5volt voltage at the USB port. (do not put a mulimeter probe in to a usb port.) Cut a old usb cable measure the ends.. If the USB voltage is less than say 5v maybe you need to tweak your power supply voltage upwards a bit. Be very careful. 0 Quote
svts Posted December 10, 2018 Posted December 10, 2018 Update: after the second day of non-stop testing I figured out, that crashes still appear but rarely. Nor isolcpus, nor numcpus really help. But all crashes are like this Unable to handle kernel NULL pointer dereference at virtual address 00000038 The difference is only the virtual address value which is always less then 0x00000100. 4 hours ago, Seasalt said: What is the 5volt voltage at the USB port. (do not put a mulimeter probe in to a usb port.) Cut a old usb cable measure the ends.. If the USB voltage is less than say 5v maybe you need to tweak your power supply voltage upwards a bit. Be very careful. I tried to check voltages eveywhere. Sure no short-circuit and no probes into USB ports directly So I use two different PSUs, and there's 5.11v and 5.02v. I even tried oscilliscope to check, but everything is okay, no fluctuations. (ps: Is that really important as I don't use USB devices during the test? I saw the circuit diagram and supply voltage isn't used directly anywhere but USB, all other voltages and their oscilloscope shapes seem to be okay) 9 hours ago, Menion said: This looks like to be a flaw in kernel internals. Please report it in linux-sunxi mainlining effort newsgroup Yep, wrote to them also. Thank you for the tip 0 Quote
svts Posted December 13, 2018 Posted December 13, 2018 UPDATE2: u-boot causes kernel crashes of OPI PC2 After some more days of debugging I figured out that problem isn't in the kernel itself and nor in system services but in u-boot. I changed different kernels, tried to black list modules and disable services using servicectl and update-rc.d but nothing really helped. Then I boot with init=/bin/bash kernel parameter and tested like that, so crashes appeared. After that I found and old "dietpi" distro based on 4.10.0 and armbian 5.27 (imho) and checked, and no crashes. Then I updated its kernel to 4.14.78 and checked again - no crashes. 4.18.8 - no crashes again. I tried to boot it with init=/bin/bash and had success as well. After that I transferred Image and uInitrd from this distro to armbian and checked again with init=/bin/bash. And I got a crash in like 5 minutes. So there was no difference... but u-boot. I tried also downgrade armbian's u-boot to 2017.05 (downloaded from apt.armbian.com) but it didn't really help. Then I extracted u-boot 2017.01-rc1 from the "dietpi" distro and wrote it to armbian SD card. And no more crashes. Then I updated it to kernel version of 4.19.6-sunxi64 (as I did at the first time too) and it works stable. It works with 100% load for like 15 hours at max. freq of 1296MHz (I limited it with cpufrequtils). root@orangepipc2:~# uptime 06:45:48 up 15:27, 4 users, load average: 7.44, 7.62, 7.59 root@orangepipc2:~# uname -a Linux orangepipc2 4.19.6-sunxi64 #5.67.181207 SMP Fri Dec 7 10:54:20 CET 2018 aarch64 aarch64 aarch64 GNU/Linux So the question is what's happened to u-boot which causes those crashes. I suppose it can be a thing which enables some SoC unused pins or probably some interrupts. I compared boot.scr files and found nothing really hw-related, which could cause some difficulties. An "Official" support for H5 SoC was made since u-boot 2017.05 but I have 2017.01-rc1 which is probably patched at the moment. And it works perfect. U-Boot 2017.01-rc1-g5df570f-dirty (May 02 2017 - 12:04:46 +0100) Allwinner Technology CPU: Allwinner H5 (SUN50I) Model: OrangePi PC 2 DRAM: 1 GiB MMC: SUNXI SD/MMC: 0 Does anybody have an idea why new u-boot could cause some hangs? Had anybody made some stress-test of Armbian running OPI PC2 boards? If yes so which HW version, which u-boot? I have two boards (bougth separately) of HW version 1.1 and it works with really old u-boot only. So let it be my "Bug report" 0 Quote
Menion Posted December 13, 2018 Posted December 13, 2018 Hi It is better if you report everything to U-Boot mailing list: https://lists.denx.de/listinfo/u-boot 0 Quote
znoxx Posted December 13, 2018 Posted December 13, 2018 Here is my Uboot version info and I don't see _any_ troubles. My armbian "cluster" works 24x7 with full load. zno@node0:~$ uptime 13:49:53 up 1 day, 7:44, 1 user, load average: 4.21, 4.07, 4.03 zno@node0:~$ dpkg -l | grep Uboot ii linux-u-boot-orangepipc2-next 5.40 arm64 Uboot loader 2017.11 Only one lame assumption. May be "other" version of u-boot clocks memory to lower values and it is working stable ? Or may be your hw is broken somehow ? For example, I have one OpiPC2 which has dead Ethernet. By the way, my boards are running at freq=1104000 (watts saved amount is bigger, than performance drop). 0 Quote
svts Posted December 13, 2018 Posted December 13, 2018 Thank you for your information, I'll try u-boot 2017.11 tomorrow. I compared DDR_PLL registers set by old and new u-boot, new one set a bit lower value of 624MHz and old one set 684MHz. Once I noticed that I fixed it directly in u-boot (wm.l) but got a crash again. Old uboot doesn't work with network interface and USB properly (as it's really old one) but it doesn't affect linux kernel which works perfect. I have bought these OPI PC2 separately from different suppliers so I'm not sure it could be some kind of hardware bug. I have hardware version of 1.1 of both boards. Which is yours? And during the tests I tried different CPU frequences from 120/480MHz upto 1104/1284MHz. Same distro works fine with older uboot and crashes in like five minutes with new one. 7z test shows almost the same value in both cases. I also compared GPU_PLL registers and SDC_PLL, they are a bit different but when I change values I have a crash again. So there must be something else. 0 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.