4 4
arno-ppsspp

Date changed to 1978,System crash,Mysql run 195% of CPU

Recommended Posts

Armbianmonitor:

Hello,Respected Developers:
I'm an Orange Pi user.In the process of using the armbian, I encountered serious problems.
My Control Board is Orange Pi PC Plus.Armbian is "ARMBIAN 5.59 stable Ubuntu 18.04.1 LTS 4.14.65-sunxi"
I installed jdk and mysql.And run a JAVA program.I set a time zone for Asia-Chongqing,The system language is Chinese,I use three serial ports.
When running for a period of time,There will be bugs.About one to three days.
The specific phenomena are as follows:
1.Run "date",the time is 1978.
2.Mysql occupy cpu 195%, occupy mem 27%,systemd occupy cpu 34%,Another systemd occupy cpu 14%
,systemd-jo+ occupy cpu 39%.
3.eth0‘s ip  disappear,Unable to connect remotely from the network for ssh.I only use serial port to connect.
4.I can't use the reboot command to restart,I input "reboot",It didn't respond.

 

Time automatically changed to 1978.


I hope you can help me solve this problem,Thanks very much.
 

 

Share this post


Link to post
Share on other sites

hi, I found this topic by searching for "armbian 1978". out of the sudden my OrangePi Plus 2E was set to 1978 and the system was unusuable. couldn't ssh into it and even at the local console the screen/monitor didn't show anything but black screen.

So I was forced to hard-reboot it. 

 

The only thing I noticed was this email from cron daemon:

/etc/cron.hourly/fake-hwclock:
Time travel detected!
fake-hwclock release date is in the future: 2016-04-15 00:00:00
Current system time: 1978-05-27 23:17:02
To force the saved system clock backwards in time anyway, use "force"

 

Is this a known bug?

 

 

Share this post


Link to post
Share on other sites
On 5/14/2019 at 12:13 PM, StefanH. said:

hi, I found this topic by searching for "armbian 1978". out of the sudden my OrangePi Plus 2E was set to 1978 and the system was unusuable. couldn't ssh into it and even at the local console the screen/monitor didn't show anything but black screen.

So I was forced to hard-reboot it. 

 

The only thing I noticed was this email from cron daemon:

/etc/cron.hourly/fake-hwclock:
Time travel detected!
fake-hwclock release date is in the future: 2016-04-15 00:00:00
Current system time: 1978-05-27 23:17:02
To force the saved system clock backwards in time anyway, use "force"

 

Is this a known bug?

 

 

Same strange behaviour here. Same machine (OrangePi Plus 2E)! Looking for a solution/explication now.

Share this post


Link to post
Share on other sites

Hello, 

I found this thread by searching for "orange pi 1978" with Google because I have the same problem here with 2 identical Orange Pi Plus 2E systems.

One both systems I installed  ARMBIAN 5.73 stable - 4.19.20-sunxi and the problem is reproducible but it takes around 2 weeks till it happens.

Both systems running in the same network and do exactly the same (the 2nd is a backup-system) but the problem does not occur on both system at the same time.

 

In the moment, were the problem occur you have limited access to the system. SSH does not work, shutdown or reboot does not work and today I tried to sync the the systemtime with the RTC but it does not work. RTC has the correct time and system date was 7th May 1978.

The only solution is to cut the power and connect it again. I use a WLAN-power-socket to do this remote. 

 

Do you have any idea what I can try? I thought an external RTC modul could help and sync the time in case of the 1978-problem happens but today I learned the RTC is not the problem (sudo hwclock -r gives the right time) and it's not possible to set the date with sudo hwclock -s when it happens. 
 

Thanks for your help!

Share this post


Link to post
Share on other sites

Same issue here :(

 

could anyone find a solution? Did try some stuff with ntp services, but doesn't seem to be the problem

Share this post


Link to post
Share on other sites

I have totally same problem..

I have connected 2 card reader to orange PI PC and maximum in 48-72h I can expirience the same problem. Can not access PI over ssh, black screen on HDMI out.

I tried (about 2months ago) to compile myself the actual version of armbian with tutorial and script what I found here on this forum, but I expirience the same issue.
My question is: Is this happening same with a newest kernel and relase?
Is thre any fix for this?

Share this post


Link to post
Share on other sites

the longest uptime without the 1978 issue was 45 days and the shortest uptime (ended yesterday) was 2 days. I tried so many things but could not solve it yet. The last change (yesterday) was to limit the cpu clockdown. Now it goes down to 800 and up to 1200Mhz in "conservative" mode. 

 

I ordered a second OrangePi +2E to make more tests. The second +2E shows the same problem and in the next step I will install the latest armbian image with the newest kernel. 

 

Share this post


Link to post
Share on other sites
3 hours ago, Marc_in_the_dark said:

the longest uptime without the 1978 issue was 45 days and the shortest uptime (ended yesterday) was 2 days. I tried so many things but could not solve it yet. The last change (yesterday) was to limit the cpu clockdown. Now it goes down to 800 and up to 1200Mhz in "conservative" mode. 

 

I ordered a second OrangePi +2E to make more tests. The second +2E shows the same problem and in the next step I will install the latest armbian image with the newest kernel. 

 

Thx for replay..but I need to disappointed you. I have even the same board what u ordered OrangePI +2E, and I was with same logic that maybe it is related with the OrangePI PC model, but no. It happening the same, totally. So you can jump over that step and try right now with newest image. I unforunately have not my orange next to me to try the newest (with compaling) image, but I will do in next 2 week.

Share this post


Link to post
Share on other sites
I got the exact same issue.  OPi_2E.
4.19.62-sunxi #5.92 SMP Wed Jul 31 22:07:23 CEST 2019 armv7l GNU/Linux
Hence. armbian with kernel v4.x isn't reliable absolutely.. :(

Sent from my MI 6 using Tapatalk

Share this post


Link to post
Share on other sites
I made a full upgrade from Stretch to Buster on two OrangePi 2E  and got the 1978 problem again after 2 days.
then thats it..we need to wait till develeopers do not fix it

Sent from my MI 6 using Tapatalk

Share this post


Link to post
Share on other sites

same problem here with orange pi pc,  was able to get the dmesg. Seems related to CPU issues

 

[Generic PHY] (mii_bus:phy_addr=0.1:01, irq=POLL)
[ 45.198097] dwmac-sun8i 1c30000.ethernet eth0: No Safety Features support found
[ 45.198114] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management Counters available
[ 45.198122] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW
[ 45.198366] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 49.126176] tun: Universal TUN/TAP device driver, 1.6
[ 49.279550] dwmac-sun8i 1c30000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[ 49.279597] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[24585.989889] hrtimer: interrupt took 1080818 ns
[33438.372119] g_ether gadget: high-speed config #2: RNDIS
[33438.668021] g_ether gadget: high-speed config #2: RNDIS
[42337.148990] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[42337.490138] Generic PHY 0.1:01: attached PHY driver
[Generic PHY] (mii_bus:phy_addr=0.1:01, irq=POLL)
[42337.492274] dwmac-sun8i 1c30000.ethernet eth0: No Safety Features support found
[42337.492289] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management Counters available
[42337.492297] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW
[42337.492534] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[42341.578518] dwmac-sun8i 1c30000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[42341.578564] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[66214.090239] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[66214.090266] rcu: 0-...!: (2 GPs behind) idle=5b6/1/0x40000000 softirq=7751741/7751741 fqs=1
[66214.090268] rcu: (detected by 2, t=53689 jiffies, g=8385141, q=6)
[66214.090282] Sending NMI from CPU 2 to CPUs 0:
[66214.090423] NMI backtrace for cpu 0
[66214.090426] CPU: 0 PID: 3449 Comm: leds Not tainted 4.19.57-sunxi #5.90.190705
[66214.090428] Hardware name: Allwinner sun8i Family
[66214.090430] PC is at 0xb6f4182e
[66214.090431] LR is at 0x4ec7c5
[66214.090433] pc :[] lr :
[<004ec7c5>] psr: 60080030
[66214.090435] sp : beda61a8 ip : 004fd024 fp : 00000000
[66214.090437] r10: beda6240 r9 : 00000000 r8 : 00000380
[66214.090439] r7 : beda6240 r6 : 00000000 r5 : 000175d2 r4 : 00000380
[66214.090441] r3 : 0000000e r2 : 004fd09c r1 : 00000000 r0 : 0000000e
[66214.090444] Flags: nZCv IRQs on FIQs on Mode USER_32 ISA Thumb Segment user
[66214.090446] Control: 50c5387d Table: 49b5806a DAC: 00000055
[66214.090448] CPU: 0 PID: 3449 Comm: leds Not tainted 4.19.57-sunxi #5.90.190705
[66214.090450] Hardware name: Allwinner sun8i Family
[66214.090452][] (unwind_backtrace) from[] (show_stack+0x11/0x14)
[66214.090455][] (show_stack) from[] (dump_stack+0x69/0x78)
[66214.090457][] (dump_stack) from[] (nmi_cpu_backtrace+0x59/0x90)
[66214.090459][] (nmi_cpu_backtrace) from[] (handle_IPI+0x85/0x2c0)
[66214.090462][] (handle_IPI) from[] (gic_handle_irq+0x67/0x68)
[66214.090464][] (gic_handle_irq) from[] (__irq_usr+0x61/0x80)
[66214.090466] Exception stack(0xe3cfbfb0 to 0xe3cfbff8)
[66214.090469] bfa0: 0000000e 00000000 004fd09c 0000000e
[66214.090471] bfc0: 00000380 000175d2 00000000 beda6240 00000380 00000000 beda6240 00000000
[66214.090473] bfe0: 004fd024 beda61a8 004ec7c5 b6f4182e 60080030 ffffffff
[66214.091295] rcu: rcu_sched kthread starved for 53687 jiffies! g8385141 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=2
[66214.091298] rcu: RCU grace-period kthread stack dump:
[66214.091301] rcu_sched I 0 10 2 0x00000000
[66214.091322][] (__schedule) from[] (schedule+0x2f/0x68)
[66214.091333][] (schedule) from[] (schedule_timeout+0x77/0x320)
[66214.091346][] (schedule_timeout) from[] (rcu_gp_kthread+0x41f/0x728)
[66214.091357][] (rcu_gp_kthread) from[] (kthread+0xfd/0x104)
[66214.091366][] (kthread) from[] (ret_from_fork+0x11/0x38)
[66214.091369] Exception stack(0xef14bfb0 to 0xef14bff8)
[66214.091374] bfa0: 00000000 00000000 00000000 00000000
[66214.091381] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[66214.091386] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 
[68055.316329] systemd-journald[243]: Failed to run event loop: Invalid argument
[81961.888905] g_ether gadget: high-speed config #2: RNDIS
[81962.216476] g_ether gadget: high-speed config #2: RNDIS

 

All systemctl commands fails with  Connection timed out Failed to  .. 


shutdown or reboot commands not working, 

 

I was able to force a kernel reboot with
echo b > /proc/sysrq-trigger

 

version:

Linux S485 4.19.57-sunxi #5.90.190705 SMP Fri Jul 5 00:47:13 CEST 2019 armv7l GNU/Linux
 

 

Share this post


Link to post
Share on other sites
6 hours ago, ehelfer said:
Quote

same problem here with orange pi pc,  was able to get the dmesg. Seems related to CPU issues

 

All systemctl commands fails with  Connection timed out Failed to  .. 


shutdown or reboot commands not working, 

 

I was able to force a kernel reboot with
echo b > /proc/sysrq-trigger

 

version:

Linux S485 4.19.57-sunxi #5.90.190705 SMP Fri Jul 5 00:47:13 CEST 2019 armv7l GNU/Linux

 

 

Good to know. Now I disabled the CPU powersafe mode by using a fixed CPU speed for minimum and maximum. 

With Buster, it's seems much easier to reproduce. Today it needed 6h and than it was 1978.

Now, I test with a fixed CPU speed.
 

 

 

Share this post


Link to post
Share on other sites

The current intermediate status...system is still running and uptime is now 13 days. With stretch sometimes I had an uptime of 40 days. But nothing in comparision with the uptime of my Raspberry systems (it runs for years).

 

keep the fingers cross and hope a fixed CPU speed is the solution for this issue.

 

Share this post


Link to post
Share on other sites

issue is getting intense... frequency of the incident is increasing.  I am on OPi+2E buster.  2 time travel incidents in 10 hours.  No changes or updates applied, same system ran for 2 weeks. 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
4 4