Date changed to 1978，System crash，Mysql run 195% of CPU

[arno-ppsspp]

Armbianmonitor:

http://ix.io/1D6H

Hello，Respected Developers：
I'm an Orange Pi user.In the process of using the armbian, I encountered serious problems.
My Control Board is Orange Pi PC Plus.Armbian is "ARMBIAN 5.59 stable Ubuntu 18.04.1 LTS 4.14.65-sunxi"
I installed jdk and mysql.And run a JAVA program.I set a time zone for Asia-Chongqing,The system language is Chinese,I use three serial ports.
When running for a period of time,There will be bugs.About one to three days.
The specific phenomena are as follows:
1.Run "date",the time is 1978.
2.Mysql occupy cpu 195%, occupy mem 27%,systemd occupy cpu 34%,Another systemd occupy cpu 14%
,systemd-jo+ occupy cpu 39%.
3.eth0‘s ip  disappear，Unable to connect remotely from the network for ssh.I only use serial port to connect.
4.I can't use the reboot command to restart,I input "reboot",It didn't respond.

 

Time automatically changed to 1978.

I hope you can help me solve this problem,Thanks very much.
 

 

[StefanH.]

hi, I found this topic by searching for "armbian 1978". out of the sudden my OrangePi Plus 2E was set to 1978 and the system was unusuable. couldn't ssh into it and even at the local console the screen/monitor didn't show anything but black screen.

So I was forced to hard-reboot it. 

 

The only thing I noticed was this email from cron daemon:

/etc/cron.hourly/fake-hwclock:
Time travel detected!
fake-hwclock release date is in the future: 2016-04-15 00:00:00
Current system time: 1978-05-27 23:17:02
To force the saved system clock backwards in time anyway, use "force"

 

Is this a known bug?

 

 

[martinayotte]

2 hours ago, StefanH. said:

Is this a known bug?

I've never seen such issue ...

[StefanH.]

weird, let's see if it happens again ;-)

[Be Osro]

On 5/14/2019 at 12:13 PM, StefanH. said:

hi, I found this topic by searching for "armbian 1978". out of the sudden my OrangePi Plus 2E was set to 1978 and the system was unusuable. couldn't ssh into it and even at the local console the screen/monitor didn't show anything but black screen.

So I was forced to hard-reboot it. 

 

The only thing I noticed was this email from cron daemon:

/etc/cron.hourly/fake-hwclock:
Time travel detected!
fake-hwclock release date is in the future: 2016-04-15 00:00:00
Current system time: 1978-05-27 23:17:02
To force the saved system clock backwards in time anyway, use "force"

 

Is this a known bug?

 

 

Same strange behaviour here. Same machine (OrangePi Plus 2E)! Looking for a solution/explication now.

[Marc_in_the_dark]

Hello, 

I found this thread by searching for "orange pi 1978" with Google because I have the same problem here with 2 identical Orange Pi Plus 2E systems.

One both systems I installed  ARMBIAN 5.73 stable - 4.19.20-sunxi and the problem is reproducible but it takes around 2 weeks till it happens.

Both systems running in the same network and do exactly the same (the 2nd is a backup-system) but the problem does not occur on both system at the same time.

 

In the moment, were the problem occur you have limited access to the system. SSH does not work, shutdown or reboot does not work and today I tried to sync the the systemtime with the RTC but it does not work. RTC has the correct time and system date was 7th May 1978.

The only solution is to cut the power and connect it again. I use a WLAN-power-socket to do this remote. 

 

Do you have any idea what I can try? I thought an external RTC modul could help and sync the time in case of the 1978-problem happens but today I learned the RTC is not the problem (sudo hwclock -r gives the right time) and it's not possible to set the date with sudo hwclock -s when it happens. 
 

Thanks for your help!

[tr1pp4]

Same issue here

 

could anyone find a solution? Did try some stuff with ntp services, but doesn't seem to be the problem

[Pol Isidor]

I have totally same problem..

I have connected 2 card reader to orange PI PC and maximum in 48-72h I can expirience the same problem. Can not access PI over ssh, black screen on HDMI out.

I tried (about 2months ago) to compile myself the actual version of armbian with tutorial and script what I found here on this forum, but I expirience the same issue.
My question is: Is this happening same with a newest kernel and relase?
Is thre any fix for this?

[Marc_in_the_dark]

the longest uptime without the 1978 issue was 45 days and the shortest uptime (ended yesterday) was 2 days. I tried so many things but could not solve it yet. The last change (yesterday) was to limit the cpu clockdown. Now it goes down to 800 and up to 1200Mhz in "conservative" mode. 

 

I ordered a second OrangePi +2E to make more tests. The second +2E shows the same problem and in the next step I will install the latest armbian image with the newest kernel. 

 

[Pol Isidor]

3 hours ago, Marc_in_the_dark said:

the longest uptime without the 1978 issue was 45 days and the shortest uptime (ended yesterday) was 2 days. I tried so many things but could not solve it yet. The last change (yesterday) was to limit the cpu clockdown. Now it goes down to 800 and up to 1200Mhz in "conservative" mode. 

 

I ordered a second OrangePi +2E to make more tests. The second +2E shows the same problem and in the next step I will install the latest armbian image with the newest kernel. 

 

Thx for replay..but I need to disappointed you. I have even the same board what u ordered OrangePI +2E, and I was with same logic that maybe it is related with the OrangePI PC model, but no. It happening the same, totally. So you can jump over that step and try right now with newest image. I unforunately have not my orange next to me to try the newest (with compaling) image, but I will do in next 2 week.

[sadnblueish]

I got the exact same issue.  OPi_2E.

4.19.62-sunxi #5.92 SMP Wed Jul 31 22:07:23 CEST 2019 armv7l GNU/Linux

[Pol Isidor]

I got the exact same issue.  OPi_2E.
4.19.62-sunxi #5.92 SMP Wed Jul 31 22:07:23 CEST 2019 armv7l GNU/Linux

Hence. armbian with kernel v4.x isn't reliable absolutely..

Sent from my MI 6 using Tapatalk

[Marc_in_the_dark]

I made a full upgrade from Stretch to Buster on two OrangePi 2E  and got the 1978 problem again after 2 days.

[Pol Isidor]

I made a full upgrade from Stretch to Buster on two OrangePi 2E  and got the 1978 problem again after 2 days.

then thats it..we need to wait till develeopers do not fix it

Sent from my MI 6 using Tapatalk

[ehelfer]

same problem here with orange pi pc,  was able to get the dmesg. Seems related to CPU issues

 

[Generic PHY] (mii_bus:phy_addr=0.1:01, irq=POLL)
[ 45.198097] dwmac-sun8i 1c30000.ethernet eth0: No Safety Features support found
[ 45.198114] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management Counters available
[ 45.198122] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW
[ 45.198366] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 49.126176] tun: Universal TUN/TAP device driver, 1.6
[ 49.279550] dwmac-sun8i 1c30000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[ 49.279597] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[24585.989889] hrtimer: interrupt took 1080818 ns
[33438.372119] g_ether gadget: high-speed config #2: RNDIS
[33438.668021] g_ether gadget: high-speed config #2: RNDIS
[42337.148990] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[42337.490138] Generic PHY 0.1:01: attached PHY driver
[Generic PHY] (mii_bus:phy_addr=0.1:01, irq=POLL)
[42337.492274] dwmac-sun8i 1c30000.ethernet eth0: No Safety Features support found
[42337.492289] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management Counters available
[42337.492297] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW
[42337.492534] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[42341.578518] dwmac-sun8i 1c30000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[42341.578564] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[66214.090239] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[66214.090266] rcu: 0-...!: (2 GPs behind) idle=5b6/1/0x40000000 softirq=7751741/7751741 fqs=1
[66214.090268] rcu: (detected by 2, t=53689 jiffies, g=8385141, q=6)
[66214.090282] Sending NMI from CPU 2 to CPUs 0:
[66214.090423] NMI backtrace for cpu 0
[66214.090426] CPU: 0 PID: 3449 Comm: leds Not tainted 4.19.57-sunxi #5.90.190705
[66214.090428] Hardware name: Allwinner sun8i Family
[66214.090430] PC is at 0xb6f4182e
[66214.090431] LR is at 0x4ec7c5
[66214.090433] pc :[] lr :
[<004ec7c5>] psr: 60080030
[66214.090435] sp : beda61a8 ip : 004fd024 fp : 00000000
[66214.090437] r10: beda6240 r9 : 00000000 r8 : 00000380
[66214.090439] r7 : beda6240 r6 : 00000000 r5 : 000175d2 r4 : 00000380
[66214.090441] r3 : 0000000e r2 : 004fd09c r1 : 00000000 r0 : 0000000e
[66214.090444] Flags: nZCv IRQs on FIQs on Mode USER_32 ISA Thumb Segment user
[66214.090446] Control: 50c5387d Table: 49b5806a DAC: 00000055
[66214.090448] CPU: 0 PID: 3449 Comm: leds Not tainted 4.19.57-sunxi #5.90.190705
[66214.090450] Hardware name: Allwinner sun8i Family
[66214.090452][] (unwind_backtrace) from[] (show_stack+0x11/0x14)
[66214.090455][] (show_stack) from[] (dump_stack+0x69/0x78)
[66214.090457][] (dump_stack) from[] (nmi_cpu_backtrace+0x59/0x90)
[66214.090459][] (nmi_cpu_backtrace) from[] (handle_IPI+0x85/0x2c0)
[66214.090462][] (handle_IPI) from[] (gic_handle_irq+0x67/0x68)
[66214.090464][] (gic_handle_irq) from[] (__irq_usr+0x61/0x80)
[66214.090466] Exception stack(0xe3cfbfb0 to 0xe3cfbff8)
[66214.090469] bfa0: 0000000e 00000000 004fd09c 0000000e
[66214.090471] bfc0: 00000380 000175d2 00000000 beda6240 00000380 00000000 beda6240 00000000
[66214.090473] bfe0: 004fd024 beda61a8 004ec7c5 b6f4182e 60080030 ffffffff
[66214.091295] rcu: rcu_sched kthread starved for 53687 jiffies! g8385141 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=2
[66214.091298] rcu: RCU grace-period kthread stack dump:
[66214.091301] rcu_sched I 0 10 2 0x00000000
[66214.091322][] (__schedule) from[] (schedule+0x2f/0x68)
[66214.091333][] (schedule) from[] (schedule_timeout+0x77/0x320)
[66214.091346][] (schedule_timeout) from[] (rcu_gp_kthread+0x41f/0x728)
[66214.091357][] (rcu_gp_kthread) from[] (kthread+0xfd/0x104)
[66214.091366][] (kthread) from[] (ret_from_fork+0x11/0x38)
[66214.091369] Exception stack(0xef14bfb0 to 0xef14bff8)
[66214.091374] bfa0: 00000000 00000000 00000000 00000000
[66214.091381] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[66214.091386] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 
[68055.316329] systemd-journald[243]: Failed to run event loop: Invalid argument
[81961.888905] g_ether gadget: high-speed config #2: RNDIS
[81962.216476] g_ether gadget: high-speed config #2: RNDIS

 

All systemctl commands fails with  Connection timed out Failed to  .. 

shutdown or reboot commands not working, 

 

I was able to force a kernel reboot with
echo b > /proc/sysrq-trigger

 

version:

Linux S485 4.19.57-sunxi #5.90.190705 SMP Fri Jul 5 00:47:13 CEST 2019 armv7l GNU/Linux
 

 

[Marc_in_the_dark]

6 hours ago, ehelfer said:

Quote

same problem here with orange pi pc,  was able to get the dmesg. Seems related to CPU issues

 

All systemctl commands fails with  Connection timed out Failed to  .. 

shutdown or reboot commands not working, 

 

I was able to force a kernel reboot with
echo b > /proc/sysrq-trigger

 

version:

Linux S485 4.19.57-sunxi #5.90.190705 SMP Fri Jul 5 00:47:13 CEST 2019 armv7l GNU/Linux

 

 

Good to know. Now I disabled the CPU powersafe mode by using a fixed CPU speed for minimum and maximum. 

With Buster, it's seems much easier to reproduce. Today it needed 6h and than it was 1978.

Now, I test with a fixed CPU speed.
 

 

 

[Pol Isidor]

 

tell us the result..thx

Sent from my MI 6 using Tapatalk

[Marc_in_the_dark]

The current intermediate status...system is still running and uptime is now 13 days. With stretch sometimes I had an uptime of 40 days. But nothing in comparision with the uptime of my Raspberry systems (it runs for years).

 

keep the fingers cross and hope a fixed CPU speed is the solution for this issue.

 

[sadnblueish]

issue is getting intense... frequency of the incident is increasing.  I am on OPi+2E buster.  2 time travel incidents in 10 hours.  No changes or updates applied, same system ran for 2 weeks. 

[DarkDream]

Hi guys:

 

I'm facing the same issue but with a NanoPI (H3 chipset).

Did the fixed CPU speed solved anything?

[Marc_in_the_dark]

Uptime is now 28 days. Maybe the fixed CPU speed is the solution for this problem.

But the longest uptime I ever had was 40 days. I have to break this record first.

[Pol Isidor]

but what exactly you did with cpu?

Sent from my MI 6 using Tapatalk

[Pol Isidor]

but what exactly you did with cpu?

Sent from my MI 6 using Tapatalk

[Pol Isidor]

but what exactly you did with cpu?

Sent from my MI 6 using Tapatalk

[Marc_in_the_dark]

set the minimum CPU speed to 110400 and maximum speed to 110400. CPU governor = conservative (armbian config > system > CPU).
So the processor will not change the CPU clock.

[StefanH.]

wow, I wasn't notified about all the replies here since I posted in May 2019. Unfortunately the problem has now occured 3 times since April. I have gone through all the dist-upgrades but the issue still persists.

This is soooo annoying as I run my home automation system on the Orange Pi. The WAF is going dramatically down :-(

Will try the CPU speed setting as well, but usually it takes a few months for the issue to re-appear on my side

[DarkDream]

In my case, the watchdog was installed but wasn´t working. Try with a fork bomb in python and if it hangs the system, it is not working properly.

You can write in the /dev/watchdog each few seconds with a cron script and this time, the watchdog will reset the system.

[Mikalai Kavalchuk]

Hello everyone!

I have same issue with Orange Pi Zero (Linux orangepizero 4.19.62-sunxi #5.92 SMP Wed Jul 31 22:07:23 CEST 2019 armv7l GNU/Linux)

I have running 20 devices. And now 4 devices have 1978 year and I can't connect thru ssh..

Devices sends every 30 seconds 'alive' message via MQTT, so I see this trouble but can't do anything because devices are remote and no access thru ssh.

I changed CPU speed to fixed 816 Mhz at available devices, and I will monitoring this situation and hope this help.

 

Maybe someone have any news/solutions for this trouble?

 

Founded how to remotely reboot OPi with 1978 date error using SSH:

ssh username@yourIP -p yourPort "echo 1 > /proc/sys/kern/sysrq;echo b > /proc/sysrq-trigger"

[sadnblueish]

I have no technical explanation to my workaround but I tried:

Remove fakehwclock

apt purge --autoremove fake-hwclock

 

Reinstall fakehwclock and install ntpdate:

apt update
apt install -y fake-hwclock ntpdate

Install hourly cron job to save the time from ntpdate command (Since system drifted in 1978, still communicates over IP (non-secured;ssh won't work):

touch /etc/cron.hourly/ntpdate

chmod 755 /etc/cron.hourly/ntpdate

echo "/usr/sbin/ntpdate -4 -b -u 0.debian.pool.ntp.org" > /etc/cron.hourly/ntpdate

 

Since my last reported problem (on this forum), I didn't get the issue (fact remains, the issue is intermittent). No harm in trying though! I didn't change anything else (CPU speed etc). I am on OPi+2E.

[denix]

Are there some new information's? We use more than 1500 pcs of orange pc plus. There are similar problems with freeze and date change. Main difference is that changed date is 1979.