Kalin

I am having clock issue with kernels 5.0.5 and 5.0.7, where "Allwinner erratum UNKNOWN1" is active. Here are some additional observations: 1) It only affects system time. Hardware clock (sudo hwclock) is still correct. 2) Trying to execute "sudo hwclock -s", returns error for invalid argument 3) After reboot "sudo hwclock -s" works and system time is updated. 4) A reboot soon after "sudo hwclock -s" results in the wrong system time being back right away. 5) If I wait for a 1-2 minutes after setting the correct system time, the reboot does not break the time. 6) I need to reconnect the keyboard attached to the board, each time the issue appears. It seems the issue affects USB, as the following is logged in kern.log: Jun 3 01:33:54 pine1 kernel: [105898.980814] ohci-platform 1c1a400.usb: frame counter not updating; disabled Jun 3 01:33:54 pine1 kernel: [105898.980828] ohci-platform 1c1a400.usb: HC died; cleaning up 7) The following types of errors are also recorded in kern.log: "rcu: INFO: rcu_sched self-detected stall on CPU" or "rcu: INFO: rcu_sched detected stalls on CPUs/tasks: ". I am attaching the full kern.log with different variations of the above errors. I am running same kernel on 5 boards and the issue happens on one board only. It was happening on all boards with previous kernels, before the "UNKNOWN1" patch. The board where it happens is receiving a bit more load, as it is kubernetes master. The load is not big at all, as the cluster is empty at the moment. Here are some details, captured after the issue appeared: + uname -a Linux pine1 5.0.5-sunxi64 #5.77.190401 SMP Mon Apr 1 17:57:15 CEST 2019 aarch64 aarch64 aarch64 GNU/Linux + grep ERRATUM /boot/config-5.0.5-sunxi64 CONFIG_ARM64_ERRATUM_826319=y CONFIG_ARM64_ERRATUM_827319=y CONFIG_ARM64_ERRATUM_824069=y CONFIG_ARM64_ERRATUM_819472=y # CONFIG_ARM64_ERRATUM_832075 is not set CONFIG_ARM64_ERRATUM_845719=y CONFIG_ARM64_ERRATUM_843419=y CONFIG_ARM64_ERRATUM_1024718=y CONFIG_ARM64_ERRATUM_1188873=y CONFIG_ARM64_ERRATUM_1165522=y CONFIG_ARM64_ERRATUM_1286807=y # CONFIG_CAVIUM_ERRATUM_22375 is not set CONFIG_CAVIUM_ERRATUM_23144=y # CONFIG_CAVIUM_ERRATUM_23154 is not set # CONFIG_CAVIUM_ERRATUM_27456 is not set # CONFIG_CAVIUM_ERRATUM_30115 is not set # CONFIG_QCOM_FALKOR_ERRATUM_1003 is not set # CONFIG_QCOM_FALKOR_ERRATUM_1009 is not set # CONFIG_QCOM_QDF2400_ERRATUM_0065 is not set CONFIG_HISILICON_ERRATUM_161600802=y CONFIG_QCOM_FALKOR_ERRATUM_E1041=y CONFIG_FSL_ERRATUM_A008585=y # CONFIG_HISILICON_ERRATUM_161010101 is not set # CONFIG_ARM64_ERRATUM_858921 is not set CONFIG_SUN50I_ERRATUM_UNKNOWN1=y + hwclock 2019-04-15 05:36:16.535140+0000 + date Mon Jun 4 19:51:04 UTC 2114 + cat /var/log/syslog + grep UNKNOWN Jun 1 21:50:34 pine1 kernel: [ 0.000000] arch_timer: Enabling global workaround for Allwinner erratum UNKNOWN1 Jun 1 21:53:07 pine1 kernel: [ 0.000000] arch_timer: Enabling global workaround for Allwinner erratum UNKNOWN1 Jun 1 21:50:34 pine1 kernel: [ 0.000000] arch_timer: Enabling global workaround for Allwinner erratum UNKNOWN1 Jun 1 21:50:34 pine1 kernel: [ 0.000000] arch_timer: Enabling global workaround for Allwinner erratum UNKNOWN1 Jun 1 21:50:34 pine1 kernel: [ 0.000000] arch_timer: Enabling global workaround for Allwinner erratum UNKNOWN1 Apr 12 05:39:44 pine1 kernel: [ 0.000000] arch_timer: Enabling global workaround for Allwinner erratum UNKNOWN1 Apr 12 05:50:18 pine1 kernel: [ 0.000000] arch_timer: Enabling global workaround for Allwinner erratum UNKNOWN1 Apr 12 05:54:21 pine1 kernel: [ 0.000000] arch_timer: Enabling global workaround for Allwinner erratum UNKNOWN1 kern.log

Thanks, this was indeed missing in dmesg output. I switched to latest kernel 5.0.7, where this is again active.

I am experiencing the issue with 5.0.6. kernel. Here are some details: + uname -a Linux pine1 5.0.6-sunxi64 #5.78.190407 SMP Mon Apr 8 00:11:35 CEST 2019 aarch64 aarch64 aarch64 GNU/Linux + grep ERRATUM /boot/config-5.0.6-sunxi64 CONFIG_ARM64_ERRATUM_826319=y CONFIG_ARM64_ERRATUM_827319=y CONFIG_ARM64_ERRATUM_824069=y CONFIG_ARM64_ERRATUM_819472=y # CONFIG_ARM64_ERRATUM_832075 is not set CONFIG_ARM64_ERRATUM_845719=y CONFIG_ARM64_ERRATUM_843419=y CONFIG_ARM64_ERRATUM_1024718=y CONFIG_ARM64_ERRATUM_1188873=y CONFIG_ARM64_ERRATUM_1165522=y CONFIG_ARM64_ERRATUM_1286807=y # CONFIG_CAVIUM_ERRATUM_22375 is not set CONFIG_CAVIUM_ERRATUM_23144=y # CONFIG_CAVIUM_ERRATUM_23154 is not set # CONFIG_CAVIUM_ERRATUM_27456 is not set # CONFIG_CAVIUM_ERRATUM_30115 is not set # CONFIG_QCOM_FALKOR_ERRATUM_1003 is not set # CONFIG_QCOM_FALKOR_ERRATUM_1009 is not set # CONFIG_QCOM_QDF2400_ERRATUM_0065 is not set CONFIG_HISILICON_ERRATUM_161600802=y CONFIG_QCOM_FALKOR_ERRATUM_E1041=y CONFIG_FSL_ERRATUM_A008585=y # CONFIG_HISILICON_ERRATUM_161010101 is not set # CONFIG_ARM64_ERRATUM_858921 is not set CONFIG_SUN50I_ERRATUM_UNKNOWN1=y + hwclock 2019-04-10 05:16:28.921505+0000 + date Wed May 30 11:22:32 UTC 2114 Relevant part of kern.log: Apr 9 13:52:45 pine1 kernel: [ 29.628109] cni0: port 3(veth778aa596) entered blocking state Apr 9 13:52:45 pine1 kernel: [ 29.628124] cni0: port 3(veth778aa596) entered disabled state Apr 9 13:52:45 pine1 kernel: [ 29.628569] device veth778aa596 entered promiscuous mode Apr 9 13:52:45 pine1 kernel: [ 29.628668] cni0: port 3(veth778aa596) entered blocking state Apr 9 13:52:45 pine1 kernel: [ 29.628676] cni0: port 3(veth778aa596) entered forwarding state May 29 20:39:46 pine1 kernel: [ 2488.514955] ohci-platform 1c1a400.usb: frame counter not updating; disabled May 29 20:39:46 pine1 kernel: [ 2488.514970] ohci-platform 1c1a400.usb: HC died; cleaning up May 29 20:39:46 pine1 kernel: [ 2488.593368] usb 2-1: USB disconnect, device number 2 May 30 11:20:02 pine1 kernel: [55306.258901] Console: switching to colour frame buffer device 240x67 May 30 11:20:02 pine1 kernel: [55306.308636] sun4i-drm display-engine: fb0: DRM emulated frame buffer device Relevant part of syslog: Apr 9 14:09:24 pine1 systemd-timesyncd[489]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com). Apr 9 14:15:01 pine1 CRON[4391]: (root) CMD (/usr/lib/armbian/armbian-truncate-logs) Apr 9 14:15:01 pine1 CRON[4392]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Apr 9 14:17:01 pine1 CRON[4631]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Apr 9 14:25:01 pine1 CRON[5548]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Apr 9 14:30:01 pine1 CRON[6130]: (root) CMD (/usr/lib/armbian/armbian-truncate-logs) May 29 20:39:46 pine1 kernel: [ 2488.514955] ohci-platform 1c1a400.usb: frame counter not updating; disabled May 29 20:39:46 pine1 kernel: [ 2488.514970] ohci-platform 1c1a400.usb: HC died; cleaning up May 29 20:39:46 pine1 kernel: [ 2488.593368] usb 2-1: USB disconnect, device number 2

Sharing some detailed explanation, available here https://lore.kernel.org/patchwork/patch/1031565/

I read somewhere that changing system time with a big difference is rejected by kernel, because it "will take too long". I have a question - are you saying that the 95 year jump happens at the place where hwclock looks or it is just in the " linux system time"? I have been running an older kernel (3.x) for few weeks and did not encounter this issue. The kernel I used has certain patches related to time, that are visible here: https://github.com/ayufan-pine64/linux-pine64/tree/my-hacks-1.2-with-drm/kernel/time I am not sure how to tell if these patches are in the armbian kernel.

Sure, here is the result grep ERRATUM /boot/config* CONFIG_ARM64_ERRATUM_826319=y CONFIG_ARM64_ERRATUM_827319=y CONFIG_ARM64_ERRATUM_824069=y CONFIG_ARM64_ERRATUM_819472=y # CONFIG_ARM64_ERRATUM_832075 is not set CONFIG_ARM64_ERRATUM_845719=y CONFIG_ARM64_ERRATUM_843419=y CONFIG_ARM64_ERRATUM_1024718=y # CONFIG_CAVIUM_ERRATUM_22375 is not set CONFIG_CAVIUM_ERRATUM_23144=y # CONFIG_CAVIUM_ERRATUM_23154 is not set # CONFIG_CAVIUM_ERRATUM_27456 is not set # CONFIG_CAVIUM_ERRATUM_30115 is not set # CONFIG_QCOM_FALKOR_ERRATUM_1003 is not set # CONFIG_QCOM_FALKOR_ERRATUM_1009 is not set # CONFIG_QCOM_QDF2400_ERRATUM_0065 is not set CONFIG_HISILICON_ERRATUM_161600802=y CONFIG_QCOM_FALKOR_ERRATUM_E1041=y CONFIG_FSL_ERRATUM_A008585=y # CONFIG_HISILICON_ERRATUM_161010101 is not set # CONFIG_ARM64_ERRATUM_858921 is not set

I am experiencing the same issue on sopine 64 with quite recent kernel: 4.19.20-sunxi64