Patrick Peters Posted February 19, 2020 Posted February 19, 2020 Armbianmonitor: http://ix.io/2caD I am using Armbian Buster on Pine64-LTS units. I recently upgraded version 19.11.3 to 20.02.1 and since having problems with high kworker load and strange warning in dmesg output: [ 7.734729] ------------[ cut here ]------------ [ 7.737084] WARNING: CPU: 1 PID: 30 at drivers/clocksource/arm_arch_timer.c:360 sun50i_a64_read_cntpct_el0+0x24/0x30 [ 7.738928] Modules linked in: cpufreq_dt realtek axp20x_usb_power industrialio pinctrl_axp209 dw_hdmi_cec lima dw_hdmi_i2s_audio gpu_sched dwmac_sun8i mdio_mux [ 7.741485] CPU: 1 PID: 30 Comm: kworker/1:1 Not tainted 5.4.20-sunxi64 #20.02.1 [ 7.743435] Hardware name: Pine64 LTS (DT) [ 7.745514] Workqueue: events dbs_work_handler [ 7.747504] pstate: 80000005 (Nzcv daif -PAN -UAO) [ 7.749508] pc : sun50i_a64_read_cntpct_el0+0x24/0x30 [ 7.751489] lr : arch_counter_get_cntpct_stable+0x38/0x78 [ 7.753420] sp : ffff8000110739c0 [ 7.755295] x29: ffff8000110739c0 x28: ffff0000721e6200 [ 7.757379] x27: ffff00007246a200 x26: ffff000077b77098 [ 7.759351] x25: 0000000000000000 x24: 00000000ffffffff [ 7.761348] x23: 0000000000000001 x22: ffff800011073b10 [ 7.763352] hrtimer: interrupt took 26239875 ns [ 7.765224] x21: 0000000000000000 x20: 0000000000000018 [ 7.767278] x19: ffff800010f4f000 x18: 0000000000000001 [ 7.769323] x17: 0000000000000018 x16: 0000000000000002 [ 7.771285] x15: 0000000000000001 x14: 0000000000000001 [ 7.773271] x13: 0000000000000005 x12: 0000000000000021 [ 7.775229] x11: 0000000000000005 x10: 00000000bcd3d800 [ 7.777217] x9 : 0000000000000004 x8 : 0000000000000004 [ 7.779167] x7 : ffff800010f1b5a0 x6 : 00000000112a8800 [ 7.781121] x5 : 0000000000000010 x4 : 000000000000ffff [ 7.783220] x3 : 0000000000000050 x2 : 000000001ad70336 [ 7.785255] x1 : 0000000000000000 x0 : 000000001ad7033c [ 7.787272] Call trace: [ 7.789327] sun50i_a64_read_cntpct_el0+0x24/0x30 [ 7.791318] __delay+0x20/0xa0 [ 7.793261] __udelay+0x28/0x30 [ 7.795273] ccu_mux_notifier_cb+0x5c/0x90 [ 7.797319] notifier_call_chain+0x54/0x90 [ 7.799359] __srcu_notifier_call_chain+0x54/0x90 [ 7.801372] srcu_notifier_call_chain+0x14/0x20 [ 7.803396] __clk_notify+0x88/0xc8 [ 7.805373] clk_propagate_rate_change+0xb8/0xd0 [ 7.807431] clk_core_set_rate_nolock+0x114/0x1b8 [ 7.809441] clk_set_rate+0x34/0xa0 [ 7.811426] dev_pm_opp_set_rate+0x260/0x498 [ 7.813493] set_target+0x3c/0x80 [cpufreq_dt] [ 7.815441] __cpufreq_driver_target+0x18c/0x580 [ 7.817405] od_dbs_update+0x138/0x190 [ 7.819404] dbs_work_handler+0x3c/0x70 [ 7.821455] process_one_work+0x1ec/0x370 [ 7.823545] worker_thread+0x4c/0x4f8 [ 7.825518] kthread+0x120/0x128 [ 7.827490] ret_from_fork+0x10/0x18 [ 7.829444] ---[ end trace 869ed0a5cc6097b4 ]--- I have traced the high kworker load back to the cpufreq-dt module. When unloading this module the high kworker load stops and system is usable again. As a workaround turning off ondemand governor and selecting performance governor looked like it did the job. But on other units with an X graphics desktop i get a much higher kworker load and the system becomes unusable (slow). The only thing that works all the time is unloading the cpufreq-dt module.
Patrick Peters Posted March 14, 2020 Author Posted March 14, 2020 UPDATE: unloading the module also in some cases did not work. I found out that using the default ondemand governor but changing the lower frequency to the max frequency does 'fix' this problem. So to test it you can use the following command: cpufreq-set -d 1.15G Keep in mind the 1.15G is an example. You can find the max. supported frequency with the command: cpufreq-info To make the changes permanent at each reboot create or edit the file at: /etc/default/cpufrequtils
markolonius Posted May 17, 2020 Posted May 17, 2020 I had this same problem with kworker processes using 20-50% of CPU at idle. Tried this method out with cpufreq-set -d 1.15G and immediately the kworker processes cpu utilization dropped. Question about making the changes permanent at each reboot. I created the file /etc/default/cpufrequtils like you mentioned. What do I put in there exactly? From googling some put: GOVERNOR="ondemand" MIN_SPEED="800MHz" MAX_SPEED="950MHz" Except I would change the MIN_SPEED and MAX_SPEED to the same MAX_SPEED of my cpu (which is 1.15G for pine64+). I've seen some put GOVERNOR='performance". Not sure if that line is needed in our case? Note: I just upgraded my pine64+ with 2 gigs ram to the latest stable kernel: Linux pine64 5.4.28-sunxi64 #20.02.7 SMP Sat Mar 28 17:25:10 CET 2020 aarch64 aarch64 aarch64 GNU/Linux prior to this I had the date/time issue that was fixed with an old patch from the 3.x legacy kernel era.
Learnincurve Posted January 14, 2021 Posted January 14, 2021 I'm having the same problem on a Pine64-plus. [ 6.465215] WARNING: CPU: 2 PID: 34 at drivers/clocksource/arm_arch_timer.c:364 sun50i_a64_read_cntpct_el0+0x2c/0x38 [ 6.467239] Modules linked in: cpufreq_dt joydev sch_fq_codel goodix panel_feiyang_fy07024di26a30d pinctrl_axp209 realtek dwmac_sun8i mdio_mux i2c_mv64xxx pwm_bl [ 6.470647] CPU: 2 PID: 34 Comm: kworker/2:1 Not tainted 5.10.4-sunxi64 #20.11.7 [ 6.472786] Hardware name: Pine64+ (DT) [ 6.475136] Workqueue: events dbs_work_handler [ 6.477389] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--) [ 6.479566] pc : sun50i_a64_read_cntpct_el0+0x2c/0x38 [ 6.481723] lr : arch_counter_get_cntpct_stable+0x3c/0x80 [ 6.483825] sp : ffff80001164ba30 [ 6.485913] x29: ffff80001164ba30 x28: ffff0000422b9a00 [ 6.488166] x27: ffff000043079f00 x26: ffff000077b6f098 [ 6.490462] x25: 0000000000000000 x24: ffff000042de0428 [ 6.492768] x23: ffff80001164bb30 [ 6.494818] hrtimer: interrupt took 27554375 ns [ 6.496877] x22: 0000000000000001 [ 6.499039] x21: 0000000000000001 x20: 0000000000000018 [ 6.501309] x19: ffff80001144e000 x18: 0000000000000001 [ 6.503570] x17: 0000000000000018 [ 6.505752] pwm-backlight backlight: supply power not found, using dummy regulator [ 6.507789] x16: 0000000000000002 [ 6.509935] x15: 0000000000000001 x14: 0000000000000001 [ 6.512194] x13: 0000000000000005 x12: 0000000000000021 [ 6.514410] x11: 0000000000000005 x10: 00000000bcd3d800 [ 6.516648] x9 : 0000000000000004 x8 : 0000000000000004 [ 6.518906] x7 : ffff800011411080 x6 : 00000000112a8800 [ 6.521209] x5 : 0000000000000010 x4 : 0000000000000012 [ 6.523465] x3 : 0000000000000050 x2 : 0000000015b25f37 [ 6.525690] x1 : 0000000000000000 x0 : 0000000015b25f3d [ 6.527922] Call trace: [ 6.530233] sun50i_a64_read_cntpct_el0+0x2c/0x38 [ 6.532385] __delay+0x24/0xa8 [ 6.534647] __udelay+0x2c/0x38 [ 6.536827] ccu_mux_notifier_cb+0x64/0xa0 [ 6.538965] srcu_notifier_call_chain+0x70/0xc0 [ 6.541146] __clk_notify+0x8c/0xc8 [ 6.543303] clk_propagate_rate_change+0xc4/0xe0 [ 6.545451] clk_core_set_rate_nolock+0x118/0x1f0 [ 6.547603] clk_set_rate+0x38/0xa8 [ 6.549779] dev_pm_opp_set_rate+0x258/0x5c8 [ 6.551994] set_target+0x30/0x40 [cpufreq_dt] [ 6.554186] __cpufreq_driver_target+0x2b0/0x698 [ 6.556342] od_dbs_update+0x140/0x1a0 [ 6.558463] dbs_work_handler+0x40/0x78 [ 6.560609] process_one_work+0x1f0/0x3c0 [ 6.562742] worker_thread+0x140/0x520 [ 6.564895] kthread+0x120/0x128 [ 6.567029] ret_from_fork+0x10/0x30 unloading the cpufreq-dt module helped somewhat, but manipulating the min/max cpu speeds does little to help. X is not installed, but I do have a simple graphical interface using FB. regarding the kworker threads eating CPU cycles I also noticed this post in a rpi forum, but not sure how to apply the recommended: dtparam=sd_poll_once dtoverlay=sdtweak,poll_once tweaks in Armbian.
Patrick Peters Posted February 8, 2021 Author Posted February 8, 2021 Hi Learnincurve, Have you tried the following settings in /etc/default/cpufrequtils ENABLE=true MIN_SPEED=1152000 MAX_SPEED=1152000 GOVERNOR=ondemand The dtparam you specified has to do with the settings contained in the DTB. I do not know what is actually changed on the rPI. Also keep in mind rPI is rather far of from the Pine when it comes to DTB settings... Please also make sure you are actually have this problem during idle situations and you have enough free memory available. The kworker also gets high load in case of high disk i/o load; in those cases it is just normal.
Recommended Posts