Pine64-LTS high kworker load with cpufreq problems after upgrade to 20.02.1


Recommended Posts

Armbianmonitor:

I am using Armbian Buster on Pine64-LTS units. I recently upgraded version 19.11.3 to 20.02.1

and since having problems with high kworker load and strange warning in dmesg output:

 

[    7.734729] ------------[ cut here ]------------
[    7.737084] WARNING: CPU: 1 PID: 30 at drivers/clocksource/arm_arch_timer.c:360 sun50i_a64_read_cntpct_el0+0x24/0x30
[    7.738928] Modules linked in: cpufreq_dt realtek axp20x_usb_power industrialio pinctrl_axp209 dw_hdmi_cec lima dw_hdmi_i2s_audio gpu_sched dwmac_sun8i mdio_mux
[    7.741485] CPU: 1 PID: 30 Comm: kworker/1:1 Not tainted 5.4.20-sunxi64 #20.02.1
[    7.743435] Hardware name: Pine64 LTS (DT)
[    7.745514] Workqueue: events dbs_work_handler
[    7.747504] pstate: 80000005 (Nzcv daif -PAN -UAO)
[    7.749508] pc : sun50i_a64_read_cntpct_el0+0x24/0x30
[    7.751489] lr : arch_counter_get_cntpct_stable+0x38/0x78
[    7.753420] sp : ffff8000110739c0
[    7.755295] x29: ffff8000110739c0 x28: ffff0000721e6200 
[    7.757379] x27: ffff00007246a200 x26: ffff000077b77098 
[    7.759351] x25: 0000000000000000 x24: 00000000ffffffff 
[    7.761348] x23: 0000000000000001 x22: ffff800011073b10 
[    7.763352] hrtimer: interrupt took 26239875 ns
[    7.765224] x21: 0000000000000000 x20: 0000000000000018 
[    7.767278] x19: ffff800010f4f000 x18: 0000000000000001 
[    7.769323] x17: 0000000000000018 x16: 0000000000000002 
[    7.771285] x15: 0000000000000001 x14: 0000000000000001 
[    7.773271] x13: 0000000000000005 x12: 0000000000000021 
[    7.775229] x11: 0000000000000005 x10: 00000000bcd3d800 
[    7.777217] x9 : 0000000000000004 x8 : 0000000000000004 
[    7.779167] x7 : ffff800010f1b5a0 x6 : 00000000112a8800 
[    7.781121] x5 : 0000000000000010 x4 : 000000000000ffff 
[    7.783220] x3 : 0000000000000050 x2 : 000000001ad70336 
[    7.785255] x1 : 0000000000000000 x0 : 000000001ad7033c 
[    7.787272] Call trace:
[    7.789327]  sun50i_a64_read_cntpct_el0+0x24/0x30
[    7.791318]  __delay+0x20/0xa0
[    7.793261]  __udelay+0x28/0x30
[    7.795273]  ccu_mux_notifier_cb+0x5c/0x90
[    7.797319]  notifier_call_chain+0x54/0x90
[    7.799359]  __srcu_notifier_call_chain+0x54/0x90
[    7.801372]  srcu_notifier_call_chain+0x14/0x20
[    7.803396]  __clk_notify+0x88/0xc8
[    7.805373]  clk_propagate_rate_change+0xb8/0xd0
[    7.807431]  clk_core_set_rate_nolock+0x114/0x1b8
[    7.809441]  clk_set_rate+0x34/0xa0
[    7.811426]  dev_pm_opp_set_rate+0x260/0x498
[    7.813493]  set_target+0x3c/0x80 [cpufreq_dt]
[    7.815441]  __cpufreq_driver_target+0x18c/0x580
[    7.817405]  od_dbs_update+0x138/0x190
[    7.819404]  dbs_work_handler+0x3c/0x70
[    7.821455]  process_one_work+0x1ec/0x370
[    7.823545]  worker_thread+0x4c/0x4f8
[    7.825518]  kthread+0x120/0x128
[    7.827490]  ret_from_fork+0x10/0x18
[    7.829444] ---[ end trace 869ed0a5cc6097b4 ]---

I have traced the high kworker load back to the cpufreq-dt module. When unloading this

module the high kworker load stops and system is usable again. As a workaround turning off

ondemand governor and selecting performance governor looked like it did the job.

 

But on other units with an X graphics desktop i get a much higher kworker load and the system

becomes unusable (slow). The only thing that works all the time is unloading the cpufreq-dt

module.

 

 

Link to post
Share on other sites
Donate and support the project!

UPDATE: unloading the module also in some cases did not work. I found out that using the default ondemand governor but changing the lower frequency to

the max frequency does 'fix' this problem. So to test it you can use the following command:

 

cpufreq-set -d 1.15G

 

Keep in mind the 1.15G is an example. You can find the max. supported frequency

with the command:

 

cpufreq-info

 

To make the changes permanent at each reboot create or edit the file at:

 

/etc/default/cpufrequtils

 

 

Link to post
Share on other sites

I had this same problem with kworker processes using 20-50% of CPU at idle.   Tried this method out with cpufreq-set -d 1.15G and immediately the kworker processes cpu utilization dropped.  

 

Question about making the changes permanent at each reboot.  I created the file /etc/default/cpufrequtils like you mentioned.  What do I put in there exactly?  From googling some put:

GOVERNOR="ondemand"
MIN_SPEED="800MHz"
MAX_SPEED="950MHz"

 

Except I would change the MIN_SPEED and MAX_SPEED to the same MAX_SPEED of my cpu (which is 1.15G for pine64+).  I've seen some put GOVERNOR='performance".  Not sure if that line is needed in our case?

 

Note: I just upgraded my pine64+ with 2 gigs ram to the latest stable kernel:

Linux pine64 5.4.28-sunxi64 #20.02.7 SMP Sat Mar 28 17:25:10 CET 2020 aarch64 aarch64 aarch64 GNU/Linux

prior to this I had the date/time issue that was fixed with an old patch from the 3.x legacy kernel era. 

Link to post
Share on other sites

I'm having the same problem on a Pine64-plus. 

 

[    6.465215] WARNING: CPU: 2 PID: 34 at drivers/clocksource/arm_arch_timer.c:364 sun50i_a64_read_cntpct_el0+0x2c/0x38
[    6.467239] Modules linked in: cpufreq_dt joydev sch_fq_codel goodix panel_feiyang_fy07024di26a30d pinctrl_axp209 realtek dwmac_sun8i mdio_mux i2c_mv64xxx pwm_bl
[    6.470647] CPU: 2 PID: 34 Comm: kworker/2:1 Not tainted 5.10.4-sunxi64 #20.11.7
[    6.472786] Hardware name: Pine64+ (DT)
[    6.475136] Workqueue: events dbs_work_handler
[    6.477389] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--)
[    6.479566] pc : sun50i_a64_read_cntpct_el0+0x2c/0x38
[    6.481723] lr : arch_counter_get_cntpct_stable+0x3c/0x80
[    6.483825] sp : ffff80001164ba30
[    6.485913] x29: ffff80001164ba30 x28: ffff0000422b9a00
[    6.488166] x27: ffff000043079f00 x26: ffff000077b6f098
[    6.490462] x25: 0000000000000000 x24: ffff000042de0428
[    6.492768] x23: ffff80001164bb30
[    6.494818] hrtimer: interrupt took 27554375 ns
[    6.496877] x22: 0000000000000001
[    6.499039] x21: 0000000000000001 x20: 0000000000000018
[    6.501309] x19: ffff80001144e000 x18: 0000000000000001
[    6.503570] x17: 0000000000000018
[    6.505752] pwm-backlight backlight: supply power not found, using dummy regulator
[    6.507789] x16: 0000000000000002
[    6.509935] x15: 0000000000000001 x14: 0000000000000001
[    6.512194] x13: 0000000000000005 x12: 0000000000000021
[    6.514410] x11: 0000000000000005 x10: 00000000bcd3d800
[    6.516648] x9 : 0000000000000004 x8 : 0000000000000004
[    6.518906] x7 : ffff800011411080 x6 : 00000000112a8800
[    6.521209] x5 : 0000000000000010 x4 : 0000000000000012
[    6.523465] x3 : 0000000000000050 x2 : 0000000015b25f37
[    6.525690] x1 : 0000000000000000 x0 : 0000000015b25f3d
[    6.527922] Call trace:
[    6.530233]  sun50i_a64_read_cntpct_el0+0x2c/0x38
[    6.532385]  __delay+0x24/0xa8
[    6.534647]  __udelay+0x2c/0x38
[    6.536827]  ccu_mux_notifier_cb+0x64/0xa0
[    6.538965]  srcu_notifier_call_chain+0x70/0xc0
[    6.541146]  __clk_notify+0x8c/0xc8
[    6.543303]  clk_propagate_rate_change+0xc4/0xe0
[    6.545451]  clk_core_set_rate_nolock+0x118/0x1f0
[    6.547603]  clk_set_rate+0x38/0xa8
[    6.549779]  dev_pm_opp_set_rate+0x258/0x5c8
[    6.551994]  set_target+0x30/0x40 [cpufreq_dt]
[    6.554186]  __cpufreq_driver_target+0x2b0/0x698
[    6.556342]  od_dbs_update+0x140/0x1a0
[    6.558463]  dbs_work_handler+0x40/0x78
[    6.560609]  process_one_work+0x1f0/0x3c0
[    6.562742]  worker_thread+0x140/0x520
[    6.564895]  kthread+0x120/0x128
[    6.567029]  ret_from_fork+0x10/0x30

unloading the cpufreq-dt module helped somewhat, but manipulating the min/max cpu speeds does little to help.

 

X is not installed, but I do have a simple graphical interface using FB.

 

regarding the kworker threads eating CPU cycles I also noticed this post in a rpi forum, but not sure how to apply the recommended:

 

dtparam=sd_poll_once dtoverlay=sdtweak,poll_once

tweaks in Armbian.

 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...