Helios64 u-boot does not build anymore after we bumped to 2022.07

[prahal]

On 1/12/2024 at 9:26 AM, ebin-dev said:

There are many values to choose from between 1000 and 40000 (regulator-ramp-delay). Why don't you try 2000, 4000, 10000, 20000 ? (It might solve your problem)

If the issue is that the cpu frequency is switched too fast and I can reproduce the crash with a regulator-ramp-delay of 1000, then there is no point in testing anything above 1000 that will make the issue worse.

regulator-ramp-delay is badly named. It is not a dealy it is a divider for the delay. The greater regulator-ramp-delay the fastest the transition (I believe the Kobol team made this mistake, but as I also believe the issue could be otherwise than the delay between transitions this is not a big deal).

I still have not tried with a lower than 1000 value for regulator-ramp-delay (ie without tweaking the opp voltages as I am currently doing).

[BipBip1981]

Hi,

For moment not crash with my pattern test.

Tthis evening i run cpufreq-switching-2-b and post result

Keep in touch

 

root@helios64:~#  btrfs check --readonly  --check-data-csum  --progress /dev/disk/by-uuid/1d4e2c84-1c43-4d73-8acb-14d5a7aa1c4d
Opening filesystem to check...
Checking filesystem on /dev/disk/by-uuid/1d4e2c84-1c43-4d73-8acb-14d5a7aa1c4d
UUID: 1d4e2c84-1c43-4d73-8acb-14d5a7aa1c4d
[1/7] checking root items                      (0:04:20 elapsed, 6258640 items checked)
[2/7] checking extents                         (0:22:26 elapsed, 613032 items checked)
[3/7] checking free space cache                (0:08:50 elapsed, 5519 items checked)
[4/7] checking fs roots                        (8:40:20 elapsed, 169002 items checked)
[5/7] checking csums against data              (35:54:03 elapsed, 2619191 items checked)

 

helios64@helios64:~$ cat /proc/mdstat
Personalities : [raid10] [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4]
md0 : active raid10 sdd1[0] sdc1[5] sde1[4] sda1[2]
      15627581440 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      [=>...................]  check =  5.2% (821480320/15627581440) finish=1681.3min speed=146770K/sec
      bitmap: 0/117 pages [0KB], 65536KB chunk

unused devices: <none>

 

helios64@helios64:~$ uptime
 11:05:34 up 1 day, 22:02,  3 users,  load average: 2,98, 2,82, 2,78

[BipBip1981]

Hi, i use cpufreq-switching-2-b and... Full Crash and reboot automatic after less than 15s...

 

After reboot not crash second try just after reboot...

 

for moment, my test pattern not crash, cpufreq-switching-2-b continu to do crash with cpufreq-switching-2-b.

 

I test 2 version here soon:

https://gist.github.com/prahal

Edited April 22 by BipBip1981

[prahal]

14 hours ago, BipBip1981 said:

After reboot not crash second try just after reboot...

@BipBip1981I don't understand what was not crashing on second try just after reboot.

Still thank you for running my test case v2 (and again sorry for pasting you the v1 which was not the correct one to reproduce the crash at first).

It is expected for the v2 to crash the board quite fast. Even if it survives a run you should test a few runs (at least 5).

 

 

Next, we need to find someone to look into the schematics to find out if upping the voltage is the best course of action. If so ship the upped opp voltages into helios64 dts.

[BipBip1981]

hi,

i build 2 version from your site:

root@helios64:~/tmp# ls
cpufreq-switching-2-b  cpufreq-switching-2.c  cpufreq-switching.b  cpufreq-switching.c

I run this version cpufreq-switching-2-b ? and after running your program, i think automatic freq change don't change but stay a last freq fixe, i all right?

[BipBip1981]

Hi,

 

Fisrt, i restore my backup with 6.6.16 and run cpufreq-switching-2-b 5 times and cpufreq-switching-b 2 times

root@helios64:~# uname -a Linux helios64 6.6.16-current-rockchip64 #1 SMP PREEMPT Fri Feb 23 08:25:28 UTC 2024 aarch64 GNU/Linux root@helios64:~# cat /etc/default/cpufrequtils ENABLE=true MIN_SPEED=408000 MAX_SPEED=1800000 GOVERNOR=schedutil
root@helios64:~/tmp# ./cpufreq-switching-2-b ; ./cpufreq-switching-2-b ; ./cpufreq-switching-2-b ; ./cpufreq-switching-2-b ; ./cpufreq-switching-2-b
root@helios64:~/tmp# ./cpufreq-switching-b ; ./cpufreq-switching-b

 

No crash with your program

 

Second, I installed build my self this over my 24.2.1 build with 6.6.28:
linux-dtb-current-rockchip64_24.5.0-trunk_arm64__6.6.28-Sba15-D440d-P9dc2-Ca832H66b1-HK01ba-Vc222-Bf15a-R448a.deb
linux-image-current-rockchip64_24.5.0-trunk_arm64__6.6.28-Sba15-D440d-P9dc2-Ca832H66b1-HK01ba-Vc222-Bf15a-R448a.deb
reboot
and do:
root@helios64:~/tmp# uname -a
Linux helios64 6.6.28-current-rockchip64 #3 SMP PREEMPT Wed Apr 17 09:19:38 UTC 2024 aarch64 GNU/Linux
root@helios64:~/tmp# ./cpufreq-switching-2-b ; ./cpufreq-switching-2-b ; ./cpufreq-switching-2-b ; ./cpufreq-switching-2-b ; ./cpufreq-switching-2-b ; ./cpufreq-switching.b ; ./cpufreq-switching.b

Not crash… freq seem change in htop program monitor but load isn’t at 100% on cpu… load seem idle…

it’s normal, same with 6.6.16 or 6.6.28?
if it’s normal:
not crash with your  cpufreq-switching-2-b & cpufreq-switching.b program on my helios
crash with my pattern test on 6.6.16 and not with 6.6.27 and 6.6.28

What do you think about my results ?

Edited April 22 by BipBip1981

[prahal]

@BipBip1981you mean you have no crash running cpufreq-switching-2-b five times with 6.6.16 and 6.6.28?

No that is not what I expected especially since you told me the first time it crashed and rebooted?

20 hours ago, BipBip1981 said:

Hi, i use cpufreq-switching-2-b and... Full Crash and reboot automatic after less than 15s...

So it crashed with which kernel? Did you rebuild cpufreq-switching-2-b between the test that crashed and the ones that did not?

 

You can run the test in a loop 100 times with:

for i in $(seq 1 100); do ./cpufreqswitching/cpufreq-switching-2-b;  done

With only one opp not upped 75mV I have seen tests crash only after 80 runs but without any changes, it seems unlikely.

Could you paste the 10 last lines from a cpufreq-switching-2-b  run?

 

I could think that any boards have defective components ... but then why did your board crash once and then no more?

 

 

By the way, do not compile cpufreq-switching.c, as I told you previously it was not the correct code for the test case. Has no use for the issue at stake.

It was a first attempt because Kobol team told the crash could be due to too fast frequency switching, so I tried the extremes only. But it turns out these are the most stable and likely the only ones that survive without upping the opp voltage by 75 mV.

 

 

4 hours ago, BipBip1981 said:

after running your program, i think automatic freq change don't change but stay a last freq fixe, i all right?

Yes, because it set the governor to userspace to be able to force switch the frequency via code.

After the run, it does not restore the cpufreq-utils governor (/etc/default/cpufrequtils)

"systemctl restart cpufrequtils.service" should restore it for you.

 

 

 

You mean cpufreq-switching-2-b that output thousands of lines for each run,

built with "gcc -o cpufreq-switching-2-b cpufreq-switching-2.c", from:

and https://gist.github.com/prahal/8fab73325eb0d7091ad7c4627bf8e25a

 

(note there is a small diff between the two, the "usleep(50);" which should not matter).

[BipBip1981]

you mean you have no crash running cpufreq-switching-2-b five times with 6.6.16 and 6.6.28? Yes not crash today.

I post soon full test pattern and result with 6.6.16 and then 6.6.28 with source of cpufreq-switching-2-b from https://gist.github.com/prahal/8fab73325eb0d7091ad7c4627bf8e25a compil with gcc -o cpufreq-switching-2-b cpufreq-switching-2.c

 

So it crashed with which kernel? Did you rebuild cpufreq-switching-2-b between the test that crashed and the ones that did not? Yes i rebuild because i build in /tmp directory that what a ramdisk... and lost binary with crash

 

Now i store binary on emmc 😉

 

I do next test with my armbian 24.2.1 build my self with official framework without any another tune then i do exact same with packages kernel armbian 24.5 build my self with official framework upgrade on my 24.2.1

Edited April 22 by BipBip1981

[BipBip1981]

Actually, i did this below and run your command, result at the end.

For information, view freq change with htop command but not load up more than few %

 

root@helios64:~/tmp# cat cpufreq-switching-2.c
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <malloc.h>
#include <unistd.h>
#include <sys/mman.h>

#define MAIN_LOOPS (100)
#define TRIALS_PER_TOGGLE (10)

#define MAX_MEGS (64)

#define CPUL 0
#define CPUB 1

#define TRANSITION_DELAY 5000

const char *cpul_freqs[] = {
    "408000",
    "600000",
    "816000",
     "1008000",
         "1200000",
         "1416000"
};

const char *cpub_freqs[] = {
    "408000",
    "600000",
    "816000",
     "1008000",
         "1200000",
         "1416000",
         "1608000",
         "1800000"
};

uint32_t *megs[MAX_MEGS];

int checked_open(char *name) {
    int fd = open(name, O_RDWR);
    char err[128];
    if (fd < 0) {
        snprintf(err, 128, "cannot open %s", name);
        perror(err);
        exit(1);
    }
    return fd;
}

#define SCALING_PATHL "/sys/devices/system/cpu/cpu0/cpufreq/"
#define SCALING_PATHB "/sys/devices/system/cpu/cpu4/cpufreq/"

void browse_freq(int *cpul_index, int *cpub_index, int *cpul_step, int *cpub_step) {
    static int inited = 0;
    int freql_target_len;
    int freqb_target_len;
    int freqfd;
    int cpul_freqs_count = 0;
    int cpub_freqs_count = 0;

    cpul_freqs_count = sizeof(cpul_freqs)/sizeof(cpul_freqs[0]);
    cpub_freqs_count = sizeof(cpub_freqs)/sizeof(cpub_freqs[0]);

    if (!inited) {
#if CPUL
        freqfd = checked_open(SCALING_PATHL "scaling_governor");
        write(freqfd, "userspace", 9);
        close(freqfd);
#endif
#if CPUB
        freqfd = checked_open(SCALING_PATHB "scaling_governor");
        write(freqfd, "userspace", 9);
        close(freqfd);
#endif
        inited = 1;
    }

    if (*cpul_index >= cpul_freqs_count - 1)
        *cpul_step = -1;
    if (*cpul_index <= 0)
        *cpul_step = 1;

    if (*cpub_index >= cpub_freqs_count - 1)
        *cpub_step = -1;
    if (*cpub_index <= 0)
        *cpub_step = 1;

    *cpul_index += *cpul_step;
    *cpub_index += *cpub_step;
#if CPUL
    printf("cpul_freq %s\n", cpul_freqs[*cpul_index]);
    freql_target_len = strlen(cpul_freqs[*cpul_index]);
    freqfd = checked_open(SCALING_PATHL "scaling_setspeed");
    write(freqfd, cpul_freqs[*cpul_index], freql_target_len);
    close(freqfd);
#endif
#if CPUB
    printf("cpub_freq %s\n", cpub_freqs[*cpub_index]);
    freqb_target_len = strlen(cpub_freqs[*cpub_index]);
    freqfd = checked_open(SCALING_PATHB "scaling_setspeed");
    write(freqfd, cpub_freqs[*cpub_index], freqb_target_len);
    close(freqfd);
#endif
    usleep(TRANSITION_DELAY);
}

void write_test_data(int nmegs, int toggle) {
    int cpul_index = 0;
    int cpub_index = 0;
    int cpul_step = 1;
    int cpub_step = 1;
    while (nmegs--) {
        browse_freq(&cpul_index, &cpub_index, &cpul_step, &cpub_step);
    }
}
void check_test_data(int nmegs, int toggle) {
    int cpul_index = 0;
    int cpub_index = 0;
    int cpul_step = 1;
    int cpub_step = 1;
    while (nmegs--) {
        browse_freq(&cpul_index, &cpub_index, &cpul_step, &cpub_step);
    }
}

int main(int argc, char **argv) {
    int nmegs = MAX_MEGS;
    printf("allocated %dMB\n", nmegs);

    int nloop, ntoggle, ntrial;

    printf("test: toggle freq before write\n");
    for (nloop = 0; nloop < MAIN_LOOPS; nloop++) {
        printf("\r%d/%d  ", nloop, MAIN_LOOPS);
        fflush(stdout);

        write_test_data(nmegs, 1);
        check_test_data(nmegs, 0);
    }
    printf("\n");

    printf("test: toggle freq before read\n");
    for (nloop = 0; nloop < MAIN_LOOPS; nloop++) {
        write_test_data(nmegs, 0);
        for (ntrial=0; ntrial < TRIALS_PER_TOGGLE; ntrial++) {
            printf("\r%d/%d, %d/%d  ", ntrial, TRIALS_PER_TOGGLE, nloop, MAIN_LOOPS);
            fflush(stdout);

            check_test_data(nmegs, 1);
        }
    }
    printf("\n");

    return 0;
}

—————————————————————————————————————————————————————————————————————————————————————

root@helios64:~/tmp# gcc -o cpufreq-switching-2-b cpufreq-switching-2.c

—————————————————————————————————————————————————————————————————————————————————————

root@helios64:~/tmp# uname -a
Linux helios64 6.6.16-current-rockchip64 #1 SMP PREEMPT Fri Feb 23 08:25:28 UTC 2024 aarch64 GNU/Linux

—————————————————————————————————————————————————————————————————————————————————————

root@helios64:~/tmp# cat /etc/default/cpufrequtils
ENABLE=true
MIN_SPEED=408000
MAX_SPEED=1800000
GOVERNOR=schedutil

—————————————————————————————————————————————————————————————————————————————————————

root@helios64:~/tmp# uptime

 19:39:19 up 5 min,  2 users,  load average: 0.27, 0.54, 0.31

—————————————————————————————————————————————————————————————————————————————————————

root@helios64:~/tmp# for i in $(seq 1 100); do ./cpufreq-switching-2-b ;  done

Edited April 22 by BipBip1981

[prahal]

4 hours ago, BipBip1981 said:

root@helios64:~/tmp# for i in $(seq 1 100); do ./cpufreq-switching-2-b ;  done

this should take time, how long did it take to complete?

Could you paste the last 10 lines of output from the command (or even a single run)?

And maybe run the test with "time for i in $(seq 1 100); do ./cpufreq-switching-2-b ;  done" to get the time it took at the end (but if it took ages it ran fine, it is not required to run the 100 iterations anew).

 

It might be that the test runs fine on your hardware. That would be interesting. But as I said as it crashed once I doubted it.

 

One option is that in one one the first attempt you tried:

and on the second you tried https://gist.github.com/prahal/8fab73325eb0d7091ad7c4627bf8e25a

which has a delay between cpub frequency transition of 5 milliseconds while the first has no delay at this point.

(again sorry I did not notice the gist github one had this 5msec delay I added to test if a delay would help.

To check you can replace:

"#define TRANSITION_DELAY 5000"

by

"#define TRANSITION_DELAY 0"

and check if it crashes. Then it will point to an issue with the delay between switching operating points for the big CPU.

 

Do you know which kernel was running when your box crashed?

 

 

Also, do you know which u-boot you have? (requires serial console output)

 

Mind you don't need to paste cpufrequtils data because the test case bypasses the cpufrequtils settings and manages the chosen frequencies and how to switch them on its own.

 

Note: if you want to quote a text from this forum, select it with your mouse, a popover box will appear above the selection "Quote selection", click on it. You can quote more than one selection to the same post.

Edited April 22 by prahal

[prahal]

I wonder if upping the voltage was the correct fix (and if it would always work).

 

From other rk3399 board schematics and TCS4525 datasheet ... it seems Kobol team designed the board for the TCS4525 regulator used before CPU BIG in a lot of designs and replaced the TCS4525 with the SYR837 later on (without taking into account the different recommendations for the SYR837 ... ie VIN with a 22uF instead of 10uF for the TCS4525).

All the components around the SYR837 on the helios64 datasheet match the reference design for the TCS4525 (from Torch Chip, datasheet behind Chinese paywalls). I don't know if replacing the VIN capacitor would be enough to get stable big CPUs...

[BipBip1981]

Hello,

I crash this night (between 3hours uptime and 6hours uptime, i was sleeping) with 6.6.16 and without voltage correction. (red led flashing)

(I crash with version "TRANSITION_DELAY 5000" that i copy/paste upper)

 

Do you know which kernel was running when your box crashed? -> 6.6.28 but maybe with your TRANSITION_DELAY 0 version because i copy/paste from forum and then until now i use github version copy/paste upper

 

I install 6.6.28 without voltage correction and look result

 

List of packages i installed:

base-files_24.5.0-trunk-12.4+deb12u5-bookworm_arm64__1-bookworm-1armbian1-Ba4f9-U12.4--deb12u5-R55fa.deb

armbian-config_24.5.0-trunk_all__1-SA8477-B0293-R448a.deb                                 

linux-dtb-current-rockchip64_24.5.0-trunk_arm64__6.6.28-Sba15-D440d-P9dc2-Ca832H66b1-HK01ba-Vc222-Bf15a-R448a.deb
armbian-plymouth-theme_24.5.0-trunk_all__1-Bfcdb-R448a.deb                                

linux-image-current-rockchip64_24.5.0-trunk_arm64__6.6.28-Sba15-D440d-P9dc2-Ca832H66b1-HK01ba-Vc222-Bf15a-R448a.deb
armbian-zsh_24.5.0-trunk_all__1-SAbfee-Baafd-R448a.deb                                    

linux-u-boot-helios64-current_24.5.0-trunk_arm64__2022.07-Se092-Pe990-H8c72-V65aa-Bf5f6-R448a.deb

 

root@helios64:~/tmp# uname -a
Linux helios64 6.6.28-current-rockchip64 #3 SMP PREEMPT Wed Apr 17 09:19:38 UTC 2024 aarch64 GNU/Linux

 

root@helios64:~/tmp# for i in $(seq 1 100); do ./cpufreq-switching-2-b ;  done

 

If not crash, i try TRANSITION_DELAY 0 version

if not crash  TRANSITION_DELAY 0 version, i stay with this setting

If crash i try voltage correction

 

Keep in touch

Edited April 23 by BipBip1981

[BipBip1981]

Result with for i in $(seq 1 100); do ./cpufreq-switching-2-b ;  done

TRANSITION_DELAY 5000

16h47 uptime

Kernel 6.6.28

NO CRASH !

Tomorrow try with TRANSITION_DELAY 0

Have a good night

 

9/10, 99/100  cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000
cpub_freq 1416000
cpub_freq 1200000
cpub_freq 1008000
cpub_freq 816000
cpub_freq 600000
cpub_freq 408000
cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000
cpub_freq 1416000
cpub_freq 1200000
cpub_freq 1008000
cpub_freq 816000
cpub_freq 600000
cpub_freq 408000
cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000
cpub_freq 1416000
cpub_freq 1200000
cpub_freq 1008000
cpub_freq 816000
cpub_freq 600000
cpub_freq 408000
cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000
cpub_freq 1416000
cpub_freq 1200000
cpub_freq 1008000
cpub_freq 816000
cpub_freq 600000
cpub_freq 408000
cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000

root@helios64:~/tmp#

Edited April 24 by BipBip1981

[BipBip1981]

Hi,

 

Today:

 

KERNEL 6.6.28

No voltage modification

 

root@helios64:~/tmp# diff cpufreq-switching-2.c cpufreq-switching-2-TRANSITION_DELAY_0.c
19c19
< #define TRANSITION_DELAY 5000
---
> #define TRANSITION_DELAY 0

 

root@helios64:~/tmp# gcc -o  cpufreq-switching-2-TRANSITION_DELAY_0.b cpufreq-switching-2-TRANSITION_DELAY_0.c

root@helios64:~/tmp# time for i in $(seq 1 100); do ./cpufreq-switching-2-TRANSITION_DELAY_0.b ;  done

 

Few minutes... less than 5min... and crash in "full reboot" seem like short off/on with same sound like push reset button

 

Second try, uptime 19min and same crash like push reset buttom

 

First conclusion: Time delay 5000 seem Okok / Time delay 0 crash

 

Keep in touch

 

Edited April 24 by BipBip1981

[snakekick]

Kernel 5.15.93 (get not "stable" with 6x kernel)

linux-u-boot-current-helios64_21.08.9_arm64.deb

 

Uptime 20 Days (but kernel panic after ±30 Days)

CPU Speed:

MIN_SPEED=600000                                                                

MAX_SPEED=1200000                                                               

GOVERNOR=ondemand  

 

Systemusage:

 

Fileserver (Mergerfs/Snapraid with 5 SSD+1USB HDD with daily diff/scrub/sync )

Webserver 

1 Docker Container

 

./cpufreq-switching-2-b_delay_5000

 

allocated 64MB

test: toggle freq before write

0/100  cpub_freq 600000

cpub_freq 816000

cpub_freq 1008000

cpub_freq 1200000

cpub_freq 1416000

cpub_freq 1608000

cpub_freq 1800000

cpub_freq 1608000

cpub_freq 1416000

cpub_freq 1200000

cpub_freq 1008000

cpub_freq 816000

cpub_freq 600000

cpub_freq 408000

cpub_freq 600000

cpub_freq 816000

cpub_freq 1008000

 

kernelpanic 

[prahal]

@snakekick Thanks, that seems to confirm my findings back a few months ago. Adding a 5ms delay in the test case did not prevent the crash.

Though it could be the system load is at play. Maybe adding a delay at the kernel level would do.

pcie is tagged on the big CPUs so the SATA disks seem to matter (as the ethernet port).

One could try in emergency mode (passing emergency to the kernel (I do it by "setenv extraboardargs emergency" after halting u-boot with a key press then enter "boot"). You will have just the root partition mounted read-only (so no network connection, a serial console is required). Then run the test.

 

Also note that the design of the GPU regulator has the same issue as the CPU b one ... (for my tests I blacklisted panfrost, ie the GPU driver).

 

After looking at the rockchip64 board schematics the design around the CPU b regulator is not similar but exactly the same as the helios64 one (rockchip64 uses a tcs4545 regulator for cpu b and tcs4546 for GPU).

I wonder if the easiest fix would not be to pay someone to desolder the syr837 regulator and solder a tcs4545 instead - same for the GPU regulator a tcs4546 instead of the syr838... except that these chips from Torch Chip seem nowhere to be found. Maybe rip them from a rockpro64 board.

 

@aprayoga can you confirm the Helios64 design for the rk3399 big cpu and gpu regulators are the same as the  RockPro64 ones? Would it make sense (and would it fix the unstable cpu_b) to desolder the syr837/syr838 to replace them with tcs4545/tcs4546?

Ie the tcs4535 datasheet (I am still unable to find the tcs4545 datasheet) I found tells tcs425 has internal pulldown for VSEL and EN which syr837 does not, the syr837 datasheet requires a 22uF capacitor for VIN but the helios64 has a 10uF one like the rockpro64 for the tcs4545. The SW pin of the helios64 has 470uH inductor with 4 x 22uF capacitors like the rockpro64 for the tcs4545 (like the typical application in the tcs4535 datasheet with 470uH inductor with two 22uF capacitors)?

Do you know a replacement for the TCS4545/TCS4546 that has closer specs than the syr837/syr838?

I cannot seem to find TCS4545/TCS4546 for sale (maybe I could buy a rockproc64 to desolder them at least for a test... or could you check on your side with a helios64 board that the cpufreq-switching-2-b test above crash with syr837 but not with tcs4545 with vanilla rk3399 opp definitions in dts?

 

 

Sadly the Helios64 filled a market that is left unfilled. People who do not have the know-how to go full low-wattage DIY NAS and who also cannot afford to pay 1K€ for a NAS (and who might need two NASs to make things worse).

In the meantime, I spend a lot of time learning about DIY NAS, but it is still hard to get wattage at full load (they tend to give all idle power usage). I probably will end up gambling and buying one build and pray... but with Helios64 I had the metrics before buying.

 

 

I found that the Rock960 has the same design for the cpu_b and gpu regulators except for the inductor which is 0.240uH on the rock960 and 0.470uH on the Helios64.

But hard to tell if the Rock960 is stable with my cpufreq switching test for the big cpus of the rk3399, might be the use of the board does not stress it as much as a raid10 on the helios64 pcie sata which is tagged to the cpu_b ... (initially it was 4 3TB WD Red - the old CMR model WD30EFRX-68EUZN0), from Helios4 setup as advertised by Kobol wiki for the Helios4... the board crashes on first boot after assembly with this raid setup.

 

Mind I found that the Pinebook Pro also has the same design as the Helios64 this time around the syr837/syr838 ... I begin to wonder if either they are all broken (could be the amount of stress of a NAS ethernet or raid10 pcie is not that common) or if this is not the issue at stake.

Edited April 25 by prahal
Add finding about Pinebook Pro being similar to helios64

[BipBip1981]

Hi,

To my side, it's too complex and risky for me to change hardware components on Helios64.

I prefer for my case of uses change CPU Freq and Governor for less performance but stable inside take risk of damage my board by change components.

Have a good day

[Trillien]

I've just tested cpufreq-switching-2-delay5msec in emergency mode.

My setup is Armbian 23.08.0-trunk Bookworm with Linux 6.6.8-edge-rockchip64

 

My first attempt ran for few minutes before freezing and rebooting.

 

Before my second attempt, I blacklist panfrost (not sure there is an impact at all in emergency mode...)

$ sudo nano /etc/modprobe.d/blacklist.conf
blacklist panfrost

I then rebooted and started linux in emergency mode.

# lsmod | grep panfrost

lsmod didn't catch any panfrost module.

 

Still, cpufreq-switching-2-delay5msec ended with a kernel panic:

10/100  cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000
cpub_freq 1416000
cpub_freq 1200000
cpub_freq 1008000
cpub_freq 816000
cpub_freq 600000
cpub_freq 408000
cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000
cpub_freq 1416000
cpub_freq 1200000
cpub_freq 1008000
cpub_freq 816000
cpub_freq 600000
cpub_freq 408000
cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000
cpub_freq 1416000
cpub_freq 1200000
cpub_freq 1008000
cpub_freq 816000
cpub_freq 600000
cpub_freq 408000
cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000
cpub_freq 1416000
cpub_freq 1200000
cpub_freq 1008000
cpub_freq 816000
cpub_freq 600000
cpub_freq 408000
cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000
cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000
cpub_freq 1416000
cpub_freq 1200000
cpub_freq 1008000
cpub_freq 816000
cpub_freq 600000
cpub_freq 408000
cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
cpub_freq 1200000
cpub_freq 1416000
cpub_freq 1608000
cpub_freq 1800000
cpub_freq 1608000
cpub_freq 1416000
cpub_freq 1200000
cpub_freq 1008000
cpub_freq 816000
cpub_freq 600000
cpub_freq 408000
cpub_freq 600000
cpub_freq 816000
cpub_freq 1008000
[   51.732314] Internal error: Oops: 0000000096000006 [#1] PREEMPT SMP
[   51.732887] Modules linked in: ip_tables x_tables autofs4 efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid0 multipath linear cdc_ncm cdc_ether usbnet raid1 r8152 realtek rockchipdrm dw_mipi_dsi dw_hdmi analogix_dp fusb302 drm_display_helper cec tcpm drm_dma_helper typec drm_kms_helper dwmac_rk stmmac_platform stmmac drm pcs_xpcs adc_keys
[   51.735962] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 6.6.8-edge-rockchip64 #1
[   51.736610] Hardware name: Helios64 (DT)
[   51.736965] pstate: 800000c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   51.737589] pc : update_blocked_averages+0x278/0x758
[   51.738051] lr : update_blocked_averages+0x264/0x758
[   51.738504] sp : ffff800082043e80
[   51.738806] x29: ffff800082043e80 x28: ffff000005cd9600 x27: 0000000c01b0a9aa
[   51.739456] x26: ffff000005cdbc80 x25: ffff000005cdbc00 x24: 0000000000000000
[   51.740104] x23: ffff0000f77a0f18 x22: 0000000000000028 x21: 0000000000000000
[   51.740750] x20: ffff000005cdbd40 x19: ffff0000f77a05c0 x18: 0000000000000000
[   51.741396] x17: ffff800075e85000 x16: ffff800082040000 x15: 0000000000000000
[   51.742043] x14: 0000000000000001 x13: 000000000000002a x12: 00000000000e7fe0
[   51.742689] x11: 0000000000000000 x10: 000000000000002a x9 : ffff000005cdbc80
[   51.743336] x8 : 0000000000000000 x7 : ffff000005cdbc00 x6 : 0000000000000014
[   51.743982] x5 : 00000000000003af x4 : 000000000000b6a8 x3 : 0000000000000000
[   51.744628] x2 : 0000000000000000 x1 : ffff000001d6c400 x0 : ffff0000019c6000
[   51.745274] Call trace:
[   51.745501]  update_blocked_averages+0x278/0x758
[   51.745924]  run_rebalance_domains+0x4c/0x80
[   51.746313]  __do_softirq+0x160/0x3fc
[   51.746649]  ____do_softirq+0x10/0x1c
[   51.746985]  call_on_irq_stack+0x24/0x4c
[   51.747344]  do_softirq_own_stack+0x1c/0x2c
[   51.747727]  irq_exit_rcu+0x94/0xd0
[   51.748051]  el1_interrupt+0x38/0x68
[   51.748382]  el1h_64_irq_handler+0x18/0x24
[   51.748758]  el1h_64_irq+0x64/0x68
[   51.749070]  cpuidle_enter_state+0xc0/0x4bc
[   51.749454]  cpuidle_enter+0x38/0x50
[   51.749783]  do_idle+0x1fc/0x270
[   51.750083]  cpu_startup_entry+0x34/0x3c
[   51.750444]  secondary_start_kernel+0x128/0x148
[   51.750859]  __secondary_switched+0xb8/0xbc
[   51.751249] Code: f940ab20 f9406400 f8766801 b4000101 (f9407020)
[   51.751797] ---[ end trace 0000000000000000 ]---
[   51.752214] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[   51.752826] SMP: stopping secondary CPUs
[   51.753358] Kernel Offset: disabled
[   51.753674] CPU features: 0x1,00000208,3c020000,1000421b
[   51.754152] Memory Limit: none
[   51.754439] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

 

[prahal]

On 4/26/2024 at 11:41 AM, BipBip1981 said:

To my side, it's too complex and risky for me to change hardware components on Helios64.

@BipBip1981I agree and I did not plan on doing it on my own. But phone repair shops have skilled technicians who can do it.

 

Still, the need to replace a hardware component is a wild guess. At this point, I was merely saying that I was ready to test a hardware change on my board to find out if the problem was a hardware issue.

In the end, I believe that if we better understand what is wrong, be it the hardware, we might even be able to work around such a hardware shortcoming in software.

I would not suggest messing with the hardware to test if it works better except if you are ready to lose the board. But mine is so unstable (probably due to my raid10 setup inherited from the helios4) that I could barely use it for years. So it is a matter of either testing if I can get it stable or buying a new NAS and sending this helios64 to the trash.

 

I hope to be able to tell you a good governor/frequency but I need to test more. At least the most reliable frequencies without voltage quirks for the big cpu seemed to be the lowest 408000 and the highest 1800000. So you might want to force the "userspace" governor and "1800000" as a frequency.

 

[prahal]

@Trillien thanks, that confirms that I am not alone with a setup that does crash even with 5 milliseconds delay 🙂

If time permits could you try with the TRANSITION_DELAY value increased 10 times in the test case code (to 50millisecs, ie 50000) then 100 times to 500000?

[ebin-dev]

On 4/28/2024 at 3:48 AM, prahal said:

I hope to be able to tell you a good governor/frequency but I need to test more. At least the most reliable frequencies without voltage quirks for the big cpu seemed to be the lowest 408000 and the highest 1800000. So you might want to force the "userspace" governor and "1800000" as a frequency.

 

When looking at the cpu frequency transition tables it would appear there are essentially only a few frequencies involved (little cores: 600 <-> 408 <-> 1416 MHz, big cores 816 <-> 1800 MHz)

 

# transition tables, linux 6.6.8

# cat /sys/devices/system/cpu/cpufreq/policy4/stats/trans_table
   From  :    To
         :    408000    600000    816000   1008000   1200000   1416000   1608000   1800000 
   408000:         0         0         0         0         0         0         0         0 
   600000:         0         0         0         0         0         0         0         1 
   816000:         0         0         0         9        12         7         2       262 
  1008000:         0         0        17         0         1         1         3         1 
  1200000:         0         0         9         2         0         5         5         4 
  1416000:         0         0         5         3         6         0        13        12 
  1608000:         0         0         4         4         2        13         0        12 
  1800000:         0         0       258         5         4        13        12         0 

# cat /sys/devices/system/cpu/cpufreq/policy0/stats/trans_table
   From  :    To
         :    408000    600000    816000   1008000   1200000   1416000 
   408000:         0      2859         4         8         6      1572 
   600000:      2780         0         5         4         3       104 
   816000:        14         4         0         7         3         1 
  1008000:         3         2         5         0        11        14 
  1200000:         2         2         3         6         0        40 
  1416000:      1651        28        12        10        30         0 

 

Edit: That behavior is specific for kernel 6.6.8. Testing 6.6.29 atm: I have noticed that it uses different frequencies: preferably switching between the lowest and the highest frequency available for each core.

Edited April 29 by ebin-dev

[ebin-dev]

On 4/28/2024 at 3:48 AM, prahal said:

I hope to be able to tell you a good governor/frequency but I need to test more. At least the most reliable frequencies without voltage quirks for the big cpu seemed to be the lowest 408000 and the highest 1800000. So you might want to force the "userspace" governor and "1800000" as a frequency.

 

Current kernel 6.6.29 uses the lowest frequencies 400 and 600 MHz also for the big cores, contrary to linux 6.6.8, where the lowest frequency used by the big cores is 800 MHz. So I would assume from what you say that linux 6.6.29 should be more stable.

 

However, I was puzzled why this is not the case with my system: actually none of the kernels 6.6.x (x>8) was stable on my system until I tried 6.6.29 with the 75mV bump for all states of the big cores !

 

The explanation I have for that is that the jump from the lowest state 408  to 1.8 GHz is somewhat larger than from 800 MHz and that the correct voltage feeded to the cores becomes more critical in those cases. 

So with the 75 mV bump for all states of the big cores, that source of instability seems to have vanished now: 6.6.29 is stable so far (at least on my system).

 

# cat /etc/default/cpufrequtils 
ENABLE=true
MIN_SPEED=408000
MAX_SPEED=1800000
GOVERNOR=ondemand

 

# transition tables, linux 6.6.29

# cat /sys/devices/system/cpu/cpufreq/policy4/stats/trans_table
   From  :    To
         :    408000    600000    816000   1008000   1200000   1416000   1608000   1800000 
   408000:         0      1705       141         1         5         1         1       394 
   600000:      1695         0         7         2         0         3         0       120 
   816000:       137         8         0         2         3         3         2        21 
  1008000:         2         1         5         0         0         0         0         1 
  1200000:         5         1         2         0         0         2         3         4 
  1416000:         2         0         3         0         3         0         3         4 
  1608000:         1         0         0         1         1         3         0         5 
  1800000:       407       111        18         3         5         3         2         0 

# cat /sys/devices/system/cpu/cpufreq/policy0/stats/trans_table
   From  :    To
         :    408000    600000    816000   1008000   1200000   1416000 
   408000:         0      2951         7         3         2      1547 
   600000:      2865         0         2         2         2       139 
   816000:         8         2         0         1         1         7 
  1008000:         4         0         1         0         4         4 
  1200000:         1         1         1         1         0        21 
  1416000:      1633        55         8         6        16         0 

 

Edited April 29 by ebin-dev

[BipBip1981]

Hi,

Kernel 6.6.29 with:

git clone --depth=1 --branch=main https://github.com/armbian/build

cd build

./compile.sh kernel BOARD=helios64 BRANCH=current

Same for me like with 6.6.26/27/28, pass my pattern test and also rsync over network of 3TO with -c option to do checksum during RAID10 Scrubbing pass

I think Governor on this Kernel is less agressive like before...

Keep in touch

Edited April 29 by BipBip1981

[BipBip1981]

Hello... and i'm crying... freeze today with raid10 srubbing during btrfs check...

But something is strange... network ssh is dead... but access with usb cable to console is working but very very very extremely slow

I have this and... waiting very long time to have prompt

 

picocom v3.1

port is        : /dev/ttyUSB0
flowcontrol    : none
baudrate is    : 1500000
parity is      : none
databits are   : 8
stopbits are   : 1
escape is      : C-a
local echo is  : no
noinit is      : no
noreset is     : no
hangup is      : no
nolock is      : no
send_cmd is    : sz -vv
receive_cmd is : rz -vv -E
imap is        :
omap is        :
emap is        : crcrlf,delbs,
logfile is     : none
initstring     : none
exit_after is  : not set
exit is        : no

Type [C-a] [C-h] to see available commands
Terminal ready

helios64 login: root
Password:
 _   _      _ _            __   _  _   
| | | | ___| (_) ___  ___ / /_ | || |  
| |_| |/ _ \ | |/ _ \/ __| '_ \| || |_
|  _  |  __/ | | (_) \__ \ (_) |__   _|
|_| |_|\___|_|_|\___/|___/\___/   |_|  
                                       
Welcome to Armbian-unofficial 24.2.1 Bookworm with Linux 6.6.29-current-rockchip64

No end-user support: built from trunk

System load:   183%               Up time:       1 day 21:18    Local users:   3                
Memory usage:  84% of 3.77G      Zram usage:    34% of 5.88G      IP:           10.88.0.1 10.0.0.155
CPU temp:      43°C               Usage of /:    46% of 14G        
RX today:      52.5 MiB      

[ General system configuration (beta): armbian-config ]

Web console: https://helios64:9090/

You have no mail.
root@helios64:~#

 

Edited April 30 by BipBip1981

[ebin-dev]

3 minutes ago, BipBip1981 said:

Hello... and i'm crying... freeze today with raid10 srubbing during btrfs chek with checksum...

 

Did you increase opp voltages by 75mV for the fast cores in this test ? (dtb attached)

rk3399-kobol-helios64.dtb-6.6.29-L2-hs400-opp

[BipBip1981]

Hi ebin-dev, i not crash, console usb is ready, HDD activity seem to be died or very very slow, i try to find why lost ssh access and scrubbing and btrfs stop and helios is terribly slow, keep in touch

 

@ebin-dev: where i copy/paste your file?

Edited April 30 by BipBip1981

[BipBip1981]

Sorry for the big copy paste but maybe that help us, i control again helios, i have a prompt,  network is fully down or crash, lost ip and TX/RX, dmesg command last interresting lines are this:

 

[118243.950487] BTRFS info (device dm-4): last unmount of filesystem 1d4e2c84-1c43-4d73-8acb-14d5a7aa1c4d
[143643.328285] hrtimer: interrupt took 33543 ns
[147029.554073] md: md0: data-check done.
[155615.761768] md: data-check of RAID array md0
[161153.336701] rcu: INFO: rcu_preempt self-detected stall on CPU
[161153.336720] rcu:     5-...0: (27185355 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=7499
[161153.336745] rcu:     (t=15000 jiffies g=12239613 q=1933 ncpus=6)
[161153.336761] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[161153.336776] Hardware name: Helios64 (DT)
[161153.336784] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[161153.336799] pc : cpuidle_enter_state+0xc0/0x4bc
[161153.336819] lr : cpuidle_enter_state+0xb8/0x4bc
[161153.336830] sp : ffff800081f23d90
[161153.336838] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[161153.336868] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[161153.336894] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[161153.336919] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[161153.336944] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[161153.336969] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[161153.336993] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[161153.337018] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[161153.337041] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[161153.337066] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[161153.337090] Call trace:
[161153.337098]  cpuidle_enter_state+0xc0/0x4bc
[161153.337111]  cpuidle_enter+0x38/0x50
[161153.337125]  do_idle+0x1fc/0x270
[161153.337143]  cpu_startup_entry+0x34/0x3c
[161153.337157]  secondary_start_kernel+0x12c/0x14c
[161153.337177]  __secondary_switched+0xb8/0xbc
[161333.343804] rcu: INFO: rcu_preempt self-detected stall on CPU
[161333.343828] rcu:     5-...0: (108721819 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=30001
[161333.343862] rcu:     (t=60003 jiffies g=12239613 q=8530 ncpus=6)
[161333.343883] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[161333.343902] Hardware name: Helios64 (DT)
[161333.343913] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[161333.343932] pc : cpuidle_enter_state+0xc0/0x4bc
[161333.343954] lr : cpuidle_enter_state+0xb8/0x4bc
[161333.343969] sp : ffff800081f23d90
[161333.343979] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[161333.344017] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[161333.344053] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[161333.344089] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[161333.344124] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[161333.344160] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[161333.344195] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[161333.344231] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[161333.344266] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[161333.344300] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[161333.344335] Call trace:
[161333.344346]  cpuidle_enter_state+0xc0/0x4bc
[161333.344362]  cpuidle_enter+0x38/0x50
[161333.344379]  do_idle+0x1fc/0x270
[161333.344400]  cpu_startup_entry+0x34/0x3c
[161333.344420]  secondary_start_kernel+0x12c/0x14c
[161333.344446]  __secondary_switched+0xb8/0xbc
[161513.350913] rcu: INFO: rcu_preempt self-detected stall on CPU
[161513.350930] rcu:     5-...0: (164767240 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=52502
[161513.350962] rcu:     (t=105006 jiffies g=12239613 q=14808 ncpus=6)
[161513.350981] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[161513.351000] Hardware name: Helios64 (DT)
[161513.351011] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[161513.351030] pc : cpuidle_enter_state+0xc0/0x4bc
[161513.351049] lr : cpuidle_enter_state+0xb8/0x4bc
[161513.351064] sp : ffff800081f23d90
[161513.351074] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[161513.351111] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[161513.351146] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[161513.351181] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[161513.351216] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[161513.351252] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[161513.351287] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[161513.351322] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[161513.351357] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[161513.351392] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[161513.351426] Call trace:
[161513.351437]  cpuidle_enter_state+0xc0/0x4bc
[161513.351453]  cpuidle_enter+0x38/0x50
[161513.351471]  do_idle+0x1fc/0x270
[161513.351492]  cpu_startup_entry+0x34/0x3c
[161513.351511]  secondary_start_kernel+0x12c/0x14c
[161513.351537]  __secondary_switched+0xb8/0xbc
[161693.358028] rcu: INFO: rcu_preempt self-detected stall on CPU
[161693.358044] rcu:     5-...0: (220402734 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=75004
[161693.358074] rcu:     (t=150009 jiffies g=12239613 q=21715 ncpus=6)
[161693.358093] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[161693.358111] Hardware name: Helios64 (DT)
[161693.358121] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[161693.358141] pc : cpuidle_enter_state+0xc0/0x4bc
[161693.358160] lr : cpuidle_enter_state+0xb8/0x4bc
[161693.358174] sp : ffff800081f23d90
[161693.358183] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[161693.358220] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[161693.358256] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[161693.358290] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[161693.358325] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[161693.358361] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[161693.358396] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[161693.358431] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[161693.358466] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[161693.358501] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[161693.358535] Call trace:
[161693.358546]  cpuidle_enter_state+0xc0/0x4bc
[161693.358561]  cpuidle_enter+0x38/0x50
[161693.358578]  do_idle+0x1fc/0x270
[161693.358599]  cpu_startup_entry+0x34/0x3c
[161693.358618]  secondary_start_kernel+0x12c/0x14c
[161693.358644]  __secondary_switched+0xb8/0xbc
[161873.365143] rcu: INFO: rcu_preempt self-detected stall on CPU
[161873.365165] rcu:     5-...0: (276037777 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=97505
[161873.365195] rcu:     (t=195012 jiffies g=12239613 q=27612 ncpus=6)
[161873.365214] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[161873.365232] Hardware name: Helios64 (DT)
[161873.365243] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[161873.365261] pc : cpuidle_enter_state+0xc0/0x4bc
[161873.365281] lr : cpuidle_enter_state+0xb8/0x4bc
[161873.365295] sp : ffff800081f23d90
[161873.365306] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[161873.365342] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[161873.365378] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[161873.365413] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[161873.365447] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[161873.365483] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[161873.365518] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[161873.365553] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[161873.365588] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[161873.365623] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[161873.365658] Call trace:
[161873.365668]  cpuidle_enter_state+0xc0/0x4bc
[161873.365684]  cpuidle_enter+0x38/0x50
[161873.365701]  do_idle+0x1fc/0x270
[161873.365721]  cpu_startup_entry+0x34/0x3c
[161873.365740]  secondary_start_kernel+0x12c/0x14c
[161873.365766]  __secondary_switched+0xb8/0xbc
[162053.372255] rcu: INFO: rcu_preempt self-detected stall on CPU
[162053.372271] rcu:     5-...0: (331695191 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=120006
[162053.372299] rcu:     (t=240015 jiffies g=12239613 q=33959 ncpus=6)
[162053.372318] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[162053.372336] Hardware name: Helios64 (DT)
[162053.372347] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[162053.372365] pc : cpuidle_enter_state+0xc0/0x4bc
[162053.372383] lr : cpuidle_enter_state+0xb8/0x4bc
[162053.372397] sp : ffff800081f23d90
[162053.372407] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[162053.372444] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[162053.372479] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[162053.372513] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[162053.372548] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[162053.372584] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[162053.372619] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[162053.372654] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[162053.372689] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[162053.372723] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[162053.372758] Call trace:
[162053.372768]  cpuidle_enter_state+0xc0/0x4bc
[162053.372783]  cpuidle_enter+0x38/0x50
[162053.372800]  do_idle+0x1fc/0x270
[162053.372821]  cpu_startup_entry+0x34/0x3c
[162053.372841]  secondary_start_kernel+0x12c/0x14c
[162053.372866]  __secondary_switched+0xb8/0xbc
[162233.379370] rcu: INFO: rcu_preempt self-detected stall on CPU
[162233.379393] rcu:     5-...0: (388627571 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=142506
[162233.379423] rcu:     (t=285018 jiffies g=12239613 q=40530 ncpus=6)
[162233.379442] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[162233.379460] Hardware name: Helios64 (DT)
[162233.379471] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[162233.379491] pc : cpuidle_enter_state+0xc0/0x4bc
[162233.379512] lr : cpuidle_enter_state+0xb8/0x4bc
[162233.379526] sp : ffff800081f23d90
[162233.379536] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[162233.379572] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[162233.379607] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[162233.379642] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[162233.379678] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[162233.379713] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[162233.379749] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[162233.379784] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[162233.379818] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[162233.379853] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[162233.379887] Call trace:
[162233.379897]  cpuidle_enter_state+0xc0/0x4bc
[162233.379913]  cpuidle_enter+0x38/0x50
[162233.379931]  do_idle+0x1fc/0x270
[162233.379952]  cpu_startup_entry+0x34/0x3c
[162233.379972]  secondary_start_kernel+0x12c/0x14c
[162233.379998]  __secondary_switched+0xb8/0xbc
[162413.386467] rcu: INFO: rcu_preempt self-detected stall on CPU
[162413.386482] rcu:     5-...0: (444291044 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=165008
[162413.386511] rcu:     (t=330021 jiffies g=12239613 q=46628 ncpus=6)
[162413.386529] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[162413.386548] Hardware name: Helios64 (DT)
[162413.386559] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[162413.386577] pc : cpuidle_enter_state+0xc0/0x4bc
[162413.386595] lr : cpuidle_enter_state+0xb8/0x4bc
[162413.386609] sp : ffff800081f23d90
[162413.386619] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[162413.386656] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[162413.386691] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[162413.386726] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[162413.386761] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[162413.386797] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[162413.386832] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[162413.386867] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[162413.386901] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[162413.386936] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[162413.386970] Call trace:
[162413.386981]  cpuidle_enter_state+0xc0/0x4bc
[162413.386996]  cpuidle_enter+0x38/0x50
[162413.387014]  do_idle+0x1fc/0x270
[162413.387034]  cpu_startup_entry+0x34/0x3c
[162413.387054]  secondary_start_kernel+0x12c/0x14c
[162413.387079]  __secondary_switched+0xb8/0xbc
[162593.393561] rcu: INFO: rcu_preempt self-detected stall on CPU
[162593.393586] rcu:     5-...0: (500093779 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=187509
[162593.393616] rcu:     (t=375024 jiffies g=12239613 q=53876 ncpus=6)
[162593.393636] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[162593.393653] Hardware name: Helios64 (DT)
[162593.393664] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[162593.393682] pc : cpuidle_enter_state+0xc0/0x4bc
[162593.393703] lr : cpuidle_enter_state+0xb8/0x4bc
[162593.393717] sp : ffff800081f23d90
[162593.393728] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[162593.393764] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[162593.393799] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[162593.393834] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[162593.393869] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[162593.393905] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[162593.393940] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[162593.393975] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[162593.394010] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[162593.394045] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[162593.394079] Call trace:
[162593.394090]  cpuidle_enter_state+0xc0/0x4bc
[162593.394105]  cpuidle_enter+0x38/0x50
[162593.394122]  do_idle+0x1fc/0x270
[162593.394143]  cpu_startup_entry+0x34/0x3c
[162593.394162]  secondary_start_kernel+0x12c/0x14c
[162593.394188]  __secondary_switched+0xb8/0xbc
[162773.400657] rcu: INFO: rcu_preempt self-detected stall on CPU
[162773.400683] rcu:     5-...0: (555916563 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=209904
[162773.400712] rcu:     (t=420027 jiffies g=12239613 q=60896 ncpus=6)
[162773.400732] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[162773.400749] Hardware name: Helios64 (DT)
[162773.400760] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[162773.400778] pc : cpuidle_enter_state+0xc0/0x4bc
[162773.400799] lr : cpuidle_enter_state+0xb8/0x4bc
[162773.400813] sp : ffff800081f23d90
[162773.400823] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[162773.400858] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[162773.400893] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[162773.400928] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[162773.400963] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[162773.400998] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[162773.401034] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[162773.401069] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[162773.401103] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[162773.401138] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[162773.401173] Call trace:
[162773.401183]  cpuidle_enter_state+0xc0/0x4bc
[162773.401198]  cpuidle_enter+0x38/0x50
[162773.401216]  do_idle+0x1fc/0x270
[162773.401237]  cpu_startup_entry+0x34/0x3c
[162773.401256]  secondary_start_kernel+0x12c/0x14c
[162773.401282]  __secondary_switched+0xb8/0xbc
[162790.809151] cdn-dp fec00000.dp: [drm:cdn_dp_pd_event_work] Not connected. Disabling cdn
[162791.124875] cdn-dp fec00000.dp: [drm:cdn_dp_pd_event_work] Not connected. Disabling cdn
[162792.091230] cdn-dp fec00000.dp: [drm:cdn_dp_pd_event_work] Not connected. Disabling cdn
[162953.407752] rcu: INFO: rcu_preempt self-detected stall on CPU
[162953.407776] rcu:     5-...0: (611582304 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=232225
[162953.407807] rcu:     (t=465030 jiffies g=12239613 q=69697 ncpus=6)
[162953.407827] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[162953.407845] Hardware name: Helios64 (DT)
[162953.407855] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[162953.407873] pc : cpuidle_enter_state+0xc0/0x4bc
[162953.407894] lr : cpuidle_enter_state+0xb8/0x4bc
[162953.407908] sp : ffff800081f23d90
[162953.407918] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[162953.407954] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[162953.407989] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[162953.408024] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[162953.408059] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[162953.408095] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[162953.408130] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[162953.408165] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[162953.408200] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[162953.408234] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[162953.408269] Call trace:
[162953.408279]  cpuidle_enter_state+0xc0/0x4bc
[162953.408295]  cpuidle_enter+0x38/0x50
[162953.408312]  do_idle+0x1fc/0x270
[162953.408333]  cpu_startup_entry+0x34/0x3c
[162953.408353]  secondary_start_kernel+0x12c/0x14c
[162953.408378]  __secondary_switched+0xb8/0xbc
[163133.414845] rcu: INFO: rcu_preempt self-detected stall on CPU
[163133.414869] rcu:     5-...0: (667562741 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=254606
[163133.414899] rcu:     (t=510033 jiffies g=12239613 q=87475 ncpus=6)
[163133.414918] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[163133.414935] Hardware name: Helios64 (DT)
[163133.414946] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[163133.414964] pc : cpuidle_enter_state+0xc0/0x4bc
[163133.414985] lr : cpuidle_enter_state+0xb8/0x4bc
[163133.414999] sp : ffff800081f23d90
[163133.415010] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[163133.415045] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[163133.415081] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[163133.415117] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[163133.415153] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[163133.415190] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[163133.415225] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[163133.415260] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[163133.415295] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[163133.415330] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[163133.415365] Call trace:
[163133.415375]  cpuidle_enter_state+0xc0/0x4bc
[163133.415390]  cpuidle_enter+0x38/0x50
[163133.415409]  do_idle+0x1fc/0x270
[163133.415432]  cpu_startup_entry+0x34/0x3c
[163133.415451]  secondary_start_kernel+0x12c/0x14c
[163133.415477]  __secondary_switched+0xb8/0xbc
[163313.421939] rcu: INFO: rcu_preempt self-detected stall on CPU
[163313.421962] rcu:     5-...0: (723272070 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=276994
[163313.421991] rcu:     (t=555036 jiffies g=12239613 q=94071 ncpus=6)
[163313.422010] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[163313.422027] Hardware name: Helios64 (DT)
[163313.422037] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[163313.422055] pc : cpuidle_enter_state+0xc0/0x4bc
[163313.422076] lr : cpuidle_enter_state+0xb8/0x4bc
[163313.422090] sp : ffff800081f23d90
[163313.422100] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[163313.422136] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[163313.422171] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[163313.422205] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[163313.422240] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[163313.422276] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[163313.422311] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[163313.422346] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[163313.422380] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[163313.422415] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[163313.422450] Call trace:
[163313.422460]  cpuidle_enter_state+0xc0/0x4bc
[163313.422475]  cpuidle_enter+0x38/0x50
[163313.422492]  do_idle+0x1fc/0x270
[163313.422513]  cpu_startup_entry+0x34/0x3c
[163313.422533]  secondary_start_kernel+0x12c/0x14c
[163313.422558]  __secondary_switched+0xb8/0xbc
[163493.429032] rcu: INFO: rcu_preempt self-detected stall on CPU
[163493.429054] rcu:     5-...0: (780722878 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=299262
[163493.429083] rcu:     (t=600039 jiffies g=12239613 q=206960 ncpus=6)
[163493.429102] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[163493.429119] Hardware name: Helios64 (DT)
[163493.429131] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[163493.429148] pc : cpuidle_enter_state+0xc0/0x4bc
[163493.429170] lr : cpuidle_enter_state+0xb8/0x4bc
[163493.429184] sp : ffff800081f23d90
[163493.429193] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[163493.429229] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[163493.429264] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[163493.429299] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[163493.429335] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[163493.429370] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[163493.429405] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[163493.429440] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[163493.429475] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[163493.429509] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[163493.429544] Call trace:
[163493.429555]  cpuidle_enter_state+0xc0/0x4bc
[163493.429570]  cpuidle_enter+0x38/0x50
[163493.429588]  do_idle+0x1fc/0x270
[163493.429609]  cpu_startup_entry+0x34/0x3c
[163493.429629]  secondary_start_kernel+0x12c/0x14c
[163493.429655]  __secondary_switched+0xb8/0xbc
[163673.436126] rcu: INFO: rcu_preempt self-detected stall on CPU
[163673.436150] rcu:     5-...0: (836489872 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=321550
[163673.436179] rcu:     (t=645042 jiffies g=12239613 q=221464 ncpus=6)
[163673.436197] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[163673.436214] Hardware name: Helios64 (DT)
[163673.436225] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[163673.436243] pc : cpuidle_enter_state+0xc0/0x4bc
[163673.436264] lr : cpuidle_enter_state+0xb8/0x4bc
[163673.436278] sp : ffff800081f23d90
[163673.436288] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[163673.436324] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[163673.436359] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[163673.436394] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[163673.436429] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[163673.436464] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[163673.436499] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[163673.436534] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[163673.436568] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[163673.436603] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[163673.436638] Call trace:
[163673.436648]  cpuidle_enter_state+0xc0/0x4bc
[163673.436663]  cpuidle_enter+0x38/0x50
[163673.436680]  do_idle+0x1fc/0x270
[163673.436701]  cpu_startup_entry+0x34/0x3c
[163673.436721]  secondary_start_kernel+0x12c/0x14c
[163673.436747]  __secondary_switched+0xb8/0xbc
[163853.443218] rcu: INFO: rcu_preempt self-detected stall on CPU
[163853.443234] rcu:     5-...0: (892316711 ticks this GP) idle=7774/1/0x4000000000000002 softirq=8220496/8220498 fqs=343827
[163853.443261] rcu:     (t=690045 jiffies g=12239613 q=241797 ncpus=6)
[163853.443280] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G         C         6.6.29-current-rockchip64 #1
[163853.443296] Hardware name: Helios64 (DT)
[163853.443307] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[163853.443324] pc : cpuidle_enter_state+0xc0/0x4bc
[163853.443344] lr : cpuidle_enter_state+0xb8/0x4bc
[163853.443358] sp : ffff800081f23d90
[163853.443368] x29: ffff800081f23d90 x28: 0000000000000000 x27: 0000000000000000
[163853.443404] x26: 0000000000000000 x25: 00009283767d7353 x24: 0000000000000000
[163853.443439] x23: 0000000000000000 x22: ffff000001b6a080 x21: ffff0000f779cf40
[163853.443474] x20: 00009283767de8ad x19: 0000000000000000 x18: 000000000006f000
[163853.443509] x17: 000000040044ffff x16: 00100074b5503510 x15: 0000000000000000
[163853.443544] x14: 00000000000002a1 x13: ffff8000813b75a8 x12: ffff800081a49b48
[163853.443579] x11: 071c71c71c71c71c x10: ffff0000f779d2ac x9 : 00000000000000a9
[163853.443614] x8 : 0000000000000049 x7 : 00000000000000c0 x6 : 00000374d8579600
[163853.443649] x5 : 00ffffffffffffff x4 : 0000000000000015 x3 : 00008fff2a050864
[163853.443683] x2 : ffff800076092000 x1 : ffff0000f779e4c0 x0 : 0000000000000000
[163853.443718] Call trace:
[163853.443728]  cpuidle_enter_state+0xc0/0x4bc
[163853.443743]  cpuidle_enter+0x38/0x50
[163853.443760]  do_idle+0x1fc/0x270
[163853.443782]  cpu_startup_entry+0x34/0x3c
[163853.443801]  secondary_start_kernel+0x12c/0x14c
[163853.443826]  __secondary_switched+0xb8/0xbc
root@helios64:~#

Edited April 30 by BipBip1981

[BipBip1981]

...and i reproduce twice same freeze with cryptosetup to unlock my raid... freeze network... access usb is ready... but terribely slow... i tape my login and password and i waiting... but no response...

 

picocom v3.1

port is        : /dev/ttyUSB0
flowcontrol    : none
baudrate is    : 1500000
parity is      : none
databits are   : 8
stopbits are   : 1
escape is      : C-a
local echo is  : no
noinit is      : no
noreset is     : no
hangup is      : no
nolock is      : no
send_cmd is    : sz -vv
receive_cmd is : rz -vv -E
imap is        :
omap is        :
emap is        : crcrlf,delbs,
logfile is     : none
initstring     : none
exit_after is  : not set
exit is        : no

Type [C-a] [C-h] to see available commands
Terminal ready

helios64 login: root
Password:

 

I type CTRL-C and...

 

^C _   _      _ _            __   _  _   
| | | | ___| (_) ___  ___ / /_ | || |  
| |_| |/ _ \ | |/ _ \/ __| '_ \| || |_
|  _  |  __/ | | (_) \__ \ (_) |__   _|
|_| |_|\___|_|_|\___/|___/\___/   |_|  
                                       
Welcome to Armbian-unofficial 24.2.1 Bookworm with Linux 6.6.29-current-rockchip64

No end-user support: built from trunk

System load:   3%               Up time:       11 min    Local users:   2                
Memory usage:  7% of 3.77G      IP:           10.0.0.155
CPU temp:      41°C               Usage of /:    46% of 14G        
RX today:      53.8 MiB      

[ General system configuration (beta): armbian-config ]

Web console: https://helios64:9090/

You have no mail.

 

I type CTRL-C

^C

 

And no prompt.... full freeze or lost control

 

I back to

ENABLE=true
MIN_SPEED=408000
MAX_SPEED=1200000
GOVERNOR=schedutil

until you explain me where and how use your file

Edited April 30 by BipBip1981

[ebin-dev]

2 hours ago, BipBip1981 said:

until you explain me where and how use your file

 

Just download the file "rk3399-kobol-helios64.dtb-6.6.29-L2-hs400-opp" and copy it to /boot/dtb/rockchip/rk3399-kobol-helios64.dtb and reboot.

It replaces the original rk3399-kobol-helios64.dtb which you may rename to rk3399-kobol-helios64.dtb-orig beforehand.

 

It clearly enhances the stability of the system - but it cannot resolve any application errors.

Edited April 30 by ebin-dev

[BipBip1981]

Hi, okok, i test with my pattern test

My brother have Helios64 also, he use it with many Docker Container and freeze everyday with default settings, same stabilty problem, he pass to 1200 1200 performance this day and test this setting for moment... if work better to my side with your file, my brother test your file on his helios

 

Your file is on my helios,

400-1800 schedutil

echo check > /sys/block/md0/md/sync_action -> running

btrfs check --check-data-csum  --readonly   --progress    /dev/disk/by-uuid/XXXXXXXXXXXXX -> running

Freeze Crash... Not Freeze Crash... Keep in touch

Edited May 1 by BipBip1981