Jump to content

jbergler

Members
  • Posts

    26
  • Joined

  • Last visited

Posts posted by jbergler

  1. 17 hours ago, Slycat34 said:

    @jbergler where did you find the mkimage flag settings for that command? I want to be sure it'll be the same for the HC4

     

    I pulled that from the postinst of the previous debian package `ar x whatever.deb` and then untar the control.tar.gz file to get it.

     

    Since you're booting without issues, I wonder if your boot.ini has some other config than me?

    I have this part that feels relevant

    # legacy and mainline kernel diff
    if ext4load mmc ${devno}:1 0x00000000 "/boot/.next" || fatload mmc ${devno}:1 0x00000000 ".next"  || ext4load mmc ${devno}:1 0x00000000 ".next"; then
    	echo "Found mainline kernel configuration"
    	setenv uartconsole "ttyAML0,115200n8"
    	setenv kernelimage "uImage"
    else
    	echo "Found legacy kernel configuration"
    	setenv uartconsole "ttyS0,115200n8"
    	setenv kernelimage "zImage"
    fi

     

  2. Can confirm I'm also seeing this. I'm able to recover by installing the 23.02.2 linux-image-current-meson64 and linux-dtb-current-meson64 packages from /var/cache/apt/archives.

     

    From what I can tell it's installing the modules for 6.1.50 in  /lib/modules but is somehow still booting 6.1.11

     

    /boot/uImage isn't getting updated

    $ strings /boot/uImage | grep "Linux version"
    Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT
    Linux version 6.1.11-meson64 (root@29682b33de96) (aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0, GNU ld (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 2.32.0.20190321) #23.02.2 SMP PREEMPT Sat Feb 18 00:07:55 UTC 2023

     

     

    I can confirm that running the following command manually gets the device back into a working state

    $ mkimage -A arm64 -O linux -T kernel -C none -a 0x1080000 -e 0x1080000 -n Linux -d /boot/vmlinuz-6.1.50-current-meson64 /boot/uImage

     

  3. @aprayoga verbosity was already up, but I've added the other args.

     

    I'm not going to provoke the system since it's somewhat stable again and it's in use, but in terms of a repro here's the setup.

    2x 8TB + 3x 12TB drives.

    tank0 5x8TB raidz1

    tank1 3x4TB raidz1 (this tank isn't mounted currently)

     

    If I want to crash the box I can start a zfs scrub on tank0.
    After some time (<~6 hours) the box crashes. On boot, if a scrub was in progress, box won't finish booting.

     

    Spoiler

    $ fdisk -l | grep model
    Disk model: WDC WD120EMFZ-11
    Disk model: WDC WD80EMAZ-00W
    Disk model: WDC WD120EMFZ-11
    Disk model: WDC WD120EMFZ-11
    Disk model: WDC WD80EMAZ-00W

    $ zpool status
      pool: tank0
     state: ONLINE
      scan: scrub canceled on Mon Mar 15 20:26:12 2021
    config:
        NAME                      STATE     READ WRITE CKSUM
        tank0                     ONLINE       0     0     0
          raidz1-0                ONLINE       0     0     0
            zfs-1b8007903f8d4b01  ONLINE       0     0     0
            zfs-946f51545d5dc02a  ONLINE       0     0     0
            zfs-98b261f32fd8b393  ONLINE       0     0     0
            zfs-9b33bb254f272a96  ONLINE       0     0     0
            zfs-ff00f9d7b0840ce1  ONLINE       0     0     0

    errors: No known data errors

      pool: tank1
     state: ONLINE
      scan: scrub repaired 0B in 00:00:21 with 0 errors on Sun Mar 14 00:24:26 2021
    config:
        NAME                      STATE     READ WRITE CKSUM
        tank1                     ONLINE       0     0     0
          raidz1-0                ONLINE       0     0     0
            zfs-e7174a8b87f51397  ONLINE       0     0     0
            zfs-ab22838b00580806  ONLINE       0     0     0
            zfs-c2dca7197e64a200  ONLINE       0     0     0

     

     

  4. My system was stable for a long time (~3-4 weeks) and then the other day it soft locked with a panic (trace was in ZFS).

    Rest of the system was still vaguely usable, great - this has happened before I thought, so I rebooted and could not get it to finish booting.

     

    Every time, one of two things would happen as the zfs pool was mounted.

    1) system would silently lock up, no red led, no panic on console, nothing

    2) system would panic, red led started flashing.

     

    The only way I've been able to get the system to boot is by unplugging the disks, waiting for the system to boot and then plugging the disks back in and mounting them.

    Even then the system crashes again within a short period of time (maybe because the ZFS is trying to scrub following the crash)

     

    I've upgraded to 21.02.3 / 5.10.21

    I never had the vdd tweaks applied, but I've tried both with and without them.

    I've explicitly run the boot-loader upload steps in armbian config (was Nov, now Mar 8)

     

    I'm relatively confident the issue I'm seeing relates to the others here, more often than not the panics are page faults (null pointer, address between kernel and user space, could not execute from non-execute memory) which seems plausible given the focus on voltage tuning.

     

    Any ideas?

    I can make an effort to collect boot logs if that's helpful, but given the frequency of these reports it seems like this is a relatively widespread issue.

  5. On 11/27/2020 at 4:51 PM, gprovost said:

    IMHO zfs-dkms would the easiest approach for most people.

    The problem here is that it‘s not possible to compile the module on Debian because of how the kernel has been built.

    I reported the issue here and while I could *fix* it, it really strikes me as something the core armbian team needs to weigh in on.

    One option is to use an older GCC in the build system, the other is to disable per task stack protections in the kernel - neither seem like great choices to me.

     

  6. 33 minutes ago, TheLinuxBug said:

    @jbergler Do you have an ATX power supply you can hook the drives to and test powering them that way?

    I believe this may be a power delivery issue under load.

     

    I do not unfortunately, but I haven't seen any errors in the lead up to the crashes I've experienced that look like problems with the drives (at least not from what I can tell)

  7. Initial attempt with the new uboot and with removing the cpufreq tweaks results in a new panic

    Spoiler

    [588872.135762] reboot: Restarting system
    DDR Version 1.24 20191016 RevNocRL
    In
    soft reset
    SRX
    channel 0
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 1
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 0 training pass!
    channel 1 training pass!
    change freq to 416MHz 0,1
    Channel 0: LPDDR4,416MHz
    Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
    Channel 1: LPDDR4,416MHz
    Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
    256B stride
    channel 0
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 1
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 0 training pass!
    channel 1 training pass!
    channel 0, cs 0, advanced training done
    channel 1, cs 0, advanced training done
    change freq to 856MHz 1,0
    ch 0 ddrconfig = 0x101, ddrsize = 0x40
    ch 1 ddrconfig = 0x101, ddrsize = 0x40
    pmugrf_os_reg[2] = 0x32C1F2C1, stride = 0xD
    ddr_set_rate to 328MHZ
    ddr_set_rate to 666MHZ
    ddr_set_rate to 416MHZ, ctl_index 0
    ddr_set_rate to 856MHZ, ctl_index 1
    support 416 856 328 666 MHz, current 856MHz
    OUT

    Boot1 Release Time: May 29 2020 17:36:36, version: 1.26
    CPUId = 0x0
    ChipType = 0x10, 447
    SdmmcInit=2 0
    BootCapSize=100000
    UserCapSize=14910MB
    FwPartOffset=2000 , 100000
    mmc0:cmd8,20
    mmc0:cmd5,20
    mmc0:cmd55,20
    mmc0:cmd1,20
    mmc0:cmd8,20
    mmc0:cmd5,20
    mmc0:cmd55,20
    mmc0:cmd1,20
    mmc0:cmd8,20
    mmc0:cmd5,20
    mmc0:cmd55,20
    mmc0:cmd1,20
    SdmmcInit=0 1
    StorageInit ok = 69105
    SecureMode = 0
    SecureInit read PBA: 0x4
    SecureInit read PBA: 0x404
    SecureInit read PBA: 0x804
    SecureInit read PBA: 0xc04
    SecureInit read PBA: 0x1004
    SecureInit read PBA: 0x1404
    SecureInit read PBA: 0x1804
    SecureInit read PBA: 0x1c04
    SecureInit ret = 0, SecureMode = 0
    atags_set_bootdev: ret:(0)
    GPT 0x3335db8 signature is wrong
    recovery gpt...
    GPT 0x3335db8 signature is wrong
    recovery gpt fail!
    Trust Addr:0x4000, 0x58334c42
    No find bl30.bin
    No find bl32.bin
    Load uboot, ReadLba = 2000
    Load OK, addr=0x200000, size=0xdd6b0
    RunBL31 0x40000 @ 96897 us
    NOTICE:  BL31: v1.3(debug):2803a2c8a
    NOTICE:  BL31: Built : 14:31:03, May 19 2020
    NOTICE:  BL31: Rockchip release version: v1.1
    INFO:    GICv3 with legacy support detected. ARM GICV3 driver initialized in EL3
    INFO:    Using opteed sec cpu_context!
    INFO:    boot cpu mask: 0
    INFO:    plat_rockchip_pmu_init(1191): pd status 3e
    INFO:    BL31: Initializing runtime services
    WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
    ERROR:   Error initializing runtime service opteed_fast
    INFO:    BL31: Preparing for EL3 exit to normal world
    INFO:    Entry point address = 0x200000
    INFO:    SPSR = 0x3c9

     

    U-Boot 2020.07-armbian (Nov 25 2020 - 07:14:05 +0700)

    SoC: Rockchip rk3399
    Reset cause: RST
    DRAM:  3.9 GiB
    PMIC:  RK808
    SF: Detected w25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
    MMC:   mmc@fe320000: 1, sdhci@fe330000: 0
    Loading Environment from MMC... *** Warning - bad CRC, using default environment

    In:    serial
    Out:   serial
    Err:   serial
    Model: Helios64
    Revision: 1.2 - 4GB non ECC
    Net:   eth0: ethernet@fe300000
    scanning bus for devices...
    Hit any key to stop autoboot:  0
    Card did not respond to voltage select!
    switch to partitions #0, OK
    mmc0(part 0) is current device
    Scanning mmc 0:1...
    Found U-Boot script /boot/boot.scr
    3185 bytes read in 18 ms (171.9 KiB/s)
    ## Executing script at 00500000
    Boot script loaded from mmc 0
    193 bytes read in 15 ms (11.7 KiB/s)
    16364137 bytes read in 1576 ms (9.9 MiB/s)
    27331072 bytes read in 2614 ms (10 MiB/s)
    79946 bytes read in 40 ms (1.9 MiB/s)
    2698 bytes read in 32 ms (82 KiB/s)

    Applying kernel provided DT fixup script (rockchip-fixup.scr)
    ## Executing script at 09000000
    ## Loading init Ramdisk from Legacy Image at 06000000 ...
       Image Name:   uInitrd
       Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
       Data Size:    16364073 Bytes = 15.6 MiB
       Load Address: 00000000
       Entry Point:  00000000
       Verifying Checksum ... OK
    ## Flattened Device Tree blob at 01f00000
       Booting using the fdt blob at 0x1f00000
       Loading Ramdisk to f4f4b000, end f5ee6229 ... OK
       Loading Device Tree to 00000000f4ecf000, end 00000000f4f4afff ... OK

    Starting kernel ...

    [   16.090622] OF: graph: no port node found in /syscon@ff770000/usb2-phy@e450/otg-port
    [   16.637382] r8152 2-1.4:1.0 (unnamed net_device) (uninitialized): netif_napi_add() called with weight 256
    [   24.585805] Unable to handle kernel NULL pointer dereference at virtual address 00000000000005cc
    [   24.586591] Mem abort info:
    [   24.586844]   ESR = 0x96000004
    [   24.587120]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.587591]   SET = 0, FnV = 0
    [   24.587865]   EA = 0, S1PTW = 0
    [   24.588145] Data abort info:
    [   24.588404]   ISV = 0, ISS = 0x00000004
    [   24.588746]   CM = 0, WnR = 0
    [   24.589014] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.589786] Mem abort info:
    [   24.590038]   ESR = 0x96000004
    [   24.590312]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.590781]   SET = 0, FnV = 0
    [   24.591055]   EA = 0, S1PTW = 0
    [   24.591336] Data abort info:
    [   24.591594]   ISV = 0, ISS = 0x00000004
    [   24.591934]   CM = 0, WnR = 0
    [   24.592201] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.592973] Mem abort info:
    [   24.593225]   ESR = 0x96000004
    [   24.593499]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.593968]   SET = 0, FnV = 0
    [   24.594241]   EA = 0, S1PTW = 0
    [   24.594522] Data abort info:
    [   24.594780]   ISV = 0, ISS = 0x00000004
    [   24.595121]   CM = 0, WnR = 0
    [   24.595388] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.596161] Mem abort info:
    [   24.596412]   ESR = 0x96000004
    [   24.596686]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.597155]   SET = 0, FnV = 0
    [   24.597428]   EA = 0, S1PTW = 0
    [   24.597709] Data abort info:
    [   24.597967]   ISV = 0, ISS = 0x00000004
    [   24.598308]   CM = 0, WnR = 0
    [   24.598576] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.599349] Mem abort info:
    [   24.599601]   ESR = 0x96000004
    [   24.599874]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.600343]   SET = 0, FnV = 0
    [   24.600616]   EA = 0, S1PTW = 0
    [   24.600896] Data abort info:
    [   24.601154]   ISV = 0, ISS = 0x00000004
    [   24.601495]   CM = 0, WnR = 0
    [   24.601762] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.602534] Mem abort info:
    [   24.602784]   ESR = 0x96000004
    [   24.603058]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.603527]   SET = 0, FnV = 0
    [   24.603800]   EA = 0, S1PTW = 0
    [   24.604081] Data abort info:
    [   24.604339]   ISV = 0, ISS = 0x00000004
    [   24.604680]   CM = 0, WnR = 0
    [   24.604946] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.605718] Mem abort info:
    [   24.605968]   ESR = 0x96000004
    [   24.606242]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.606711]   SET = 0, FnV = 0
    [   24.606984]   EA = 0, S1PTW = 0
    [   24.607263] Data abort info:
    [   24.607521]   ISV = 0, ISS = 0x00000004
    [   24.607863]   CM = 0, WnR = 0
    [   24.608129] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.608901] Mem abort info:
    [   24.609153]   ESR = 0x96000004
    [   24.609426]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.609895]   SET = 0, FnV = 0
    [   24.610168]   EA = 0, S1PTW = 0
    [   24.610448] Data abort info:
    [   24.610706]   ISV = 0, ISS = 0x00000004
    [   24.611048]   CM = 0, WnR = 0
    [   24.611314] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.612086] Mem abort info:
    [   24.612336]   ESR = 0x96000004
    [   24.612610]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.613079]   SET = 0, FnV = 0
    [   24.613352]   EA = 0, S1PTW = 0
    [   24.613633] Data abort info:
    [   24.613891]   ISV = 0, ISS = 0x00000004
    [   24.614232]   CM = 0, WnR = 0
    [   24.614500] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.615272] Mem abort info:
    [   24.615522]   ESR = 0x96000004
    [   24.615796]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.616265]   SET = 0, FnV = 0
    [   24.616538]   EA = 0, S1PTW = 0
    [   24.616819] Data abort info:
    [   24.617077]   ISV = 0, ISS = 0x00000004
    [   24.617418]   CM = 0, WnR = 0
    [   24.617684] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.618457] Mem abort info:
    [   24.618707]   ESR = 0x96000004
    [   24.618980]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.619450]   SET = 0, FnV = 0
    [   24.619723]   EA = 0, S1PTW = 0
    [   24.620004] Data abort info:
    [   24.620262]   ISV = 0, ISS = 0x00000004
    [   24.620603]   CM = 0, WnR = 0
    [   24.620871] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.621644] Mem abort info:
    [   24.621893]   ESR = 0x96000004
    [   24.622167]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.622637]   SET = 0, FnV = 0
    [   24.622910]   EA = 0, S1PTW = 0
    [   24.623191] Data abort info:
    [   24.623448]   ISV = 0, ISS = 0x00000004
    [   24.623790]   CM = 0, WnR = 0
    [   24.624056] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.624828] Mem abort info:
    [   24.625080]   ESR = 0x96000004
    [   24.625354]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.625823]   SET = 0, FnV = 0
    [   24.626096]   EA = 0, S1PTW = 0
    [   24.626377] Data abort info:
    [   24.626635]   ISV = 0, ISS = 0x00000004
    [   24.626976]   CM = 0, WnR = 0
    [   24.627244] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.628016] Mem abort info:
    [   24.628266]   ESR = 0x96000004
    [   24.628539]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.629009]   SET = 0, FnV = 0
    [   24.629282]   EA = 0, S1PTW = 0
    [   24.629563] Data abort info:
    [   24.629821]   ISV = 0, ISS = 0x00000004
    [   24.630162]   CM = 0, WnR = 0
    [   24.630428] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.631201] Mem abort info:
    [   24.631451]   ESR = 0x96000004
    [   24.631724]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.632193]   SET = 0, FnV = 0
    [   24.632466]   EA = 0, S1PTW = 0
    [   24.632747] Data abort info:
    [   24.633005]   ISV = 0, ISS = 0x00000004
    [   24.633346]   CM = 0, WnR = 0
    [   24.633612] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.634384] Mem abort info:
    [   24.634634]   ESR = 0x96000004
    [   24.634908]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.635377]   SET = 0, FnV = 0
    [   24.635650]   EA = 0, S1PTW = 0
    [   24.635931] Data abort info:
    [   24.636189]   ISV = 0, ISS = 0x00000004
    [   24.636530]   CM = 0, WnR = 0
    [   24.636798] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.637570] Mem abort info:
    [   24.637820]   ESR = 0x96000004
    [   24.638094]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.638563]   SET = 0, FnV = 0
    [   24.638836]   EA = 0, S1PTW = 0
    [   24.639116] Data abort info:
    [   24.639375]   ISV = 0, ISS = 0x00000004
    [   24.639715]   CM = 0, WnR = 0
    [   24.639982] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.640754] Mem abort info:
    [   24.641004]   ESR = 0x96000004
    [   24.641277]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.641746]   SET = 0, FnV = 0
    [   24.642019]   EA = 0, S1PTW = 0
    [   24.642300] Data abort info:
    [   24.642558]   ISV = 0, ISS = 0x00000004
    [   24.642899]   CM = 0, WnR = 0
    [   24.643165] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.643938] Mem abort info:
    [   24.644188]   ESR = 0x96000004
    [   24.644461]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.644930]   SET = 0, FnV = 0
    [   24.645203]   EA = 0, S1PTW = 0
    [   24.645484] Data abort info:
    [   24.645742]   ISV = 0, ISS = 0x00000004
    [   24.646083]   CM = 0, WnR = 0
    [   24.646351] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.647123] Mem abort info:
    [   24.647373]   ESR = 0x96000004
    [   24.647646]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.648115]   SET = 0, FnV = 0
    [   24.648388]   EA = 0, S1PTW = 0
    [   24.648668] Data abort info:
    [   24.648926]   ISV = 0, ISS = 0x00000004
    [   24.649267]   CM = 0, WnR = 0
    [   24.649533] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.650305] Mem abort info:
    [   24.650555]   ESR = 0x96000004
    [   24.650829]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.651298]   SET = 0, FnV = 0
    [   24.651571]   EA = 0, S1PTW = 0
    [   24.651852] Data abort info:
    [   24.652109]   ISV = 0, ISS = 0x00000004
    [   24.652451]   CM = 0, WnR = 0
    [   24.652717] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
    [   24.653489] Mem abort info:
    [   24.653739]   ESR = 0x96000004
    [   24.654013]   EC = 0x25: DABT (current EL), IL = 32 bits
    [   24.654482]   SET = 0, FnV = 0
    [   24.654755]   EA = 0, S1PTW = 0
    [   24.655036] Data abort info:
    [   24.655294]   ISV = 0, ISS = 0x00000004
    [   24.655635]   CM = 0, WnR = 0
    [   24.656017] Insufficient stack space to handle exception!
    [   24.656021] ESR: 0x96000047 -- DABT (current EL)
    [   24.656022] FAR: 0xffff800011b9fff0
    [   24.656024] Task stack:     [0xffff800011ba0000..0xffff800011ba4000]
    [   24.656026] IRQ stack:      [0xffff800011ad8000..0xffff800011adc000]
    [   24.656028] Overflow stack: [0xffff0000f77932b0..0xffff0000f77942b0]
    [   24.656031] CPU: 4 PID: 0 Comm: swapper/4 Tainted: P         C OE     5.8.17-rockchip64 #20.08.21
    [   24.656032] Hardware name: Helios64 (DT)
    [   24.656034] pstate: 80000085 (Nzcv daIf -PAN -UAO BTYPE=--)
    [   24.656036] pc : format_decode+0x4/0x4a8
    [   24.656037] lr : vsnprintf+0x8c/0x728
    [   24.656039] sp : ffff800011ba0020
    [   24.656040] x29: ffff800011ba0020 x28: ffff8000111db6b8
    [   24.656045] x27: ffff800011a1d238 x26: 0000000000000020
    [   24.656049] x25: 0000000000000000 x24: 00000000000003e0
    [   24.656053] x23: 00000000ffffffc8 x22: ffff800010ecb890
    [   24.656056] x21: ffff800011ba0350 x20: ffff800011a1d238
    [   24.656060] x19: ffff800011a1d618 x18: 0000000000000010
    [   24.656064] x17: 0000000000000001 x16: 0000000000000019
    [   24.656068] x15: ffff0000f6ea5ba8 x14: 0720072007200720
    [   24.656072] x13: 0720072007200720 x12: 0720072007200720
    [   24.656075] x11: ffff800011ba0350 x10: ffff800011ba0350
    [   24.656079] x9 : ffff800011ba0350 x8 : ffff800011ba0350
    [   24.656083] x7 : ffff800011ba0350 x6 : ffff800011ba0350
    [   24.656086] x5 : 0000000000000000 x4 : ffff0000f6ea5700
    [   24.656090] x3 : ffff800011ba00d0 x2 : ffff8000111db6b8
    [   24.656094] x1 : ffff800011ba00a0 x0 : ffff8000111db6b8
    [   24.656098] Kernel panic - not syncing: kernel stack overflow
    [   24.656100] SMP: stopping secondary CPUs
    [   24.656102] Kernel Offset: disabled
    [   24.656103] CPU features: 0x240022,2000600c
    [   24.656105] Memory Limit: none

     

     

    And trying again

    Spoiler

    [   19.133928] Unable to handle kernel paging request at virtual address ffff80000ee0257c
    [   19.134640] Mem abort info:
    [   19.134892]   ESR = 0x86000006
    [   19.135169]   EC = 0x21: IABT (current EL), IL = 32 bits
    [   19.135639]   SET = 0, FnV = 0
    [   19.135913]   EA = 0, S1PTW = 0
    [   19.136197] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000035ec000
    [   19.136789] [ffff80000ee0257c] pgd=00000000f7fff003, p4d=00000000f7fff003, pud=00000000f7ffe003, pmd=0000000000000000
    [   19.137730] Internal error: Oops: 86000006 [#1] PREEMPT SMP
    [   19.138224] Modules linked in: zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zcommon(POE) znvpair(POE) zavl(POE) icp(POE) spl(OE) r8152 snd_soc_hdmi_codec panfrost snd_soc_rockchip_i2s gpu_sched snd_soc_core leds_pwm snd_pcm_dmaengine pwm_fan gpio_charger snd_pcm hantro_vpu(C) snd_timer rockchip_rga rockchip_vdec(C) snd videobuf2_dma_sg soundcore v4l2_h264 videobuf2_dma_contig videobuf2_vmalloc fusb30x(C) v4l2_mem2mem videobuf2_memops videobuf2_v4l2 videobuf2_common videodev mc sg gpio_beeper cpufreq_dt sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace lm75 sunrpc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod realtek rockchipdrm analogix_dp dw_hdmi dw_mipi_dsi drm_kms_helper cec rc_core dwmac_rk stmmac_platform drm stmmac mdio_xpcs drm_panel_orientation_quirks adc_keys
    [   19.144940] CPU: 4 PID: 0 Comm: swapper/4 Tainted: P         C OE     5.8.17-rockchip64 #20.08.21
    [   19.145721] Hardware name: Helios64 (DT)
    [   19.146073] pstate: 80000085 (Nzcv daIf -PAN -UAO BTYPE=--)
    [   19.146572] pc : 0xffff80000ee0257c
    [   19.146894] lr : _raw_spin_lock_irqsave+0x28/0xa0
    [   19.147313] sp : ffff800011adbeb0
    [   19.147609] x29: ffff800011adbeb0 x28: 0000000000000001
    [   19.148083] x27: ffff0000f6ea5700 x26: ffff800011adc000
    [   19.148555] x25: ffff800011501d20 x24: 0000000000000000
    [   19.149027] x23: 0000000000000000 x22: ffff0000f6ea5700
    [   19.149498] x21: ffff0000f77a8b40 x20: 0000000000000080
    [   19.149970] x19: ffff0000f77a8b40 x18: 0000000000000000
    [   19.150441] x17: 0000000000000001 x16: 0000000000000019
    [   19.150912] x15: 0000000000000006 x14: 000010670d7edc0e
    [   19.151384] x13: 00000000000003fd x12: 0000000000000006
    [   19.151855] x11: 0000000000000001 x10: 0000000000000a20
    [   19.152327] x9 : ffff800011ba3e70 x8 : ffff0000f6ea6180
    [   19.152798] x7 : 00000000ffffffff x6 : 00000000351d78da
    [   19.153270] x5 : 00ffffffffffffff x4 : 002b646607bcf500
    [   19.153741] x3 : 0000000000000000 x2 : 0000000000000001
    [   19.154212] x1 : 0000000000000000 x0 : 0000000000000000
    [   19.154684] Call trace:
    [   19.154906]  0xffff80000ee0257c
    [   19.155196]  sched_ttwu_pending+0x58/0x168
    [   19.155566]  flush_smp_call_function_queue+0xec/0x258
    [   19.156018]  generic_smp_call_function_single_interrupt+0x14/0x20
    [   19.156561]  handle_IPI+0x258/0x3e8
    [   19.156876]  gic_handle_irq+0x154/0x158
    [   19.157220]  el1_irq+0xb8/0x180
    [   19.157505]  arch_cpu_idle+0x28/0x218
    [   19.157836]  default_idle_call+0x1c/0x44
    [   19.158188]  do_idle+0x210/0x288
    [   19.158478]  cpu_startup_entry+0x28/0x68
    [   19.158830]  secondary_start_kernel+0x140/0x178
    [   19.159239] Code: bad PC value
    [   19.159524] ---[ end trace 99042d0e071b2912 ]---
    [   19.159936] Kernel panic - not syncing: Fatal exception in interrupt
    [   19.160500] SMP: stopping secondary CPUs
    [   20.327519] SMP: failed to stop secondary CPUs 3-5
    [   20.327945] Kernel Offset: disabled
    [   20.328257] CPU features: 0x240022,2000600c
    [   20.328629] Memory Limit: none
    [   20.328915] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

     

  8. I'll defer to the Kobol folks, in the previous mega thread the statement was made that the issues should have been fixed in a new version that ensured it was correctly applying the hardware tweaks, for me things have never been properly stable, even on just a vanilla install. The only semi-stable solution has been to reduce the clock speed, which is fine for now.

  9. I had 1 more crash and another soft lockup, but otherwise the box is much more usable.

     

    @aprayoga Definitely still something not running right, even at the lower clock speeds.
    My limited knowledge suggests something memory related, but that's all I've got. If you'd like me to test anything else, let me know.

  10. After about an hour of the ZFS scrub the "bad PC value" error happened again, however this time the system didn't hard lock.

    A decent number of processes related to ZFS are stuck in uninterruptible IO, I can't export the pool, etc.

     

    I did see the system crash like this occasionally without the cpufreq tweaks, so I'm not sure it tells us anything new.

    I will try again.

     

    note, the relatively high uptime is from the system sitting idle for ~5 days before I put it under load again.

     

    Spoiler

    [433046.690213] Unable to handle kernel paging request at virtual address f9ff8000091f3190

    [433046.690218] Internal error: SP/PC alignment exception: 8a000000 [#1] PREEMPT SMP

    [433046.690224] Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge rfkill governor_performance zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zcommon(POE) znvpair(POE) zavl(POE) icp(POE) spl(OE) r8152 snd_soc_hdmi_codec snd_soc_rockchip_i2s snd_soc_core snd_pcm_dmaengine snd_pcm snd_timer panfrost snd gpu_sched soundcore leds_pwm gpio_charger pwm_fan rockchip_rga videobuf2_dma_sg hantro_vpu(C) rockchip_vdec(C) v4l2_h264 videobuf2_dma_contig videobuf2_vmalloc v4l2_mem2mem videobuf2_memops videobuf2_v4l2 videobuf2_common videodev mc fusb30x(C) zstd sg gpio_beeper cpufreq_dt zram sch_fq_codel lm75 ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod realtek rockchipdrm analogix_dp dw_hdmi dwmac_rk dw_mipi_dsi stmmac_platform drm_kms_helper cec stmmac rc_core

    [433046.690323] mdio_xpcs

    [433046.690976] Mem abort info:

    [433046.691593] drm drm_panel_orientation_quirks adc_keys

    [433046.699701]   ESR = 0x86000004

    [433046.700155] CPU: 5 PID: 248302 Comm: z_rd_int Tainted: P         C OE     5.8.17-rockchip64 #20.08.21

    [433046.700433]   EC = 0x21: IABT (current EL), IL = 32 bits

    [433046.701245] Hardware name: Helios64 (DT)

    [433046.701718]   SET = 0, FnV = 0

    [433046.702073] pstate: 40000005 (nZcv daif -PAN -UAO BTYPE=--)

    [433046.702373]   EA = 0, S1PTW = 0

    [433046.702850] pc : 0xb

    [433046.703132] [f9ff8000091f3190] address between user and kernel address ranges

    [433046.703334] lr : 0xb

    [433046.704168] sp : ffff800019d53a40

    [433046.704469] x29: ffff0000b604c000 x28: ffff0000f6c03a00

    [433046.704946] x27: ffff000045281600 x26: 000000000000000b

    [433046.705421] x25: ffff800011a10000 x24: 0000000000000000

    [433046.705897] x23: 0000000000000000 x22: 0080000000000000

    [433046.706374] x21: 0000000000042c00 x20: ffff000092ff8d88

    [433046.706849] x19: ffff000045281600 x18: 00001e1e0a99c21b

    [433046.707326] x17: 00000030510320ae x16: 000000fe01cf8d4b

    [433046.707801] x15: 0000000000000000 x14: 0000000000000000

    [433046.708277] x13: 0000000000000008 x12: ffff0000d8f2ea28

    [433046.708753] x11: 0000000000000020 x10: 0000000000000001

    [433046.709229] x9 : 0000000000000000 x8 : ffff00006fb62b00

    [433046.709705] x7 : 0000000000000000 x6 : 000000000000003f

    [433046.710181] x5 : 0000000000000040 x4 : 0000000000000000

    [433046.710657] x3 : 0000000000000004 x2 : 0000000000000000

    [433046.711133] x1 : ffff000000000000 x0 : ffff00006fb62a00

    [433046.711610] Call trace:

    [433046.711837] 0xb

    [433046.712016] Code: bad PC value

    [433046.712298] ---[ end trace ac904cdd631dd942 ]---

    [433046.712714] Internal error: Oops: 86000004 [#2] PREEMPT SMP

    [433046.713212] Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge rfkill governor_performance zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zcommon(POE) znvpair(POE) zavl(POE) icp(POE) spl(OE) r8152 snd_soc_hdmi_codec snd_soc_rockchip_i2s snd_soc_core snd_pcm_dmaengine snd_pcm snd_timer panfrost snd gpu_sched soundcore leds_pwm gpio_charger pwm_fan rockchip_rga videobuf2_dma_sg hantro_vpu(C) rockchip_vdec(C) v4l2_h264 videobuf2_dma_contig videobuf2_vmalloc v4l2_mem2mem videobuf2_memops videobuf2_v4l2 videobuf2_common videodev mc fusb30x(C) zstd sg gpio_beeper cpufreq_dt zram sch_fq_codel lm75 ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod realtek rockchipdrm analogix_dp dw_hdmi dwmac_rk dw_mipi_dsi stmmac_platform drm_kms_helper cec stmmac rc_core

    [433046.713298] mdio_xpcs drm drm_panel_orientation_quirks adc_keys

    [433046.721466] CPU: 4 PID: 248273 Comm: z_rd_int Tainted: P      D  C OE     5.8.17-rockchip64 #20.08.21

    [433046.722281] Hardware name: Helios64 (DT)

    [433046.722637] pstate: 80000005 (Nzcv daif -PAN -UAO BTYPE=--)

    [433046.723135] pc : 0xf9ff8000091f3190

    [433046.723464] lr : avl_find+0x68/0xc8 [zavl]

    [433046.723833] sp : ffff800019c73a40

    [433046.724134] x29: ffff800019c73a40 x28: ffff000080c2afa8

    [433046.724611] x27: ffff0000b604c9a8 x26: ffff0000b604c9c8

    [433046.725088] x25: ffff0000b6743090 x24: 0000000000000000

    [433046.725565] x23: ffff800019c73af0 x22: ffff8000091f40d8

    [433046.726041] x21: ffff00005c0be900 x20: ffff000056059e00

    [433046.726517] x19: ffff000056059e00 x18: 000021ba4d598e5d

    [433046.726994] x17: 0000003fa86bd6e8 x16: 0000014c01a9b0f1

    [433046.727470] x15: 0000000000000000 x14: 0000000000000000

    [433046.727946] x13: 0000000000000008 x12: ffff0000e5b3b028

    [433046.728422] x11: 0000000000000100 x10: 0000000000000001

    [433046.728898] x9 : 0000000000000000 x8 : 000000000023e0e8

    [433046.729373] x7 : 000000000023e120 x6 : 0000000000000001

    [433046.729849] x5 : 0000000000000001 x4 : 0000000000000000

    [433046.730325] x3 : 0000000000000000 x2 : 0000000000000100

    [433046.730801] x1 : 0000000000000000 x0 : 00000000ffffffff

    [433046.731277] Call trace:

    [433046.731503] 0xf9ff8000091f3190

    [433046.731948] dsl_scan_prefetch+0x1a8/0x228 [zfs]

    [433046.732490] dsl_scan_prefetch_dnode+0x8c/0x110 [zfs]

    [433046.733068] dsl_scan_prefetch_cb+0x21c/0x268 [zfs]

    [433046.733630] arc_read_done+0x20c/0x3f8 [zfs]

    [433046.734140] zio_done+0x254/0xd40 [zfs]

    [433046.734634] zio_execute+0xac/0x110 [zfs]

    [433046.735016] taskq_thread+0x298/0x440 [spl]

    [433046.735402] kthread+0x118/0x150

    [433046.735700] ret_from_fork+0x10/0x34

    [433046.736031] Code: bad PC value

    [433046.736315] ---[ end trace ac904cdd631dd943 ]---

     

  11. On 11/13/2020 at 5:31 PM, aprayoga said:

    Did you encounter kernel crash on 20.08.10 ?

     

    It's hard to say for sure, I never quite had a stable system, but I also wasn't generating the kind of load I am now back then.

     

    On 11/13/2020 at 5:31 PM, aprayoga said:

    What was the max cpu freq you set?

     

     

    I had only reduced it one step, I'm trying again now with the settings you suggest.

     

    root@helios64:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | uniq
    performance
    root@helios64:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq | uniq
    816000
    root@helios64:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq | uniq
    1200000

     

    The load I'm generating is running a zfs scrub on a 37TB pool across all five disks.

     

  12. Continuing the discussion from here

     

    On a clean install of 20.08.21 im able to crash the box within a few hours of it being under load.

    It appears as if the optimisations are being applied

    root@helios64:~# cat /proc/sys/net/core/rps_sock_flow_entries
    32768

     

    The suggestion @ShadowDance made to switch to the performance governor hasn't helped.

     

    Anecdotally, I think I remember the crashes always mentioning page faults, and early on there was some discussion about memory timing. Is it possible this continues to be that issue?

     

  13. root@helios64:~# cat /proc/sys/net/core/rps_sock_flow_entries
    32768

     

    I also tried the suggestion to set a performance governor, and for shits and giggles I reduced the max cpu frequency, but that hasn’t made a difference.

    System still locks up within a few hours.

     

    I did finally manage to get the serial console to print something meaningful during boot, and one thing that stands out is this

     

    Loading Environment from MMC... *** Warning - bad CRC, using default environment

     

    Full boot log is below

    Spoiler

    DDR Version 1.24 20191016
    In
    channel 0
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 1
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 0 training pass!
    channel 1 training pass!
    change freq to 416MHz 0,1
    Channel 0: LPDDR4,416MHz
    Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
    Channel 1: LPDDR4,416MHz
    Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB
    256B stride
    channel 0
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 1
    CS = 0
    MR0=0x18
    MR4=0x1
    MR5=0x1
    MR8=0x10
    MR12=0x72
    MR14=0x72
    MR18=0x0
    MR19=0x0
    MR24=0x8
    MR25=0x0
    channel 0 training pass!
    channel 1 training pass!
    channel 0, cs 0, advanced training done
    channel 1, cs 0, advanced training done
    change freq to 856MHz 1,0
    ch 0 ddrconfig = 0x101, ddrsize = 0x40
    ch 1 ddrconfig = 0x101, ddrsize = 0x40
    pmugrf_os_reg[2] = 0x32C1F2C1, stride = 0xD
    ddr_set_rate to 328MHZ
    ddr_set_rate to 666MHZ
    ddr_set_rate to 928MHZ
    channel 0, cs 0, advanced training done
    channel 1, cs 0, advanced training done
    ddr_set_rate to 416MHZ, ctl_index 0
    ddr_set_rate to 856MHZ, ctl_index 1
    support 416 856 328 666 928 MHz, current 856MHz
    OUT
    Boot1: 2019-03-14, version: 1.19
    CPUId = 0x0
    ChipType = 0x10, 254
    SdmmcInit=2 0
    BootCapSize=100000
    UserCapSize=14910MB
    FwPartOffset=2000 , 100000
    mmc0:cmd8,20
    mmc0:cmd5,20
    mmc0:cmd55,20
    mmc0:cmd1,20
    mmc0:cmd8,20
    mmc0:cmd5,20
    mmc0:cmd55,20
    mmc0:cmd1,20
    mmc0:cmd8,20
    mmc0:cmd5,20
    mmc0:cmd55,20
    mmc0:cmd1,20
    SdmmcInit=0 1
    StorageInit ok = 67811
    SecureMode = 0
    SecureInit read PBA: 0x4
    SecureInit read PBA: 0x404
    SecureInit read PBA: 0x804
    SecureInit read PBA: 0xc04
    SecureInit read PBA: 0x1004
    SecureInit read PBA: 0x1404
    SecureInit read PBA: 0x1804
    SecureInit read PBA: 0x1c04
    SecureInit ret = 0, SecureMode = 0
    atags_set_bootdev: ret:(0)
    GPT 0x3380ec0 signature is wrong
    recovery gpt...
    GPT 0x3380ec0 signature is wrong
    recovery gpt fail!
    LoadTrust Addr:0x4000
    No find bl30.bin
    No find bl32.bin
    Load uboot, ReadLba = 2000
    Load OK, addr=0x200000, size=0xdd6b0
    RunBL31 0x40000
    NOTICE:  BL31: v1.3(debug):42583b6
    NOTICE:  BL31: Built : 07:55:13, Oct 15 2019
    NOTICE:  BL31: Rockchip release version: v1.1
    INFO:    GICv3 with legacy support detected. ARM GICV3 driver initialized in EL3
    INFO:    Using opteed sec cpu_context!
    INFO:    boot cpu mask: 0
    INFO:    plat_rockchip_pmu_init(1190): pd status 3e
    INFO:    BL31: Initializing runtime services
    WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
    ERROR:   Error initializing runtime service opteed_fast
    INFO:    BL31: Preparing for EL3 exit to normal world
    INFO:    Entry point address = 0x200000
    INFO:    SPSR = 0x3c9


    U-Boot 2020.07-armbian (Oct 31 2020 - 08:21:38 +0100)

    SoC: Rockchip rk3399
    Reset cause: POR
    DRAM:  3.9 GiB
    PMIC:  RK808 
    SF: Detected w25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
    MMC:   mmc@fe320000: 1, sdhci@fe330000: 0
    Loading Environment from MMC... *** Warning - bad CRC, using default environment

    In:    serial
    Out:   serial
    Err:   serial
    Model: Helios64
    Revision: 1.2 - 4GB non ECC
    Net:   eth0: ethernet@fe300000
    scanning bus for devices...
    Hit any key to stop autoboot:  0 
    Card did not respond to voltage select!
    switch to partitions #0, OK
    mmc0(part 0) is current device
    Scanning mmc 0:1...
    Found U-Boot script /boot/boot.scr
    3185 bytes read in 18 ms (171.9 KiB/s)
    ## Executing script at 00500000
    Boot script loaded from mmc 0
    193 bytes read in 15 ms (11.7 KiB/s)
    16364137 bytes read in 1576 ms (9.9 MiB/s)
    27331072 bytes read in 2614 ms (10 MiB/s)
    79946 bytes read in 39 ms (2 MiB/s)
    2698 bytes read in 32 ms (82 KiB/s)
    Applying kernel provided DT fixup script (rockchip-fixup.scr)
    ## Executing script at 09000000
    ## Loading init Ramdisk from Legacy Image at 06000000 ...
       Image Name:   uInitrd
       Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
       Data Size:    16364073 Bytes = 15.6 MiB
       Load Address: 00000000
       Entry Point:  00000000
       Verifying Checksum ... OK
    ## Flattened Device Tree blob at 01f00000
       Booting using the fdt blob at 0x1f00000
       Loading Ramdisk to f4f4b000, end f5ee6229 ... OK
       Loading Device Tree to 00000000f4ecf000, end 00000000f4f4afff ... OK

    Starting kernel ...

    [   16.262031] OF: graph: no port node found in /syscon@ff770000/usb2-phy@e450/otg-port
    [   16.620006] systemd[1]: Failed to start Set console font and keymap.
    [   17.073543] r8152 2-1.4:1.0 (unnamed net_device) (uninitialized): netif_napi_add() called with weight 256

    Armbian 20.08.21 Focal ttyS2 

    helios64 login: 
     

     

    5 hours ago, gprovost said:

    You mean even after clean install you still have the instability issue ?

     

    Yes,

  14. I'm still seeing regular panics, to the point where the box won't stay up for more than a few hours.

    To ensure it was in a clean state, I re-installed 20.08.21 focal and only added samba + zfs back.

     

    Spoiler

    [13245.115739] Unable to handle kernel paging request at virtual address 7ff60000f77a94f0
    [13245.116452] Mem abort info:
    [13245.116700]   ESR = 0x96000004
    [13245.116976]   EC = 0x25: DABT (current EL), IL = 32 bits
    [13245.117441]   SET = 0, FnV = 0
    [13245.117711]   EA = 0, S1PTW = 0
    [13245.117988] Data abort info:
    [13245.118243]   ISV = 0, ISS = 0x00000004
    [13245.118579]   CM = 0, WnR = 0
    [13245.118842] [7ff60000f77a94f0] address between user and kernel address ranges
    [13245.119469] Internal error: Oops: 96000004 [#1] PREEMPT SMP
    [13245.119960] Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter br_netfilter bridge rfkill governor_performance zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zcommon(POE) znvpair(POE) zavl(POE) icp(POE) spl(OE) r8152 snd_soc_hdmi_codec leds_pwm gpio_charger pwm_fan snd_soc_rockchip_i2s snd_soc_core snd_pcm_dmaengine snd_pcm snd_timer snd soundcore panfrost gpu_sched fusb30x(C) hantro_vpu(C) rockchip_vdec(C) rockchip_rga v4l2_h264 videobuf2_dma_contig videobuf2_dma_sg v4l2_mem2mem videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev mc sg gpio_beeper cpufreq_dt zstd zram sch_fq_codel lm75 ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod realtek dwmac_rk stmmac_platform stmmac mdio_xpcs rockchipdrm analogix_dp dw_hdmi dw_mipi_dsi drm_kms_helper
    [13245.120050]  cec rc_core drm drm_panel_orientation_quirks adc_keys
    [13245.128189] CPU: 4 PID: 0 Comm: swapper/4 Tainted: P         C OE     5.8.17-rockchip64 #20.08.21
    [13245.128964] Hardware name: Helios64 (DT)
    [13245.129312] pstate: 40000085 (nZcv daIf -PAN -UAO BTYPE=--)
    [13245.129811] pc : __schedule+0xb4/0x808
    [13245.130142] lr : __schedule+0xb4/0x808
    [13245.130471] sp : ffff800011ba3ea0
    [13245.130763] x29: ffff800011ba3ea0 x28: ffff0000f6ea5700
    [13245.131229] x27: ffff800011a10000 x26: 0000000000000000
    [13245.131695] x25: 0000000000000000 x24: ffff800011809980
    [13245.132160] x23: ffff0000f6ea5700 x22: 0000000000000000
    [13245.132626] x21: ffff800010dfc8b0 x20: ffff0000f6ea5700
    [13245.133091] x19: 7ff60000f77a8b40 x18: 0000000000000014
    [13245.133556] x17: 0000000088c8bd3f x16: 00000000fe39365a
    [13245.134021] x15: 00000000a4a528e1 x14: 0000000100316213
    [13245.134487] x13: 0000000000000101 x12: 0000000000000110
    [13245.134952] x11: 0000000000000001 x10: 0000000000000a20
    [13245.135418] x9 : ffff800011ba3e70 x8 : ffff0000f6ea6180
    [13245.135883] x7 : 00000000ffffffff x6 : 0000004a1d089376
    [13245.136349] x5 : 00ffffffffffffff x4 : ffff0000f77a9898
    [13245.136815] x3 : 0000000000000000 x2 : 0000000000000000
    [13245.137280] x1 : 182800a06258e900 x0 : 0000000000000000
    [13245.137746] Call trace:
    [13245.137966]  __schedule+0xb4/0x808
    [13245.138267]  schedule_idle+0x28/0x48
    [13245.138585]  do_idle+0x184/0x288
    [13245.138870]  cpu_startup_entry+0x28/0x68
    [13245.139221]  secondary_start_kernel+0x140/0x178
    [13245.139621] Code: 2a1903e0 97cc5f9e aa1303e0 9400189e (b949b260)
    [13245.140163] ---[ end trace c9acabedf32c9896 ]---
    [13245.140569] Kernel panic - not syncing: Attempted to kill the idle task!
    [13245.141156] SMP: stopping secondary CPUs
    [13246.308168] SMP: failed to stop secondary CPUs 0,4
    [13246.308588] Kernel Offset: disabled
    [13246.308895] CPU features: 0x240022,2000600c
    [13246.309261] Memory Limit: none
    [13246.309538] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

     

     

  15. Have there been any changes to how the serial console is initialised?
     
    I previously had no issues using the serial console to interact with uboot, but now the serial console doesn't seem to initialise until linux boots, and even then there's a bunch of gibberish until the tty does something.
     

    [  326.906283] reboot: Restarting system
    7�[�A5_U�zz�=E:�{�;:���/{��_�Z{�h^�;���[�xz��ap��E^�x^��[6zp��xz{[��[���>va[/{=A5_U�zz�=E:�s�?:���/{��_�[{�h^�;���Z[xz?z��ap��E^�x^��[6zp��xz{[��;'~E){=a[{=Armbian 20.08.21 Focal ttyS2
    
    helios64 login: 
     

     

  16. @aprayoga if you still need it here's a full boot log of the crash (the actual stacktrace of the crash is inconsistent for me)

     

    Spoiler

    DDR Version 1.24 20191016

    In

    A: 0x1804

    SecureInit read PBA: 0x1c04

    SecureInit ret = 0, SecureMode = 0

    atags_set_bootdev: ret:(0)

    GPT 0x3380ec0 signature is wrong

    recovery gpt...

    GPT 0x3380ec0 signature is wrong

    recovery gpt fail!

    LoadTrust Addr:0x4000

    No find bl30.bin

    No find bl32.bin

    Load uboot, ReadLba = 2000

    Load OK, addr=0x200000, size=0xdcfd8

    RunBL31 0x40000

    NOTICE:  BL31: v1.3(debug):42583b6

    NOTICE:  BL31: Built : 07:55:13, Oct 15 2019

    NOTICE:  BL31: Rockchip release version: v1.1

    INFO:    GICv3 with legacy support detected. ARM GICV3 driver initialized in EL3

    INFO:    Using opteed sec cpu_context!

    INFO:    boot cpu mask: 0

    INFO:    plat_rockchip_pmu_init(1190): pd status 3e

    INFO:    BL31: Initializing runtime services

    WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK

    ERROR:   Error initializing runtime service opteed_fast

    INFO:    BL31: Preparing for EL3 exit to normal world

    INFO:    Entry point address = 0x200000

    INFO:    SPSR = 0x3c9

     

     

    U-Boot 2020.07-armbian (Oct 13 2020 - 16:49:59 +0200)

     

    SoC: Rockchip rk3399

    Reset cause: POR

    DRAM:  3.9 GiB

    PMIC:  RK808

    SF: Detected w25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB

    MMC:   mmc@fe320000: 1, sdhci@fe330000: 0

    Loading Environment from MMC... scanning bus for devices...

    Hit any key to stop autoboot:  0

    Card did not respond to voltage select!

    switch to partitions #0, OK

    mmc0(part 0) is current device

    Scanning mmc 0:1...

    Found U-Boot script /boot/boot.scr

    27236864 bytes read in 2646 ms (9.8 MiB/s)

     

    Starting kernel ...

     

    [   16.291853] systemd[1]: Failed to start Import ZFS pools by cache file.

    [   16.768236] systemd[1]: Failed to start Set console font and keymap.

    ckchipdrm analogix_dp dw_hdmi dw_mipi_dsi drm_kms_helper cec rc_core drm adc_keys drm_panel_orientation_quirks

    [   23.935692] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G         C        5.8.16-rockchip64 #20.08.14

    [   23.936474] Hardware name: Helios64 (DT)

    [   23.936827] pstate: 80000085 (Nzcv daIf -PAN -UAO BTYPE=--)

    [   23.937336] pc : __update_load_avg_se+0x64/0x308

    [   23.937751] lr : update_load_avg+0x278/0x5e0

    [   23.938131] sp : ffff800011abbd80

    [   23.938428] x29: ffff800011abbd80 x28: 0000000000000001

    [   23.938901] x27: ffff0000f6ea5700 x26: ffff800011abc000

    [   23.939373] x25: ffff8000114f1d20 x24: 0000000000000000

    [   23.939846] x23: ffff0000f77a7b40 x22: 0000000000000005

    [   23.940317] x21: 000000058fae126f x20: ffff0000f61c80c0

    [   23.940789] x19: ffff0000f61c80c0 x18: 0000000000000000

    [   23.941261] x17: 0000000000000000 x16: 0000000000000000

    [   23.941733] x15: 0000000000000000 x14: 0000000000000000

    [   23.942205] x13: 0000000000000332 x12: 0000000000000339

    [   23.942676] x11: 0000000000000001 x10: 0000000000000a20

    [   23.943148] x9 : ffff800011c53e70 x8 : 0000000000000000

    [   23.943620] x7 : 0000000000000000 x6 : 0000000019d0a216

    [   23.944092] x5 : 0000000000000000 x4 : 000000000000005e

    [   23.944563] x3 : 0000000000000000 x2 : 000000058fae1000

    [   23.945034] x1 : ffff0000f77a7bc0 x0 : 000000000000005e

    [   23.945508] Call trace:

    [   23.945734]  __update_load_avg_se+0x64/0x308

    [   23.946118]  update_load_avg+0x278/0x5e0

    [   23.946472]  enqueue_entity+0x68/0x228

    [   23.946809]  enqueue_task_fair+0x78/0x4d8

    [   23.947171]  ttwu_do_activate+0x84/0xe8

    [   23.947516]  sched_ttwu_pending+0xc0/0x168

    [   23.947884]  flush_smp_call_function_queue+0xec/0x258

    [   23.948334]  generic_smp_call_function_single_interrupt+0x14/0x20

    [   23.948879]  handle_IPI+0x258/0x3e8

    [   23.949195]  gic_handle_irq+0x154/0x158

    [   23.949539]  el1_irq+0xb8/0x180

    [   23.949824]  arch_cpu_idle+0x28/0x218

    [   23.950158]  default_idle_call+0x1c/0x44

    [   23.950510]  do_idle+0x210/0x288

    [   23.950801]  cpu_startup_entry+0x24/0x68

    [   23.951155]  secondary_start_kernel+0x140/0x178

    [   23.951564] Code: 2a0403e0 8b030042 f9402023 f9006262 (b9401cc1)

    [   23.952116] ---[ end trace fd422e59e1cb21cd ]---

    [   23.952527] Kernel panic - not syncing: Fatal exception in interrupt

    [   23.953092] SMP: stopping secondary CPUs

    [   23.953448] Kernel Offset: disabled

    [   23.953763] CPU features: 0x240022,2000600c

    [   23.954135] Memory Limit: none

    [   23.954422] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

     

    Spoiler

    channel 0

    CS = 0

    MR0=0x18

    MR4=0x1

    MR5=0x1

    MR8=0x10

    MR12=0x72

    MR14=0x72

    MR18=0x0

    MR19=0x0

    MR24=0x8

    MR25=0x0

    channel 1

    CS = 0

    MR0=0x18

    MR4=0x1

    MR5=0x1

    MR8=0x10

    MR12=0x72

    MR14=0x72

    MR18=0x0

    MR19=0x0

    MR24=0x8

    MR25=0x0

    channel 0 training pass!

    channel 1 training pass!

    change freq to 416MHz 0,1

    Channel 0: LPDDR4,416MHz

    Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB

    Channel 1: LPDDR4,416MHz

    Bus Width=32 Col=10 Bank=8 Row=16 CS=1 Die Bus-Width=16 Size=2048MB

    256B stride

    channel 0

    CS = 0

    MR0=0x18

    MR4=0x1

    MR5=0x1

    MR8=0x10

    MR12=0x72

    MR14=0x72

    MR18=0x0

    MR19=0x0

    MR24=0x8

    MR25=0x0

    channel 1

    CS = 0

    MR0=0x18

    MR4=0x1

    MR5=0x1

    MR8=0x10

    MR12=0x72

    MR14=0x72

    MR18=0x0

    MR19=0x0

    MR24=0x8

    MR25=0x0

    channel 0 training pass!

    channel 1 training pass!

    channel 0, cs 0, advanced training done

    channel 1, cs 0, advanced training done

    change freq to 856MHz 1,0

    ch 0 ddrconfig = 0x101, ddrsize = 0x40

    ch 1 ddrconfig = 0x101, ddrsize = 0x40

    pmugrf_os_reg[2] = 0x32C1F2C1, stride = 0xD

    ddr_set_rate to 328MHZ

    ddr_set_rate to 666MHZ

    ddr_set_rate to 928MHZ

    channel 0, cs 0, advanced training done

    ll return SMC_UNK

    ERROR:   Error initializing runtime service opteed_fast

    INFO:    BL31: Preparing for EL3 exit to normal world

    INFO:    Entry point address = 0x200000

    INFO:    SPSR = 0x3c9

     

     

    U-Boot 2020.07-armbian (Oct 13 2020 - 16:49:59 +0200)

     

    SoC: Rockchip rk3399

    Reset cause: POR

    DRAM:  3.9 GiB

    PMIC:  RK808

    eth0: ethernet@fe300000

    scanning bus for devices...

    Hit any key to stop autoboot:  0

    Card did not respond to voltage select!

    Scanning mmc 0:1...

    Found U-Boot script /boot/boot.scr

    3185 bytes read in 18 ms (171.9 KiB/s)

    ## Executing script at 00500000

    Boot script loaded from mmc 0

    166 bytes read in 12 ms (12.7 KiB/s)

    16311106 bytes read in 1601 ms (9.7 MiB/s)

    27236864 bytes read in 2645 ms (9.8 MiB/s)

    OK

    ## Flattened Device Tree blob at 01f00000

       Booting using the fdt blob at 0x1f00000

       Loading Ramdisk to f4f58000, end f5ee6302 ... OK

       Loading Device Tree to 00000000f4edc000, end 00000000f4f57fff ... OK

    [   15.396135] OF: graph: no port node found in /syscon@ff770000/usb2-phy@e450/otg-port

    [   16.304499] systemd[1]: Failed to start Import ZFS pools by cache file.

    [   16.784363] systemd[1]: Failed to start Set console font and keymap.

    [   21.126776] kernel BUG at arch/arm64/kernel/traps.c:470!

    [   21.127260] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP

    [   21.127750] Modules linked in: rfkill governor_performance zstd zram r8152 snd_soc_hdmi_codec leds_pwm gpio_charger pwm_fan snd_soc_rockchip_i2s snd_soc_core snd_pcm_dmaengine panfrost snd_pcm snd_timer snd soundcore gpu_sched sg rockchip_rga hantro_vpu(C) rockchip_vdec(C) videobuf2_dma_sg v4l2_h264 videobuf2_dma_contig videobuf2_vmalloc v4l2_mem2mem videobuf2_memopsx5 : 00000000d5300000 x4 : ffff800011806118

    [   21.142542] x3 : 00000000d6500000 x2 : 0000000000000002

    [   21.143014] x1 : ffff0000f6ea6580 x0 : 0000000040000085

    [   21.143488] Call trace:

    [   21.143717]  do_undefinstr+0x2ec/0x310

    [   21.144061]  el1_sync_handler+0x88/0x110

    [   21.144414]  el1_sync+0x7c/0x100

    [   21.144713]  check_preemption_disabled+0x48/0x108

    [   21.145143]  irq_exit+0x70/0xe0

    [   21.145434]  handle_IPI+0x25c/0x3e8

    [   21.145758]  gic_handle_irq+0x154/0x158

    [   21.146103]  el1_irq+0xb8/0x180

    [   21.146386]  arch_cpu_idle+0x28/0x218

    [   21.146721]  default_idle_call+0x1c/0x44

    [   21.147082]  do_idle+0x210/0x288

    [   21.147373]  cpu_startup_entry+0x28/0x68

    [   21.147725]  secondary_start_kernel+0x140/0x178

    [   21.148137] Code: f9401bf7 17ffff7d a9025bf5 f9001bf7 (d4210000)

    [   21.148691] ---[ end trace 9f0e999f37a83ba9 ]---

    [   21.149105] Kernel panic - not syncing: Attempted to kill the idle task!

    [   21.149704] SMP: stopping secondary CPUs

    [   21.150064] Kernel Offset: disabled

    [   21.150378] CPU features: 0x240022,2000600c

    [   21.150750] Memory Limit: none

    [   21.151036] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

     

  17. If you, like myself, installed on eMMC and are experiencing the crashes on 20.08.14 - I booted up via a 20.08.10 sdcard and fixed the environment on emmc

     

    # mount the emmc + get ready to chroot
    mkdir /mnt/chroot
    mount /dev/mmcblk1p2 /mnt/chroot/
    mount /dev/mmcblk1p1 /mnt/chroot/media/mmcboot
    mount --bind /mnt/chroot/media/mmcboot/boot/ /mnt/chroot/boot/
    mount --bind /dev /mnt/chroot/dev/
    mount --bind /proc /mnt/chroot/proc/
    mount --bind /tmp /mnt/chroot/tmp/
    
    # chroot in and downgrade to 20.08.10
    chroot /mnt/chroot/ /bin/bash
    apt install \
      linux-dtb-current-rockchip64=20.08.10 \
      linux-headers-current-rockchip64=20.08.10 \
      linux-image-current-rockchip64=20.08.10 \
      armbian-config=20.08.10 \
      armbian-firmware=20.08.10 \
      linux-focal-root-current-helios64=20.08.10 \
      linux-u-boot-helios64-current=20.08.10
    
    exit
    # now remove the sd card and hit reset

     

    @aprayoga It's probably unrelated, but while working through the above I noticed that I ran out of space on /boot.

    I installed to eMMC the first version that was working, if that helps. I chose f2fs when I installed on eMMAC and this is the resulting partition layout

    mmcblk1      179:32   0 14.6G  0 disk 
    ├─mmcblk1p1  179:33   0   96M  0 part
    └─mmcblk1p2  179:34   0 14.3G  0 part
    mmcblk1boot0 179:64   0    4M  1 disk 
    mmcblk1boot1 179:96   0    4M  1 disk

    Sadly I didn't grab enough info from what was in the boot partition before I nuked it and reinstalled the appropriate packages.

     

  18. These won't be exact instructions, since I decided to switch to focal (mostly for other reasons).

     

    mkdir zfs-scratch
    cd zfs-scratch
    
    apt-get download linux-headers-current-rockchip64
    git clone -b zfs-0.8.5 https://github.com/openzfs/zfs.git 
    
    docker run --rm -it -v $(pwd):/scratch ubuntu:focal
    
    # inside the container
    cd /scratch
    apt update
    apt install build-essential autoconf automake bison flex libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev
    dpkg -i linux-headers-current-*.deb
    
    cd zfs
    sh autogen.sh
    ./configure
    make -s -j$(nproc) deb

     

    At that point you can exit the container (it'l vanish because of the --rm) and inside zfs-scratch/zfs you should have a bunch of Debian packages you can install

     

  19. On 10/8/2020 at 1:09 AM, antsu said:

    Just tried building 0.8.5 on the Buster current image.  Unfortunately is still fails with the "__stack_chk_guard" errors.

    This topic and this GitHub issue (by the same author) have more info about the possible cause for this.

     

    In the meantime, I built a zfs 0.8.5 module that you can use on buster (but only the 5.8.13 kernel)

    First install this package with dpkg -i kmod-zfs-5.8.13-rockchip64_0.8.5-1_arm64.deb and then install the zfs utils with apt install -t buster-backports zfsutils-linux

     

  20. @Brocklobsta sounds like you're setting GCC correctly, and the issue thats left is probably that the zfs-dkms module is too old (try finding 0.8.4+ per here)

     

    For the issue on buster, I think I've nailed the problem down.

     

    STACKPROTECTOR_PER_TASK is defined as such

    config CC_HAVE_STACKPROTECTOR_SYSREG
       def_bool $(cc-option,-mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0)
    
    config STACKPROTECTOR_PER_TASK
       def_bool y
       depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_SYSREG

     

    GCC in the focal build environment supports the options required for CC_HAVE_STACKPROTECTOR_SYSREG

    $ gcc --version
    gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0
    
    $ gcc -mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0
    gcc: fatal error: no input files
    compilation terminated.

     

    Whereas the gcc version buster doesn't support these.

    $ gcc --version
    gcc (Debian 8.3.0-6) 8.3.0
    
    $ gcc -mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0
    gcc: error: unrecognized command line option '-mstack-protector-guard=sysreg'; did you mean '-fstack-protector-strong'?
    gcc: error: unrecognized command line option '-mstack-protector-guard-reg=sp_el0'; did you mean '-fstack-protector-all'?
    gcc: error: unrecognized command line option '-mstack-protector-guard-offset=0'; did you mean '-fstack-protector-strong'?
    gcc: fatal error: no input files
    compilation terminated.

     

    Since the kernel is built for both focal + buster, should features like this be disabled or are there some other workarounds?

  21. The suggestion Igor made in that thread solves the 0.8.3 problems on 5.8 because certain kernel methods were flagged as GPL and the module won't compile.

    As others in that thread pointed out, using the newer 0.8.4 package works on focal but not buster.

     

    0.8.4 compiles correctly against the rockchip64 kernel on focal (gcc9) and bullseye (gcc10), but not with buster (gcc8).

    2.0.0-rc follows the same pattern.

     

    the rockchip64 kernel is built using the armbian toolchain, which is focal based and thus using gcc 9.

     

    I haven't been able to come up with any other theory than how the compiler is detecting if it should use CONFIG_STACKPROTECTOR or CONFIG_STACKPROTECTOR_PER_TASK.

     

    It's only a theory, and I don't know how to prove or disprove it, short of building the same 5.8.13 kernel with gcc 8 and trying to build the module against it (which is probably my next step)

×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines