ROCK Pi 4B Rock pi 4B boot failed after update

[MaxT]

Hi!

 

After update to kernel 5.15.75 via apt upgrade, boot fails.

 

Though I can flash an older image, I would prefer trying to recover existing setup and rollback to a previous kernel.

 

What would be the best approach, ie flash image to usb, boot from usb and what then?

 

I will appreciate any advice.

 

U-Boot 2022.07-armbian (Oct 18 2022 - 07:36:22 +0000)SoC: Rockchip rk3399Reset cause: PORModel: Radxa ROCK Pi 4BDRAM:  3.9 GiBPMIC:  RK808Core:  285 devices, 31 uclasses, devicetree: separateMMC:   mmc@fe310000: 2, mmc@fe320000: 1, mmc@fe330000: 0Loading Environment from SPIFlash... jedec_spi_nor flash@0: unrecognized JEDEC id bytes: ff, ff, ff*** Warning - spi_flash_probe_bus_cs() failed, using default environmentIn:    serialOut:   serialErr:   serialModel: Radxa ROCK Pi 4BNet:   eth0: ethernet@fe300000Hit any key to stop autoboot:  0starting USB...Bus usb@fe380000: ehci_generic usb@fe380000: Failed to get clocks (ret=-19)Port not available.Bus usb@fe3c0000: ehci_generic usb@fe3c0000: Failed to get clocks (ret=-19)Port not available.Bus usb@fe800000: Register 2000140 NbrPorts 2Starting the controllerUSB XHCI 1.10Bus usb@fe900000: Register 2000140 NbrPorts 2Starting the controllerUSB XHCI 1.10scanning bus usb@fe800000 for devices... 1 USB Device(s) foundscanning bus usb@fe900000 for devices... 1 USB Device(s) found      scanning usb for storage devices... 0 Storage Device(s) foundDevice 0: unknown deviceCard did not respond to voltage select! : -110switch to partitions #0, OKmmc0(part 0) is current deviceScanning mmc 0:1...Found U-Boot script /boot/boot.scr3185 bytes read in 19 ms (163.1 KiB/s)## Executing script at 00500000Boot script loaded from mmc 0266 bytes read in 15 ms (16.6 KiB/s)17307706 bytes read in 1665 ms (9.9 MiB/s)30540288 bytes read in 2916 ms (10 MiB/s)76240 bytes read in 48 ms (1.5 MiB/s)376 bytes read in 37 ms (9.8 KiB/s)Applying kernel provided DT overlay rockchip-rk3399-opp-2ghz.dtbo1314 bytes read in 44 ms (28.3 KiB/s)Applying kernel provided DT overlay rockchip-spi-jedec-nor.dtbo2698 bytes read in 39 ms (67.4 KiB/s)Applying kernel provided DT fixup script (rockchip-fixup.scr)## Executing script at 09000000Moving Image from 0x2080000 to 0x2200000, end=3fc0000## Loading init Ramdisk from Legacy Image at 06000000 ...  Image Name:   uInitrd  Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)  Data Size:    17307642 Bytes = 16.5 MiB  Load Address: 00000000  Entry Point:  00000000  Verifying Checksum ... OK## Flattened Device Tree blob at 01f00000  Booting using the fdt blob at 0x1f00000  Loading Ramdisk to f0e8a000, end f1f0b7fa ... OK  Loading Device Tree to 00000000f0e0f000, end 00000000f0e89fff ... OKStarting kernel ...efi_free_pool: illegal free 0x00000000f0f0d040efi_free_pool: illegal free 0x00000000f0f0a040[    3.806881] Internal error: implementation fault (unsupported exclusive): 96000035 [#1] PREEMPT SMP[    3.807711] Modules linked in:[    3.808001] CPU: 2 PID: 224 Comm: kworker/u12:4 Not tainted 5.15.75-rockchip64 #22.08.8[    3.808728] Hardware name: Radxa ROCK Pi 4B (DT)[    3.809152] Workqueue: events_unbound deferred_probe_work_func[    3.809698] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)[    3.810334] pc : ww_mutex_lock_interruptible+0x1a4/0x308[    3.810830] lr : phy_power_off+0x28/0xd8[    3.811200] sp : ffff80000a713ab0[    3.811505] x29: ffff80000a713ab0 x28: 00000000fffffffe x27: 0000000000a000c4[    3.812167] x26: 0000000000a000c4 x25: 0000000010624dd3 x24: 0000000000900004[    3.812828] x23: 0000000000900004 x22: 0000000000000000 x21: ffff80000f000300[    3.813489] x20: 00000000e2808257 x19: ffff80000f000000 x18: 0000000000000001[    3.814149] x17: 7469772074756f65 x16: 6d697420316e6567 x15: 00000000000001b5[    3.814810] x14: ffff80000a713730 x13: 00000000ffffffea x12: ffff800009b1fd10[    3.815472] x11: 0000000000000003 x10: ffff800009b07cd0 x9 : ffff800009b07d28[    3.816133] x8 : 0000000000017fe8 x7 : c0000000ffffefff x6 : 0000000000000001[    3.816793] x5 : 0000000000057fa8 x4 : 0000000000000000 x3 : ffff80000f000300[    3.817453] x2 : ffff00000120d880 x1 : 0000000000000000 x0 : ffff80000f000300[    3.818113] Call trace:[    3.818341]  ww_mutex_lock_interruptible+0x1a4/0x308[    3.818803]  phy_power_off+0x28/0xd8[    3.819142]  rockchip_pcie_host_init_port+0x564/0x5a0[    3.819616]  rockchip_pcie_probe+0xf8/0x4c0[    3.820011]  platform_probe+0x68/0xd8[    3.820359]  really_probe+0xbc/0x428[    3.820696]  __driver_probe_device+0x114/0x188[    3.821111]  driver_probe_device+0xb0/0x110[    3.821502]  __device_attach_driver+0xbc/0x158[    3.821917]  bus_for_each_drv+0x7c/0xd0[    3.822284]  __device_attach+0xec/0x1a8[    3.822644]  device_initial_probe+0x14/0x20[    3.823036]  bus_probe_device+0x9c/0xa8[    3.823396]  deferred_probe_work_func+0x9c/0xf0[    3.823817]  process_one_work+0x20c/0x4c8[    3.824199]  worker_thread+0x208/0x478[    3.824554]  kthread+0x138/0x150[    3.824863]  ret_from_fork+0x10/0x20[    3.825211] Code: c8017e60 35ffffa1 17fffd08 f9800071 (c85ffc60)[    3.825769] ---[ end trace a4970fb3a1b23494 ]---

[Igor]

Upgrade was disabled. Investigation is in progress ... some board works, some don't.

[psmedley]

Hey all,

 

Just a heads up - I ran 'apt upgrade' today and there were some new kernel updates - 5.15.75. Both my RockPi 4B's failed to boot after applying these updates.

 

Booting off an SD card, and copying the kernel components of 5.15.63 from  /boot of the SD card to /boot of the EMMC, and removing the 5.15.75 components got me back up and running.

 

Cheers

 

Paul

[Igor]

Fixed upgrade component is coming up ASAP. Just doing last tests.

[ccar]

hi all, same here, i was running sid bleeding edge on rock 4B and after apt upgrade this morning it doesn't boot anymore... if i remember correctly, bleeding edge moved to kernel 6 this morning

i'll keep an eye on this thread, thanks

[fier]

I can confirm that the latest Armbian kernel (6.0.6) boots just fine on rockpi 4a (and 6.0.5 did not).

[psmedley]

My fix was:

• flash an image to an SD Card
• boot from the SD card
• mount the emmc drive
• copy the kernel files from /boot on the SD card to /(emmc mount point)/boot
• remove SD card and reboot

[fier]

My rootfs is on the sdcard, I've taken the card out, booted an old laptop on https://www.system-rescue.org/ and using an SD card reader copied /boot and /lib/modules from a known-working Armbian image to the card. Boot, then apt update/apt upgrade to get the fixed kernel. I've also started backing up the current kernel every time I run an upgrade 🙂 

Edited October 31, 2022 by fier

[David Manouchehri]

Would one of you be able to help me troubleshoot with a serial console on your Rock Pi 4B?

[MaxT]

Would one of you be able to help me troubleshoot with a serial console on your Rock Pi 4B?

Sure, please tell what is necessary

[denni_isl]

connect tx rx gnd vires  to corresponding gpio(not votage)

lsusb➜ 

sudo picocom -b 1500000 -r -l /dev/ttyUSB0                           

[MaxT]

Ok, I'm in

[David Manouchehri]

Do you see a message with "PCIe link training gen1 timeout" when you attempt to boot up with the kernel that was failing to boot earlier?

[MaxT]

I've already recovered that setup to 5.15.76, so to debug that will need to install than broken 5.15.75. Is it still in the repo?

Just in case, I've attached full boot log until kernel hang in the first post.

[David Manouchehri]

Where have you attached it? 😃 

[MaxT]

It is in the very first post in this topic

[MaxT]

There were nothing in the terminal after "3.825769] ---[ end trace a4970fb3a1b23494 ]---"

[David Manouchehri]

Odd. The dmesg you posted at first doesn't even seem to be hitting my patch.

[MaxT]

He the log is in txt

5.15.75-rockchip64.txt

[MaxT]

6 минут назад, David Manouchehri сказал:

Odd. The dmesg you posted at first doesn't even seem to be hitting my patch.

It is not dmesg, it is full terminal output of failing boot (tried several times), only cut memory init 

Edited October 31, 2022 by MaxT

[David Manouchehri]

Thanks. It's a head scratcher.. You aren't seeing the dev_err message being printed at all, so I don't see how my patch can impact you.

[MaxT]

Let me know if I can help somehow

[MaxT]

Though I see pci timeout in 5.15.76 and timing is more or less consistent with the hang of 5.15.75:

root@rockpi-4b:~# dmesg | grep PCI
[ 0.008978] PCI/MSI: /interrupt-controller@fee00000/interrupt-controller@fee20000 domain created
[ 1.335872] PCI: CLS 0 bytes, default 64
[ 2.660682] ehci-pci: EHCI PCI platform driver
[ 2.696246] ohci-pci: OHCI PCI platform driver
[ 3.582653] rockchip-pcie f8000000.pcie: PCIe link training gen1 timeout!
root@rockpi-4b:~#

[David Manouchehri]

Thank you, that's quite a useful bit of info there! Do you ever see a timeout message with 5.15.75 (if so, logs please?).

[MaxT]

With 5.15.75 I've posted everything - there were nothing after the end of trace

[fier]

Maybe you need to change:

int err, i = MAX_LANE_NUM, attempt_counter = 0;
u32 status;

err_retry_init:

to

int err, i, attempt_counter = 0;
u32 status;

err_retry_init:
i = MAX_LANE_NUM;

?

Or just change the "while"s into for loops.

Edited November 1, 2022 by fier