Jump to content

tinkerboard S's bricked after 5.70 upgrade

AdamD

By AdamD
in Tinkerboard

 Share

Recommended Posts

AdamD

AdamD

Posted
Posted

Hi all,

 

first post here so please forgive me if I've get any bits wrong.  Thanks for all the work put into supporting this project, and thanks in advance for any and all considerations extended in helping me step through my issue here. 

 

so i've had an 8 node kubernetes cluster running fairly smoothly on my tb's for nearly a full year now, so far I have not had any major issues with armbian updates until this weekends...  I did a round of updates to my cluster and, after rebooting all 3 of my model S tinkerboards seem bricked (redlight only, no output to HDMI)..  The remaining five orig TB models with Samsung EVO SD's all came up without issue...  since my k8s cluster master is one of the S models i'm really hoping there is someway somebody knows to quickly back out and salvage these machines without a full cluster rebuild...  I was thinking there might be an option because I can connect the boards via usb to pc and I am able to view the drive contents... just not sure where to look to fix a u-boot issue or if it's even possible this way.    I've just ordered a UART->usb adapter as it looks like I'm going to need one now,  but it won't be here until mid-week.    Im not sure what other information you might need or that i can get from the boards for you, but I've also attached the last console log of the machines at time of final reboot, HTH..

 

I'm guessing my questions at this point are:

  

1) Any known issues or anyone else having issues with S models after the last update to 5.70 ? 

2) Any other ideas on how I can debug or recover these machines before I receive a UART adapter?

3) Any other references to point me at to further debug this issue properly?

 

best regards,

-ad

 

 

Spoiler

Welcome to ARMBIAN 5.60 stable Debian GNU/Linux 9 (stretch) 4.14.70-rockchip
System load:   0.53 0.68 0.70   Up time:       36 days
Memory usage:  59 % of 2006MB   IP:            xxx.xxx.xxx.xxx
CPU temp:      35°C
Usage of /:    37% of 15G

xxxx@yyyy:~$ sudo apt update
[sudo] password for xxxx:
Get:1 http://security.debian.org stretch/updates InRelease [94.3 kB]
Ign:3 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Get:5 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:6 https://download.docker.com/linux/debian stretch InRelease [44.8 kB]
Get:7 http://cdn-fastly.deb.debian.org/debian stretch-backports InRelease [91.8 kB]
Hit:8 http://cdn-fastly.deb.debian.org/debian stretch Release
Hit:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Get:4 https://apt.armbian.com stretch InRelease [19.0 kB]
Get:9 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages.diff/Index [27.8 kB]
Get:10 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Contents (deb).diff/Index [28.0 kB]
Get:11 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Packages.diff/Index [12.5 kB]
Get:12 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Contents (deb).diff/Index [9,592 B]
Get:13 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages 2019-01-13-0212.27.pdiff [293 B]
Get:15 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages 2019-01-13-0810.11.pdiff [215 B]
Get:15 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages 2019-01-13-0810.11.pdiff [215 B]
Get:16 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Contents (deb) 2019-01-13-0212.27.pdiff [304 B]
Get:16 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Contents (deb) 2019-01-13-0212.27.pdiff [304 B]
Get:17 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Packages 2019-01-12-2012.01.pdiff [227 B]
Get:17 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Packages 2019-01-12-2012.01.pdiff [227 B]
Get:18 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Contents (deb) 2019-01-12-2012.01.pdiff [271 B]
Get:18 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Contents (deb) 2019-01-12-2012.01.pdiff [271 B]
Get:19 https://apt.armbian.com stretch/main armhf Packages [322 kB]
Get:20 https://apt.armbian.com stretch/stretch-utils armhf Packages [4,893 B]
Fetched 747 kB in 7s (104 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
5 packages can be upgraded. Run 'apt list --upgradable' to see them.
xxxx@yyyy:~$ sudo apt upgrade -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  linux-dtb-next-rockchip linux-image-next-rockchip linux-libc-dev
  linux-stretch-root-next-tinkerboard linux-u-boot-tinkerboard-next
5 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 20.8 MB of archives.
After this operation, 26.9 MB of additional disk space will be used.
Get:1 https://apt.armbian.com stretch/main armhf linux-dtb-next-rockchip armhf 5.70 [84.5 kB]
Get:2 https://apt.armbian.com stretch/main armhf linux-image-next-rockchip armhf 5.70 [18.8 MB]
Get:3 https://apt.armbian.com stretch/main armhf linux-libc-dev armhf 5.70 [1,007 kB]
Get:4 https://apt.armbian.com stretch/main armhf linux-stretch-root-next-tinkerboard armhf 5.70 [425 kB]
Get:5 https://apt.armbian.com stretch/main armhf linux-u-boot-tinkerboard-next armhf 5.70 [489 kB]
Fetched 20.8 MB in 4s (4,641 kB/s)
(Reading database ... 31727 files and directories currently installed.)
Preparing to unpack .../linux-dtb-next-rockchip_5.70_armhf.deb ...
Unpacking linux-dtb-next-rockchip (5.70) over (5.60) ...
Preparing to unpack .../linux-image-next-rockchip_5.70_armhf.deb ...
update-initramfs: Deleting /boot/initrd.img-4.14.70-rockchip
Removing obsolete file uInitrd-4.14.70-rockchip
Unpacking linux-image-next-rockchip (5.70) over (5.60) ...
Preparing to unpack .../linux-libc-dev_5.70_armhf.deb ...
Unpacking linux-libc-dev (5.70) over (4.9.130-2) ...
Preparing to unpack .../linux-stretch-root-next-tinkerboard_5.70_armhf.deb ...
Leaving 'diversion of /etc/mpv/mpv.conf to /etc/mpv/mpv-dist.conf by linux-stretch-root-next-tinkerboard'
Unpacking linux-stretch-root-next-tinkerboard (5.70) over (5.60) ...
Preparing to unpack .../linux-u-boot-tinkerboard-next_5.70_armhf.deb ...
Unpacking linux-u-boot-tinkerboard-next (5.70) over (5.60) ...
Setting up linux-image-next-rockchip (5.70) ...
update-initramfs: Generating /boot/initrd.img-4.19.14-rockchip
update-initramfs: Converting to u-boot format
Setting up linux-u-boot-tinkerboard-next (5.70) ...
Setting up linux-libc-dev (5.70) ...
Processing triggers for initramfs-tools (0.130) ...
update-initramfs: Generating /boot/initrd.img-4.19.14-rockchip
update-initramfs: Converting to u-boot format
Setting up linux-stretch-root-next-tinkerboard (5.70) ...
Setting up linux-dtb-next-rockchip (5.70) ...
Processing triggers for initramfs-tools (0.130) ...
update-initramfs: Generating /boot/initrd.img-4.19.14-rockchip
update-initramfs: Converting to u-boot format
xxxx@yyyy:~$ sudo reboot

 

........
 

 

Link to comment
Share on other sites
AdamD

AdamD

Posted
  • Author
Posted
2 minutes ago, TonyMac32 said:

 

Sounds like it's not getting past u-boot.  You say the non- eMMC Tinkerboards were fine after upgrade?

 

yes. all 5 of the non-eMMC are functioning fine so far.

Link to comment
Share on other sites
TonyMac32

TonyMac32

Posted
Posted
1 hour ago, AdamD said:

yes. all 5 of the non-eMMC are functioning fine so far.

OK, mine is wanting to see "rk3288-miniarm.dtb" and can't find it.  We had a symlink there, it must have gotten lost, but why that isn't angering the SD cards I'm not sure...

 

if you edit /boot/armbianEnv.txt to read "fdt_file=rk3288-tinker.dtb" instead of "fdt_file=rk3288-miniarm.dtb" you are a step closer, but mine is hanging after a few seconds, can you verify that behavior?

 

I am trying something else quickly.

 

@Igor does u-boot not get updated on eMMC?  My board is still listing the 2018.05 revision after the update

Link to comment
Share on other sites
AdamD

AdamD

Posted
  • Author
Posted
14 minutes ago, TonyMac32 said:

if you edit /boot/armbianEnv.txt to read "fdt_file=rk3288-tinker.dtb" instead of "fdt_file=rk3288-miniarm.dtb" you are a step closer, but mine is hanging after a few seconds, can you verify that behavior?

I am trying something else quickly.

 

Thank you!

Link to comment
Share on other sites
Igor

Igor

Posted
Posted
15 minutes ago, TonyMac32 said:

I'll push this patch, basically creating this device tree for the older u-boot systems. 

rk3288-miniarm.dtb


So I need to push out an updated kernel / dtb pack for rockchip-next? ... U-boot is now updated manually only. Which solves other problems .. but I guess we have some new ones :)

Link to comment
Share on other sites
AdamD

AdamD

Posted
  • Author
Posted

 

 

# DO NOT EDIT THIS FILE

#

# Please edit /boot/armbianEnv.txt to set supported parameters

#

 

setenv rootdev "/dev/mmcblk0p1"

setenv rootfstype "ext4"

setenv fdt_file "rk3288-miqi.dtb"

setenv ramdisk_addr_r "0x21000000"

setenv console "ttyS2,115200n8 console=tty1"

setenv verbosity "1"

 

itest.b ${devnum} == 0 && echo "U-boot loaded from SD"

itest.b ${devnum} == 1 && echo "U-boot loaded from eMMC"

 

if load ${devtype} ${devnum}:1 ${ramdisk_addr_r} /boot/armbianEnv.txt || load ${devtype} ${devnum}:1 ${ramdisk_addr_r} armbianEnv.txt; then

    env import -t ${ramdisk_addr_r} ${filesize}

fi

 

setenv bootargs "consoleblank=0 scandelay root=${rootdev} rw console=${console} rootfstype=${rootfstype} loglevel=${verbosity} rootwait usb-storage.quirks=${usbstoragequirks} ${extraargs}"

ext4load ${devtype} ${devnum}:1 ${fdt_addr_r} /boot/dtb/${fdt_file} || fatload ${devtype} ${devnum}:1 ${fdt_addr_r} dtb/${fdt_file} || ext4load ${devtype} ${devnum}:1 ${fdt_addr_r} dtb/${fdt_file}

ext4load ${devtype} ${devnum}:1 ${ramdisk_addr_r} /boot/uInitrd || fatload ${devtype} ${devnum}:1 ${ramdisk_addr_r} uInitrd || ext4load ${devtype} ${devnum}:1 ${ramdisk_addr_r} uInitrd

ext4load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/zImage || fatload ${devtype} ${devnum}:1 ${kernel_addr_r} zImage || ext4load ${devtype} ${devnum}:1 ${kernel_addr_r} zImage

bootz ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}

# mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr

 

Link to comment
Share on other sites
AdamD

AdamD

Posted
  • Author
Posted
Just now, TonyMac32 said:

 

That looks the same, now, did you reboot it with the USB attached to the PC still (crazy question, but if I don't ask...)

 

no, unplugged, replugged..  tried on 2 of the 3 units before reply as well.. :(

Link to comment
Share on other sites
TonyMac32

TonyMac32

Posted
Posted

no SD card in the slot, right?  Otherwise I'll try to install a fresh 4.14 and repeat the process again and see what happens tomorrow my time (is 1:30 here, I have to sleep.  ;)).  If I can't reproduce it, then we will have to wait on your USB-UART adapter

Link to comment
Share on other sites
AdamD

AdamD

Posted
  • Author
Posted
5 minutes ago, TonyMac32 said:

 Maybe check to be sure the file has the right file listed.

ah your right,  the file i transferred over didn't keep.. .. i just reconnected the unit and the files missing again..  will try again...  and no sd in these units.

Thanks for the help! will pick up tomorrow 

Link to comment
Share on other sites
Igor

Igor

Posted
Posted

I could reproduce this strange problem on my Tinkerboard S, but don't have a solution yet. My five cents:

1. Started with https://dl.armbian.com/tinkerboard/archive/Armbian_5.59_Tinkerboard_Debian_stretch_next_4.14.67.7z

2. Boots up fine. apt update and upgrade + reboot

3. It display error about not finding miniarm DTB but boots up.

4. Correcting DTB and it doesn't boot at all. It stuck at "loading kernel"

5. Repeated the same with updated u-boot. It also stucks.
6. Loaded https://dl.armbian.com/tinkerboard/archive/Armbian_5.70_Tinkerboard_Debian_stretch_next_4.19.14.7z and it boots up fine.

Link to comment
Share on other sites
TonyMac32

TonyMac32

Posted
Posted
I could reproduce this strange problem on my Tinkerboard S, but don't have a solution yet. My five cents:

1. Started with https://dl.armbian.com/tinkerboard/archive/Armbian_5.59_Tinkerboard_Debian_stretch_next_4.14.67.7z
2. Boots up fine. apt update and upgrade + reboot
3. It display error about not finding miniarm DTB but boots up.
4. Correcting DTB and it doesn't boot at all. It stuck at "loading kernel"
5. Repeated the same with updated u-boot. It also stucks.
6. Loaded https://dl.armbian.com/tinkerboard/archive/Armbian_5.70_Tinkerboard_Debian_stretch_next_4.19.14.7z and it boots up fine.
If you don't pull the SD card it will fall back to SD, reload the boot script from there, and boot. Probably how the issue escaped in the first place.

Sent from my Pixel using Tapatalk

Link to comment
Share on other sites
TonyMac32

TonyMac32

Posted
Posted

@igor the solution that worked on mine was simply dropping the miniarm DTB into the directory and rebooting with the jumper off and the SD out. It sounds like that worked for the OP as well, I think it's safe to push the update.

Sent from my Pixel using Tapatalk

Link to comment
Share on other sites
Igor

Igor

Posted
Posted
10 minutes ago, TonyMac32 said:

If you don't pull the SD card it will fall back to SD, reload the boot script from there, and boot. Probably how the issue escaped in the first place.


I was doing this on eMMC. I will do a few more tests.

Link to comment
Share on other sites
TonyMac32

TonyMac32

Posted
Posted

I was doing this on eMMC. I will do a few more tests.
Ok. I'm just relating what I saw last night, changes that failed:

-editing armbianEnv locked up after a few seconds at loading kernel.
-symlink failed same as above


-I forgot to pull the "boot from SD" jumper and it was booting ok after massive errors because of overlays and pulling the SD script after the eMMC one failed.

Sent from my Pixel using Tapatalk

Link to comment
Share on other sites
AdamD

AdamD

Posted
  • Author
Posted

@tonymac32 & @igor -  I just wanted to follow up here as I believe I might have been mistaken regarding the SD card updates.

 

Although, the emmc models originally bricked right after first reboot after that 5.70 update,   the SD units did not... I thought the updates had completed successfully after getting a chance to check up further today, it appears i was wrong, and the SD models did not fully update as I had thought:

 

This is how the 5 SD units were reporting at time of update, 

Welcome to ARMBIAN 5.60 stable Debian GNU/Linux 9 (stretch) 4.4.156-rockchip

 

after now after update:
Welcome to ARMBIAN 5.70 stable Debian GNU/Linux 9 (stretch) 4.4.166-rockchip
 

where as, for the 3 emmc models, after the initial update, reboot, and fix provided by tony above, they now report:

 

Welcome to ARMBIAN 5.70 stable Debian GNU/Linux 9 (stretch) 4.19.14-rockchip

 

Is the 4.19.14 vs 4.4.166 difference expected for SD / emmc boot ?   or, could this be additional update issue on the SD models that i overlooked when dealing with the dead emmc's...?

 

 

Link to comment
Share on other sites
TonyMac32

TonyMac32

Posted
Posted

Your classic Tinkers were running "default" vendor kernel, your S's were running "Next". Next was updated to 4.19, default was not.
@igor I will try to repeat the fix tonight, I'm still thinking you did something differently than myself or the OP. The device tree for Tinker in 4.19 is set up for overlays and has a lot of things disabled, including the debug UART (maybe was bad idea, but it is enabled in the images by default via overlay). To fix old installs, I reintroduced the 4.14 device tree as "miniarm", which those installs were looking for anyway. That's what worked for me and the OP. It's not beautiful, but neither is u-boot in general.

Sent from my Pixel using Tapatalk

Link to comment
Share on other sites
TonyMac32

TonyMac32

Posted
Posted

@Igor just verified.  Flashed 4.14 from our download archive, then did an apt update/upgrade.  Bricked eMMC.  So then:

 

  1. Plugged Tinker into USB port on Linux Machine, it booted in UMS.
  2. As root on Linux machine, copied rk3288-miniarm.dtb from link above into /boot/dtb folder
  3. sync'd umounted/etc.
  4. Booted tinker with proper supply, booted normally.

I do notice the linux boot messages aren't showing up, it just says the starting kernel line and then after a bit shows the login prompt.  I can check that out, probably bootargs.

 

We can push the updated dtb package, which will include the miniarm dtb.

Link to comment
Share on other sites
Guest
This topic is now closed to further replies.
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept