2 2
AdamD

tinkerboard S's bricked after 5.70 upgrade

Recommended Posts

Hi all,

 

first post here so please forgive me if I've get any bits wrong.  Thanks for all the work put into supporting this project, and thanks in advance for any and all considerations extended in helping me step through my issue here. 

 

so i've had an 8 node kubernetes cluster running fairly smoothly on my tb's for nearly a full year now, so far I have not had any major issues with armbian updates until this weekends...  I did a round of updates to my cluster and, after rebooting all 3 of my model S tinkerboards seem bricked (redlight only, no output to HDMI)..  The remaining five orig TB models with Samsung EVO SD's all came up without issue...  since my k8s cluster master is one of the S models i'm really hoping there is someway somebody knows to quickly back out and salvage these machines without a full cluster rebuild...  I was thinking there might be an option because I can connect the boards via usb to pc and I am able to view the drive contents... just not sure where to look to fix a u-boot issue or if it's even possible this way.    I've just ordered a UART->usb adapter as it looks like I'm going to need one now,  but it won't be here until mid-week.    Im not sure what other information you might need or that i can get from the boards for you, but I've also attached the last console log of the machines at time of final reboot, HTH..

 

I'm guessing my questions at this point are:

  

1) Any known issues or anyone else having issues with S models after the last update to 5.70 ? 

2) Any other ideas on how I can debug or recover these machines before I receive a UART adapter?

3) Any other references to point me at to further debug this issue properly?

 

best regards,

-ad

 

 

Spoiler

Welcome to ARMBIAN 5.60 stable Debian GNU/Linux 9 (stretch) 4.14.70-rockchip
System load:   0.53 0.68 0.70   Up time:       36 days
Memory usage:  59 % of 2006MB   IP:            xxx.xxx.xxx.xxx
CPU temp:      35°C
Usage of /:    37% of 15G

xxxx@yyyy:~$ sudo apt update
[sudo] password for xxxx:
Get:1 http://security.debian.org stretch/updates InRelease [94.3 kB]
Ign:3 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Get:5 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:6 https://download.docker.com/linux/debian stretch InRelease [44.8 kB]
Get:7 http://cdn-fastly.deb.debian.org/debian stretch-backports InRelease [91.8 kB]
Hit:8 http://cdn-fastly.deb.debian.org/debian stretch Release
Hit:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Get:4 https://apt.armbian.com stretch InRelease [19.0 kB]
Get:9 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages.diff/Index [27.8 kB]
Get:10 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Contents (deb).diff/Index [28.0 kB]
Get:11 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Packages.diff/Index [12.5 kB]
Get:12 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Contents (deb).diff/Index [9,592 B]
Get:13 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages 2019-01-13-0212.27.pdiff [293 B]
Get:15 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages 2019-01-13-0810.11.pdiff [215 B]
Get:15 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages 2019-01-13-0810.11.pdiff [215 B]
Get:16 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Contents (deb) 2019-01-13-0212.27.pdiff [304 B]
Get:16 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Contents (deb) 2019-01-13-0212.27.pdiff [304 B]
Get:17 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Packages 2019-01-12-2012.01.pdiff [227 B]
Get:17 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Packages 2019-01-12-2012.01.pdiff [227 B]
Get:18 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Contents (deb) 2019-01-12-2012.01.pdiff [271 B]
Get:18 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Contents (deb) 2019-01-12-2012.01.pdiff [271 B]
Get:19 https://apt.armbian.com stretch/main armhf Packages [322 kB]
Get:20 https://apt.armbian.com stretch/stretch-utils armhf Packages [4,893 B]
Fetched 747 kB in 7s (104 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
5 packages can be upgraded. Run 'apt list --upgradable' to see them.
xxxx@yyyy:~$ sudo apt upgrade -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  linux-dtb-next-rockchip linux-image-next-rockchip linux-libc-dev
  linux-stretch-root-next-tinkerboard linux-u-boot-tinkerboard-next
5 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 20.8 MB of archives.
After this operation, 26.9 MB of additional disk space will be used.
Get:1 https://apt.armbian.com stretch/main armhf linux-dtb-next-rockchip armhf 5.70 [84.5 kB]
Get:2 https://apt.armbian.com stretch/main armhf linux-image-next-rockchip armhf 5.70 [18.8 MB]
Get:3 https://apt.armbian.com stretch/main armhf linux-libc-dev armhf 5.70 [1,007 kB]
Get:4 https://apt.armbian.com stretch/main armhf linux-stretch-root-next-tinkerboard armhf 5.70 [425 kB]
Get:5 https://apt.armbian.com stretch/main armhf linux-u-boot-tinkerboard-next armhf 5.70 [489 kB]
Fetched 20.8 MB in 4s (4,641 kB/s)
(Reading database ... 31727 files and directories currently installed.)
Preparing to unpack .../linux-dtb-next-rockchip_5.70_armhf.deb ...
Unpacking linux-dtb-next-rockchip (5.70) over (5.60) ...
Preparing to unpack .../linux-image-next-rockchip_5.70_armhf.deb ...
update-initramfs: Deleting /boot/initrd.img-4.14.70-rockchip
Removing obsolete file uInitrd-4.14.70-rockchip
Unpacking linux-image-next-rockchip (5.70) over (5.60) ...
Preparing to unpack .../linux-libc-dev_5.70_armhf.deb ...
Unpacking linux-libc-dev (5.70) over (4.9.130-2) ...
Preparing to unpack .../linux-stretch-root-next-tinkerboard_5.70_armhf.deb ...
Leaving 'diversion of /etc/mpv/mpv.conf to /etc/mpv/mpv-dist.conf by linux-stretch-root-next-tinkerboard'
Unpacking linux-stretch-root-next-tinkerboard (5.70) over (5.60) ...
Preparing to unpack .../linux-u-boot-tinkerboard-next_5.70_armhf.deb ...
Unpacking linux-u-boot-tinkerboard-next (5.70) over (5.60) ...
Setting up linux-image-next-rockchip (5.70) ...
update-initramfs: Generating /boot/initrd.img-4.19.14-rockchip
update-initramfs: Converting to u-boot format
Setting up linux-u-boot-tinkerboard-next (5.70) ...
Setting up linux-libc-dev (5.70) ...
Processing triggers for initramfs-tools (0.130) ...
update-initramfs: Generating /boot/initrd.img-4.19.14-rockchip
update-initramfs: Converting to u-boot format
Setting up linux-stretch-root-next-tinkerboard (5.70) ...
Setting up linux-dtb-next-rockchip (5.70) ...
Processing triggers for initramfs-tools (0.130) ...
update-initramfs: Generating /boot/initrd.img-4.19.14-rockchip
update-initramfs: Converting to u-boot format
xxxx@yyyy:~$ sudo reboot

 

........
 

 

Share this post


Link to post
Share on other sites
5 minutes ago, AdamD said:

redlight only,

 

Sounds like it's not getting past u-boot.  You say the non- eMMC Tinkerboards were fine after upgrade?

Share this post


Link to post
Share on other sites
2 minutes ago, TonyMac32 said:

 

Sounds like it's not getting past u-boot.  You say the non- eMMC Tinkerboards were fine after upgrade?

 

yes. all 5 of the non-eMMC are functioning fine so far.

Share this post


Link to post
Share on other sites

OK, I'll have a look.  I haven't tried an in-place update on the eMMC, but the board in front of me is set up with 4.14 on eMMC.  Give me a couple minutes.

Share this post


Link to post
Share on other sites
1 hour ago, AdamD said:

yes. all 5 of the non-eMMC are functioning fine so far.

OK, mine is wanting to see "rk3288-miniarm.dtb" and can't find it.  We had a symlink there, it must have gotten lost, but why that isn't angering the SD cards I'm not sure...

 

if you edit /boot/armbianEnv.txt to read "fdt_file=rk3288-tinker.dtb" instead of "fdt_file=rk3288-miniarm.dtb" you are a step closer, but mine is hanging after a few seconds, can you verify that behavior?

 

I am trying something else quickly.

 

@Igor does u-boot not get updated on eMMC?  My board is still listing the 2018.05 revision after the update

Share this post


Link to post
Share on other sites
14 minutes ago, TonyMac32 said:

if you edit /boot/armbianEnv.txt to read "fdt_file=rk3288-tinker.dtb" instead of "fdt_file=rk3288-miniarm.dtb" you are a step closer, but mine is hanging after a few seconds, can you verify that behavior?

I am trying something else quickly.

 

Thank you!

Share this post


Link to post
Share on other sites
11 minutes ago, AdamD said:

Thank you! 

 

OK, don't edit anything, just copy this file into the "dtb" folder with the other device trees.

 

@Igor I'll push this patch, basically creating this device tree for the older u-boot systems. 

rk3288-miniarm.dtb

Share this post


Link to post
Share on other sites
8 minutes ago, TonyMac32 said:

 

OK, don't edit anything, just copy this file into the "dtb" folder with the other device trees.

rk3288-miniarm.dtb

 

so just copying rk3288-miniarm.dtb into /boot/dtb-4.19.14-rockchip folder and powering on is same red light.. another step?

Share this post


Link to post
Share on other sites
15 minutes ago, TonyMac32 said:

I'll push this patch, basically creating this device tree for the older u-boot systems. 

rk3288-miniarm.dtb


So I need to push out an updated kernel / dtb pack for rockchip-next? ... U-boot is now updated manually only. Which solves other problems .. but I guess we have some new ones :)

Share this post


Link to post
Share on other sites

 

 

# DO NOT EDIT THIS FILE

#

# Please edit /boot/armbianEnv.txt to set supported parameters

#

 

setenv rootdev "/dev/mmcblk0p1"

setenv rootfstype "ext4"

setenv fdt_file "rk3288-miqi.dtb"

setenv ramdisk_addr_r "0x21000000"

setenv console "ttyS2,115200n8 console=tty1"

setenv verbosity "1"

 

itest.b ${devnum} == 0 && echo "U-boot loaded from SD"

itest.b ${devnum} == 1 && echo "U-boot loaded from eMMC"

 

if load ${devtype} ${devnum}:1 ${ramdisk_addr_r} /boot/armbianEnv.txt || load ${devtype} ${devnum}:1 ${ramdisk_addr_r} armbianEnv.txt; then

    env import -t ${ramdisk_addr_r} ${filesize}

fi

 

setenv bootargs "consoleblank=0 scandelay root=${rootdev} rw console=${console} rootfstype=${rootfstype} loglevel=${verbosity} rootwait usb-storage.quirks=${usbstoragequirks} ${extraargs}"

ext4load ${devtype} ${devnum}:1 ${fdt_addr_r} /boot/dtb/${fdt_file} || fatload ${devtype} ${devnum}:1 ${fdt_addr_r} dtb/${fdt_file} || ext4load ${devtype} ${devnum}:1 ${fdt_addr_r} dtb/${fdt_file}

ext4load ${devtype} ${devnum}:1 ${ramdisk_addr_r} /boot/uInitrd || fatload ${devtype} ${devnum}:1 ${ramdisk_addr_r} uInitrd || ext4load ${devtype} ${devnum}:1 ${ramdisk_addr_r} uInitrd

ext4load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/zImage || fatload ${devtype} ${devnum}:1 ${kernel_addr_r} zImage || ext4load ${devtype} ${devnum}:1 ${kernel_addr_r} zImage

bootz ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}

# mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr

 

Share this post


Link to post
Share on other sites
Just now, Igor said:

U-boot is now updated manually only

I'll have to catch up on that.  This worked on mine, but has not worked on @AdamD's I want to verify the boot comands/etc

Share this post


Link to post
Share on other sites
Just now, TonyMac32 said:

I'll have to catch up on that.  This worked on mine, but has not worked on @AdamD's I want to verify the boot comands/etc

 

Is this my prob? 

setenv fdt_file "rk3288-miqi.dtb"

Share this post


Link to post
Share on other sites
3 minutes ago, AdamD said:

# DO NOT EDIT THIS FILE

 

That looks the same, now, did you reboot it with the USB attached to the PC still (crazy question, but if I don't ask...)  (This would get it stuck in UMS mode)

Share this post


Link to post
Share on other sites
Just now, AdamD said:

setenv fdt_file "rk3288-miqi.dtb"

No, the armbianEnv.txt file overrides it.  Maybe check to be sure the file has the right file listed.

Share this post


Link to post
Share on other sites
Just now, TonyMac32 said:

 

That looks the same, now, did you reboot it with the USB attached to the PC still (crazy question, but if I don't ask...)

 

no, unplugged, replugged..  tried on 2 of the 3 units before reply as well.. :(

Share this post


Link to post
Share on other sites

no SD card in the slot, right?  Otherwise I'll try to install a fresh 4.14 and repeat the process again and see what happens tomorrow my time (is 1:30 here, I have to sleep.  ;)).  If I can't reproduce it, then we will have to wait on your USB-UART adapter

Share this post


Link to post
Share on other sites
5 minutes ago, TonyMac32 said:

 Maybe check to be sure the file has the right file listed.

ah your right,  the file i transferred over didn't keep.. .. i just reconnected the unit and the files missing again..  will try again...  and no sd in these units.

Thanks for the help! will pick up tomorrow 

Share this post


Link to post
Share on other sites
1 hour ago, TonyMac32 said:

 

I had to do it as root, then I sync'd and ejected

 

you are my hero tonight.  my cluster is running again. i can go back to sleep now..

Share this post


Link to post
Share on other sites

I could reproduce this strange problem on my Tinkerboard S, but don't have a solution yet. My five cents:

1. Started with https://dl.armbian.com/tinkerboard/archive/Armbian_5.59_Tinkerboard_Debian_stretch_next_4.14.67.7z

2. Boots up fine. apt update and upgrade + reboot

3. It display error about not finding miniarm DTB but boots up.

4. Correcting DTB and it doesn't boot at all. It stuck at "loading kernel"

5. Repeated the same with updated u-boot. It also stucks.
6. Loaded https://dl.armbian.com/tinkerboard/archive/Armbian_5.70_Tinkerboard_Debian_stretch_next_4.19.14.7z and it boots up fine.

Share this post


Link to post
Share on other sites
I could reproduce this strange problem on my Tinkerboard S, but don't have a solution yet. My five cents:

1. Started with https://dl.armbian.com/tinkerboard/archive/Armbian_5.59_Tinkerboard_Debian_stretch_next_4.14.67.7z
2. Boots up fine. apt update and upgrade + reboot
3. It display error about not finding miniarm DTB but boots up.
4. Correcting DTB and it doesn't boot at all. It stuck at "loading kernel"
5. Repeated the same with updated u-boot. It also stucks.
6. Loaded https://dl.armbian.com/tinkerboard/archive/Armbian_5.70_Tinkerboard_Debian_stretch_next_4.19.14.7z and it boots up fine.
If you don't pull the SD card it will fall back to SD, reload the boot script from there, and boot. Probably how the issue escaped in the first place.

Sent from my Pixel using Tapatalk

Share this post


Link to post
Share on other sites

@igor the solution that worked on mine was simply dropping the miniarm DTB into the directory and rebooting with the jumper off and the SD out. It sounds like that worked for the OP as well, I think it's safe to push the update.

Sent from my Pixel using Tapatalk

Share this post


Link to post
Share on other sites
10 minutes ago, TonyMac32 said:

If you don't pull the SD card it will fall back to SD, reload the boot script from there, and boot. Probably how the issue escaped in the first place.


I was doing this on eMMC. I will do a few more tests.

Share this post


Link to post
Share on other sites

I was doing this on eMMC. I will do a few more tests.
Ok. I'm just relating what I saw last night, changes that failed:

-editing armbianEnv locked up after a few seconds at loading kernel.
-symlink failed same as above


-I forgot to pull the "boot from SD" jumper and it was booting ok after massive errors because of overlays and pulling the SD script after the eMMC one failed.

Sent from my Pixel using Tapatalk

Share this post


Link to post
Share on other sites

@tonymac32 & @igor -  I just wanted to follow up here as I believe I might have been mistaken regarding the SD card updates.

 

Although, the emmc models originally bricked right after first reboot after that 5.70 update,   the SD units did not... I thought the updates had completed successfully after getting a chance to check up further today, it appears i was wrong, and the SD models did not fully update as I had thought:

 

This is how the 5 SD units were reporting at time of update, 

Welcome to ARMBIAN 5.60 stable Debian GNU/Linux 9 (stretch) 4.4.156-rockchip

 

after now after update:
Welcome to ARMBIAN 5.70 stable Debian GNU/Linux 9 (stretch) 4.4.166-rockchip
 

where as, for the 3 emmc models, after the initial update, reboot, and fix provided by tony above, they now report:

 

Welcome to ARMBIAN 5.70 stable Debian GNU/Linux 9 (stretch) 4.19.14-rockchip

 

Is the 4.19.14 vs 4.4.166 difference expected for SD / emmc boot ?   or, could this be additional update issue on the SD models that i overlooked when dealing with the dead emmc's...?

 

 

Share this post


Link to post
Share on other sites

Your classic Tinkers were running "default" vendor kernel, your S's were running "Next". Next was updated to 4.19, default was not.
@igor I will try to repeat the fix tonight, I'm still thinking you did something differently than myself or the OP. The device tree for Tinker in 4.19 is set up for overlays and has a lot of things disabled, including the debug UART (maybe was bad idea, but it is enabled in the images by default via overlay). To fix old installs, I reintroduced the 4.14 device tree as "miniarm", which those installs were looking for anyway. That's what worked for me and the OP. It's not beautiful, but neither is u-boot in general.

Sent from my Pixel using Tapatalk

Share this post


Link to post
Share on other sites

@Igor just verified.  Flashed 4.14 from our download archive, then did an apt update/upgrade.  Bricked eMMC.  So then:

 

  1. Plugged Tinker into USB port on Linux Machine, it booted in UMS.
  2. As root on Linux machine, copied rk3288-miniarm.dtb from link above into /boot/dtb folder
  3. sync'd umounted/etc.
  4. Booted tinker with proper supply, booted normally.

I do notice the linux boot messages aren't showing up, it just says the starting kernel line and then after a bit shows the login prompt.  I can check that out, probably bootargs.

 

We can push the updated dtb package, which will include the miniarm dtb.

Share this post


Link to post
Share on other sites

Even if you specify verbosity=7, there are no boot messages.

 

But if you ALSO specify console=tty0 instead of console=ttyS3... the boot messages will appear,

 

Share this post


Link to post
Share on other sites
38 minutes ago, FanDjango said:

Even if you specify verbosity=7, there are no boot messages.

In /boot/armbianEnv.txt, not only "verbosity=7" is needed to get log on debug serial port, but change "console=both" to "console=serial"

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
2 2