AdamD Posted January 15, 2019 Posted January 15, 2019 Hi all, first post here so please forgive me if I've get any bits wrong. Thanks for all the work put into supporting this project, and thanks in advance for any and all considerations extended in helping me step through my issue here. so i've had an 8 node kubernetes cluster running fairly smoothly on my tb's for nearly a full year now, so far I have not had any major issues with armbian updates until this weekends... I did a round of updates to my cluster and, after rebooting all 3 of my model S tinkerboards seem bricked (redlight only, no output to HDMI).. The remaining five orig TB models with Samsung EVO SD's all came up without issue... since my k8s cluster master is one of the S models i'm really hoping there is someway somebody knows to quickly back out and salvage these machines without a full cluster rebuild... I was thinking there might be an option because I can connect the boards via usb to pc and I am able to view the drive contents... just not sure where to look to fix a u-boot issue or if it's even possible this way. I've just ordered a UART->usb adapter as it looks like I'm going to need one now, but it won't be here until mid-week. Im not sure what other information you might need or that i can get from the boards for you, but I've also attached the last console log of the machines at time of final reboot, HTH.. I'm guessing my questions at this point are: 1) Any known issues or anyone else having issues with S models after the last update to 5.70 ? 2) Any other ideas on how I can debug or recover these machines before I receive a UART adapter? 3) Any other references to point me at to further debug this issue properly? best regards, -ad Spoiler Welcome to ARMBIAN 5.60 stable Debian GNU/Linux 9 (stretch) 4.14.70-rockchip System load: 0.53 0.68 0.70 Up time: 36 days Memory usage: 59 % of 2006MB IP: xxx.xxx.xxx.xxx CPU temp: 35°C Usage of /: 37% of 15G xxxx@yyyy:~$ sudo apt update [sudo] password for xxxx: Get:1 http://security.debian.org stretch/updates InRelease [94.3 kB] Ign:3 http://cdn-fastly.deb.debian.org/debian stretch InRelease Get:5 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease [91.0 kB] Get:6 https://download.docker.com/linux/debian stretch InRelease [44.8 kB] Get:7 http://cdn-fastly.deb.debian.org/debian stretch-backports InRelease [91.8 kB] Hit:8 http://cdn-fastly.deb.debian.org/debian stretch Release Hit:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease Get:4 https://apt.armbian.com stretch InRelease [19.0 kB] Get:9 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages.diff/Index [27.8 kB] Get:10 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Contents (deb).diff/Index [28.0 kB] Get:11 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Packages.diff/Index [12.5 kB] Get:12 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Contents (deb).diff/Index [9,592 B] Get:13 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages 2019-01-13-0212.27.pdiff [293 B] Get:15 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages 2019-01-13-0810.11.pdiff [215 B] Get:15 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Packages 2019-01-13-0810.11.pdiff [215 B] Get:16 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Contents (deb) 2019-01-13-0212.27.pdiff [304 B] Get:16 http://cdn-fastly.deb.debian.org/debian stretch-backports/main armhf Contents (deb) 2019-01-13-0212.27.pdiff [304 B] Get:17 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Packages 2019-01-12-2012.01.pdiff [227 B] Get:17 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Packages 2019-01-12-2012.01.pdiff [227 B] Get:18 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Contents (deb) 2019-01-12-2012.01.pdiff [271 B] Get:18 http://cdn-fastly.deb.debian.org/debian stretch-backports/non-free armhf Contents (deb) 2019-01-12-2012.01.pdiff [271 B] Get:19 https://apt.armbian.com stretch/main armhf Packages [322 kB] Get:20 https://apt.armbian.com stretch/stretch-utils armhf Packages [4,893 B] Fetched 747 kB in 7s (104 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done 5 packages can be upgraded. Run 'apt list --upgradable' to see them. xxxx@yyyy:~$ sudo apt upgrade -y Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following packages will be upgraded: linux-dtb-next-rockchip linux-image-next-rockchip linux-libc-dev linux-stretch-root-next-tinkerboard linux-u-boot-tinkerboard-next 5 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 20.8 MB of archives. After this operation, 26.9 MB of additional disk space will be used. Get:1 https://apt.armbian.com stretch/main armhf linux-dtb-next-rockchip armhf 5.70 [84.5 kB] Get:2 https://apt.armbian.com stretch/main armhf linux-image-next-rockchip armhf 5.70 [18.8 MB] Get:3 https://apt.armbian.com stretch/main armhf linux-libc-dev armhf 5.70 [1,007 kB] Get:4 https://apt.armbian.com stretch/main armhf linux-stretch-root-next-tinkerboard armhf 5.70 [425 kB] Get:5 https://apt.armbian.com stretch/main armhf linux-u-boot-tinkerboard-next armhf 5.70 [489 kB] Fetched 20.8 MB in 4s (4,641 kB/s) (Reading database ... 31727 files and directories currently installed.) Preparing to unpack .../linux-dtb-next-rockchip_5.70_armhf.deb ... Unpacking linux-dtb-next-rockchip (5.70) over (5.60) ... Preparing to unpack .../linux-image-next-rockchip_5.70_armhf.deb ... update-initramfs: Deleting /boot/initrd.img-4.14.70-rockchip Removing obsolete file uInitrd-4.14.70-rockchip Unpacking linux-image-next-rockchip (5.70) over (5.60) ... Preparing to unpack .../linux-libc-dev_5.70_armhf.deb ... Unpacking linux-libc-dev (5.70) over (4.9.130-2) ... Preparing to unpack .../linux-stretch-root-next-tinkerboard_5.70_armhf.deb ... Leaving 'diversion of /etc/mpv/mpv.conf to /etc/mpv/mpv-dist.conf by linux-stretch-root-next-tinkerboard' Unpacking linux-stretch-root-next-tinkerboard (5.70) over (5.60) ... Preparing to unpack .../linux-u-boot-tinkerboard-next_5.70_armhf.deb ... Unpacking linux-u-boot-tinkerboard-next (5.70) over (5.60) ... Setting up linux-image-next-rockchip (5.70) ... update-initramfs: Generating /boot/initrd.img-4.19.14-rockchip update-initramfs: Converting to u-boot format Setting up linux-u-boot-tinkerboard-next (5.70) ... Setting up linux-libc-dev (5.70) ... Processing triggers for initramfs-tools (0.130) ... update-initramfs: Generating /boot/initrd.img-4.19.14-rockchip update-initramfs: Converting to u-boot format Setting up linux-stretch-root-next-tinkerboard (5.70) ... Setting up linux-dtb-next-rockchip (5.70) ... Processing triggers for initramfs-tools (0.130) ... update-initramfs: Generating /boot/initrd.img-4.19.14-rockchip update-initramfs: Converting to u-boot format xxxx@yyyy:~$ sudo reboot ........
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 5 minutes ago, AdamD said: redlight only, Sounds like it's not getting past u-boot. You say the non- eMMC Tinkerboards were fine after upgrade?
AdamD Posted January 15, 2019 Author Posted January 15, 2019 2 minutes ago, TonyMac32 said: Sounds like it's not getting past u-boot. You say the non- eMMC Tinkerboards were fine after upgrade? yes. all 5 of the non-eMMC are functioning fine so far.
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 OK, I'll have a look. I haven't tried an in-place update on the eMMC, but the board in front of me is set up with 4.14 on eMMC. Give me a couple minutes.
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 1 hour ago, AdamD said: yes. all 5 of the non-eMMC are functioning fine so far. OK, mine is wanting to see "rk3288-miniarm.dtb" and can't find it. We had a symlink there, it must have gotten lost, but why that isn't angering the SD cards I'm not sure... if you edit /boot/armbianEnv.txt to read "fdt_file=rk3288-tinker.dtb" instead of "fdt_file=rk3288-miniarm.dtb" you are a step closer, but mine is hanging after a few seconds, can you verify that behavior? I am trying something else quickly. @Igor does u-boot not get updated on eMMC? My board is still listing the 2018.05 revision after the update
AdamD Posted January 15, 2019 Author Posted January 15, 2019 14 minutes ago, TonyMac32 said: if you edit /boot/armbianEnv.txt to read "fdt_file=rk3288-tinker.dtb" instead of "fdt_file=rk3288-miniarm.dtb" you are a step closer, but mine is hanging after a few seconds, can you verify that behavior? I am trying something else quickly. Thank you!
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 11 minutes ago, AdamD said: Thank you! OK, don't edit anything, just copy this file into the "dtb" folder with the other device trees. @Igor I'll push this patch, basically creating this device tree for the older u-boot systems. rk3288-miniarm.dtb
AdamD Posted January 15, 2019 Author Posted January 15, 2019 8 minutes ago, TonyMac32 said: OK, don't edit anything, just copy this file into the "dtb" folder with the other device trees. rk3288-miniarm.dtb so just copying rk3288-miniarm.dtb into /boot/dtb-4.19.14-rockchip folder and powering on is same red light.. another step?
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 OK, show me what your boot.cmd file looks like.
Igor Posted January 15, 2019 Posted January 15, 2019 15 minutes ago, TonyMac32 said: I'll push this patch, basically creating this device tree for the older u-boot systems. rk3288-miniarm.dtb So I need to push out an updated kernel / dtb pack for rockchip-next? ... U-boot is now updated manually only. Which solves other problems .. but I guess we have some new ones
AdamD Posted January 15, 2019 Author Posted January 15, 2019 # DO NOT EDIT THIS FILE # # Please edit /boot/armbianEnv.txt to set supported parameters # setenv rootdev "/dev/mmcblk0p1" setenv rootfstype "ext4" setenv fdt_file "rk3288-miqi.dtb" setenv ramdisk_addr_r "0x21000000" setenv console "ttyS2,115200n8 console=tty1" setenv verbosity "1" itest.b ${devnum} == 0 && echo "U-boot loaded from SD" itest.b ${devnum} == 1 && echo "U-boot loaded from eMMC" if load ${devtype} ${devnum}:1 ${ramdisk_addr_r} /boot/armbianEnv.txt || load ${devtype} ${devnum}:1 ${ramdisk_addr_r} armbianEnv.txt; then env import -t ${ramdisk_addr_r} ${filesize} fi setenv bootargs "consoleblank=0 scandelay root=${rootdev} rw console=${console} rootfstype=${rootfstype} loglevel=${verbosity} rootwait usb-storage.quirks=${usbstoragequirks} ${extraargs}" ext4load ${devtype} ${devnum}:1 ${fdt_addr_r} /boot/dtb/${fdt_file} || fatload ${devtype} ${devnum}:1 ${fdt_addr_r} dtb/${fdt_file} || ext4load ${devtype} ${devnum}:1 ${fdt_addr_r} dtb/${fdt_file} ext4load ${devtype} ${devnum}:1 ${ramdisk_addr_r} /boot/uInitrd || fatload ${devtype} ${devnum}:1 ${ramdisk_addr_r} uInitrd || ext4load ${devtype} ${devnum}:1 ${ramdisk_addr_r} uInitrd ext4load ${devtype} ${devnum}:1 ${kernel_addr_r} /boot/zImage || fatload ${devtype} ${devnum}:1 ${kernel_addr_r} zImage || ext4load ${devtype} ${devnum}:1 ${kernel_addr_r} zImage bootz ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} # mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 Just now, Igor said: U-boot is now updated manually only I'll have to catch up on that. This worked on mine, but has not worked on @AdamD's I want to verify the boot comands/etc
AdamD Posted January 15, 2019 Author Posted January 15, 2019 Just now, TonyMac32 said: I'll have to catch up on that. This worked on mine, but has not worked on @AdamD's I want to verify the boot comands/etc Is this my prob? setenv fdt_file "rk3288-miqi.dtb"
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 3 minutes ago, AdamD said: # DO NOT EDIT THIS FILE That looks the same, now, did you reboot it with the USB attached to the PC still (crazy question, but if I don't ask...) (This would get it stuck in UMS mode)
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 Just now, AdamD said: setenv fdt_file "rk3288-miqi.dtb" No, the armbianEnv.txt file overrides it. Maybe check to be sure the file has the right file listed.
AdamD Posted January 15, 2019 Author Posted January 15, 2019 Just now, TonyMac32 said: That looks the same, now, did you reboot it with the USB attached to the PC still (crazy question, but if I don't ask...) no, unplugged, replugged.. tried on 2 of the 3 units before reply as well..
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 no SD card in the slot, right? Otherwise I'll try to install a fresh 4.14 and repeat the process again and see what happens tomorrow my time (is 1:30 here, I have to sleep. ). If I can't reproduce it, then we will have to wait on your USB-UART adapter
AdamD Posted January 15, 2019 Author Posted January 15, 2019 5 minutes ago, TonyMac32 said: Maybe check to be sure the file has the right file listed. ah your right, the file i transferred over didn't keep.. .. i just reconnected the unit and the files missing again.. will try again... and no sd in these units. Thanks for the help! will pick up tomorrow
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 Just now, AdamD said: will try again. I had to do it as root, then I sync'd and ejected
AdamD Posted January 15, 2019 Author Posted January 15, 2019 1 hour ago, TonyMac32 said: I had to do it as root, then I sync'd and ejected you are my hero tonight. my cluster is running again. i can go back to sleep now.. 1
Igor Posted January 15, 2019 Posted January 15, 2019 I could reproduce this strange problem on my Tinkerboard S, but don't have a solution yet. My five cents: 1. Started with https://dl.armbian.com/tinkerboard/archive/Armbian_5.59_Tinkerboard_Debian_stretch_next_4.14.67.7z 2. Boots up fine. apt update and upgrade + reboot 3. It display error about not finding miniarm DTB but boots up. 4. Correcting DTB and it doesn't boot at all. It stuck at "loading kernel" 5. Repeated the same with updated u-boot. It also stucks. 6. Loaded https://dl.armbian.com/tinkerboard/archive/Armbian_5.70_Tinkerboard_Debian_stretch_next_4.19.14.7z and it boots up fine.
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 I could reproduce this strange problem on my Tinkerboard S, but don't have a solution yet. My five cents: 1. Started with https://dl.armbian.com/tinkerboard/archive/Armbian_5.59_Tinkerboard_Debian_stretch_next_4.14.67.7z 2. Boots up fine. apt update and upgrade + reboot 3. It display error about not finding miniarm DTB but boots up. 4. Correcting DTB and it doesn't boot at all. It stuck at "loading kernel" 5. Repeated the same with updated u-boot. It also stucks. 6. Loaded https://dl.armbian.com/tinkerboard/archive/Armbian_5.70_Tinkerboard_Debian_stretch_next_4.19.14.7z and it boots up fine.If you don't pull the SD card it will fall back to SD, reload the boot script from there, and boot. Probably how the issue escaped in the first place.Sent from my Pixel using Tapatalk
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 @igor the solution that worked on mine was simply dropping the miniarm DTB into the directory and rebooting with the jumper off and the SD out. It sounds like that worked for the OP as well, I think it's safe to push the update.Sent from my Pixel using Tapatalk
Igor Posted January 15, 2019 Posted January 15, 2019 10 minutes ago, TonyMac32 said: If you don't pull the SD card it will fall back to SD, reload the boot script from there, and boot. Probably how the issue escaped in the first place. I was doing this on eMMC. I will do a few more tests.
TonyMac32 Posted January 15, 2019 Posted January 15, 2019 I was doing this on eMMC. I will do a few more tests.Ok. I'm just relating what I saw last night, changes that failed:-editing armbianEnv locked up after a few seconds at loading kernel.-symlink failed same as above-I forgot to pull the "boot from SD" jumper and it was booting ok after massive errors because of overlays and pulling the SD script after the eMMC one failed.Sent from my Pixel using Tapatalk
AdamD Posted January 16, 2019 Author Posted January 16, 2019 @tonymac32 & @igor - I just wanted to follow up here as I believe I might have been mistaken regarding the SD card updates. Although, the emmc models originally bricked right after first reboot after that 5.70 update, the SD units did not... I thought the updates had completed successfully after getting a chance to check up further today, it appears i was wrong, and the SD models did not fully update as I had thought: This is how the 5 SD units were reporting at time of update, Welcome to ARMBIAN 5.60 stable Debian GNU/Linux 9 (stretch) 4.4.156-rockchip after now after update: Welcome to ARMBIAN 5.70 stable Debian GNU/Linux 9 (stretch) 4.4.166-rockchip where as, for the 3 emmc models, after the initial update, reboot, and fix provided by tony above, they now report: Welcome to ARMBIAN 5.70 stable Debian GNU/Linux 9 (stretch) 4.19.14-rockchip Is the 4.19.14 vs 4.4.166 difference expected for SD / emmc boot ? or, could this be additional update issue on the SD models that i overlooked when dealing with the dead emmc's...?
TonyMac32 Posted January 16, 2019 Posted January 16, 2019 Your classic Tinkers were running "default" vendor kernel, your S's were running "Next". Next was updated to 4.19, default was not. @igor I will try to repeat the fix tonight, I'm still thinking you did something differently than myself or the OP. The device tree for Tinker in 4.19 is set up for overlays and has a lot of things disabled, including the debug UART (maybe was bad idea, but it is enabled in the images by default via overlay). To fix old installs, I reintroduced the 4.14 device tree as "miniarm", which those installs were looking for anyway. That's what worked for me and the OP. It's not beautiful, but neither is u-boot in general. Sent from my Pixel using Tapatalk
TonyMac32 Posted January 17, 2019 Posted January 17, 2019 @Igor just verified. Flashed 4.14 from our download archive, then did an apt update/upgrade. Bricked eMMC. So then: Plugged Tinker into USB port on Linux Machine, it booted in UMS. As root on Linux machine, copied rk3288-miniarm.dtb from link above into /boot/dtb folder sync'd umounted/etc. Booted tinker with proper supply, booted normally. I do notice the linux boot messages aren't showing up, it just says the starting kernel line and then after a bit shows the login prompt. I can check that out, probably bootargs. We can push the updated dtb package, which will include the miniarm dtb.
FanDjango Posted January 17, 2019 Posted January 17, 2019 Even if you specify verbosity=7, there are no boot messages. But if you ALSO specify console=tty0 instead of console=ttyS3... the boot messages will appear,
martinayotte Posted January 17, 2019 Posted January 17, 2019 38 minutes ago, FanDjango said: Even if you specify verbosity=7, there are no boot messages. In /boot/armbianEnv.txt, not only "verbosity=7" is needed to get log on debug serial port, but change "console=both" to "console=serial"
Recommended Posts