freak Posted September 25, 2018 Share Posted September 25, 2018 I have 85 tinkers running with armbian 5.41 xenial next 4.14.23. Normally I run a script that apt upgrades them all but I run a small batch of them first as a test. Today I ran apt update/upgrade on three of them and they are all bricked. No output to the screen. Can't ping them. Any ideas? 1 Link to comment Share on other sites More sharing options...
Igor Posted September 25, 2018 Share Posted September 25, 2018 5 minutes ago, freak said: I have 85 tinkers running with armbian 5.41 xenial next 4.14.23. Normally I run a script that apt upgrades them all but I run a small batch of them first as a test. Today I ran apt update/upgrade on three of them and they are all bricked. No output to the screen. Can't ping them. Any ideas? We need to see serial console logs. 1 Link to comment Share on other sites More sharing options...
freak Posted September 25, 2018 Author Share Posted September 25, 2018 Ok... Not familiar with how to get a serial console. Link to a doc perhaps? Link to comment Share on other sites More sharing options...
Igor Posted September 25, 2018 Share Posted September 25, 2018 44 minutes ago, freak said: Ok... Not familiar with how to get a serial console. Link to a doc perhaps? Paremeters:https://www.armbian.com/tinkerboard/ Pins: https://tinkerboarding.co.uk/wiki/index.php/Hardware Pinout: Hardware:http://amzn.to/2wbNkLp Link to comment Share on other sites More sharing options...
freak Posted September 25, 2018 Author Share Posted September 25, 2018 TTL is ok on those pins? Link to comment Share on other sites More sharing options...
Igor Posted September 25, 2018 Share Posted September 25, 2018 9 minutes ago, freak said: TTL is ok on those pins? Yes, but you don't need to do anything. I already found the cause of the problem. Just do a power cycle. Link to comment Share on other sites More sharing options...
freak Posted September 25, 2018 Author Share Posted September 25, 2018 Thanks I've tried power cycling several times but they still don't boot. I can try again. Link to comment Share on other sites More sharing options...
freak Posted September 25, 2018 Author Share Posted September 25, 2018 Power cycling isn't solving it. Link to comment Share on other sites More sharing options...
Igor Posted September 25, 2018 Share Posted September 25, 2018 I did try to replicate and start with some ancient build (older than yours, 5.33) than making an upgrade. It stalled at reboot ... cutting power and powering back resolved the problem and board is alive. Those are my logs: http://ix.io/1nwv Serial console log: Spoiler [ 812.477918] watchdog: watchdog0: watchdog did not stop! [ 813.213620] reboot: Restarting system U-Boot SPL 2018.07-armbian (Sep 19 2018 - 13:23:29 +0200) Returning to boot ROM... U-Boot 2018.07-armbian (Sep 19 2018 - 13:23:29 +0200) Model: Tinker-RK3288 DRAM: 2 GiB MMC: dwmmc@ff0c0000: 1, dwmmc@ff0f0000: 0 Loading Environment from EXT4... Card did not respond to voltage select! ** Bad device mmc 0 ** Failed (-5) In: serial Out: vidconsole Err: vidconsole Model: Tinker-RK3288 Net: eth0: ethernet@ff290000 Hit any key to stop autoboot: 0 switch to partitions #0, OK mmc1 is current device Scanning mmc 1:1... Found U-Boot script /boot/boot.scr 1446 bytes read in 1 ms (1.4 MiB/s) ## Executing script at 00000000 U-boot loaded from eMMC 155 bytes read in 2 ms (75.2 KiB/s) 43206 bytes read in 6 ms (6.9 MiB/s) 4603549 bytes read in 204 ms (21.5 MiB/s) 8546016 bytes read in 376 ms (21.7 MiB/s) ## Loading init Ramdisk from Legacy Image at 21000000 ... Image Name: uInitrd Image Type: ARM Linux RAMDisk Image (gzip compressed) Data Size: 4603485 Bytes = 4.4 MiB Load Address: 00000000 Entry Point: 00000000 Verifying Checksum ... OK ## Flattened Device Tree blob at 01f00000 Booting using the fdt blob at 0x1f00000 Loading Ramdisk to 0fb9c000, end 0ffffe5d ... OK Loading Device Tree to 0fb8e000, end 0fb9b8c5 ... OK Starting kernel ... Loading, please wait... starting version 229 Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... Scanning for Btrfs filesystems done. Begin: Will now check root file system ... fsck from util-linux 2.27.1 [/sbin/fsck.ext4 (1) -- /dev/mmcblk0p1] fsck.ext4 -a -C0 /dev/mmcblk0p1 /dev/mmcblk0p1: clean, 60184/1789824 files, 443515/7736504 blocks done. done. Begin: Running /scripts/local-bottom ... done. Begin: Running /scripts/init-bottom ... done. Welcome to Ubuntu 16.04.5 LTS! [ OK ] Listening on fsck to fsckd communication Socket. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Encrypted Volumes. [ OK ] Listening on Syslog Socket. [ OK ] Created slice System Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Remote File Systems. [ OK ] Listening on Journal Socket. Starting Restore / save the current clock... Starting Set console keymap... Mounting POSIX Message Queue File System... Starting Nameserver information manager... Starting Remount Root and Kernel File Systems... Starting Load Kernel Modules... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st... nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Mounting Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Started Trigger resolvconf update for networkd DNS. [ OK ] Mounted Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Restore / save the current clock. [ OK ] Started Set console keymap. [ OK ] Started Remount Root and Kernel File Systems. [ OK ] Started Load Kernel Modules. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Nameserver information manager. [ OK ] Reached target Network (Pre). Starting Create Static Device Nodes in /dev... Mounting Configuration File System... Starting Apply Kernel Variables... Starting Load/Save Random Seed... Starting udev Coldplug all Devices... Activating swap /var/swap... [ OK ] Mounted Configuration File System. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Apply Kernel Variables. [ OK ] Started Load/Save Random Seed. [ OK ] Activated swap /var/swap. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems (Pre). Mounting /tmp... Starting udev Kernel Device Manager... [ OK ] Mounted /tmp. [ OK ] Reached target Local File Systems. Starting Armbian ZRAM config... Starting Set console font and keymap... Starting Raise network interfaces... [ OK ] Started Entropy daemon using the HAVEGE algorithm. Starting Journal Service... [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ OK ] Found device /dev/ttyS2. [ OK ] Reached target Sound Card. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. [ OK ] Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. Starting Update UTMP about System Boot/Shutdown... [ OK ] Reached target System Time Synchronized. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started ifup for eth0. Starting Load/Save RF Kill Switch Status... [ OK ] Found device /sys/subsystem/net/devices/eth0. [ OK ] Started Load/Save RF Kill Switch Status. [ OK ] Started Set console font and keymap. [ OK ] Created slice system-getty.slice. [ OK ] Started Armbian ZRAM config. Starting Armbian memory supported logging... [ OK ] Started Raise network interfaces. [ OK ] Started Armbian memory supported logging. [ OK ] Reached target System Initialization. Starting Armbian hardware optimization... [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Armbian hardware monitoring... [ OK ] Started Daily apt download activities. [ OK ] Started Daily apt upgrade and clean activities. [ OK ] Reached target Timers. [ OK ] Started Armbian hardware optimization. [ OK ] Started Armbian hardware monitoring. [ OK ] Reached target Basic System. Starting LSB: Load kernel modules needed to enable cpufreq scaling... Starting LSB: Starts LIRC daemon.... Starting System Logging Service... [ OK ] Started Regular background program processing daemon. Starting LSB: Start/stop sysstat's sadc... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Permit User Sessions... Starting Login Service... Starting Save/Restore Sound Card State... [ OK ] Started System Logging Service. [ OK ] Started Permit User Sessions. [ OK ] Started Save/Restore Sound Card State. [ OK ] Started LSB: Start/stop sysstat's sadc. [ OK ] Started LSB: Starts LIRC daemon.. [ OK ] Started LSB: Load kernel modules needed to enable cpufreq scaling. Starting LSB: set CPUFreq kernel parameters... Starting Set console scheme... [ OK ] Started Set console scheme. [ OK ] Started Login Service. [ OK ] Started LSB: set CPUFreq kernel parameters. Starting LSB: Set sysfs variables from /etc/sysfs.conf... [ OK ] Started LSB: Set sysfs variables from /etc/sysfs.conf. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. [ OK ] Started Unattended Upgrades Shutdown. Starting OpenBSD Secure Shell server... Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. Starting Hostname Service... [ OK ] Started OpenBSD Secure Shell server. [ OK ] Started Hostname Service. Starting Authenticate and Authorize Users to Run Privileged Tasks... Starting WPA supplicant... [ OK ] Started Authenticate and Authorize Users to Run Privileged Tasks. [ OK ] Started WPA supplicant. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting /etc/rc.local Compatibility... Starting LSB: Advanced IEEE 802.11 management daemon... Starting LSB: Start NTP daemon... Starting LSB: disk temperature monitoring daemon... [ OK ] Started /etc/rc.local Compatibility. [ OK ] Started LSB: Advanced IEEE 802.11 management daemon. [ OK ] Started LSB: disk temperature monitoring daemon. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS2. [ OK ] Reached target Login Prompts. [ OK ] Started LSB: Start NTP daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Ubuntu 16.04.5 LTS tinkerboard ttyS2 tinkerboard login: Link to comment Share on other sites More sharing options...
Igor Posted September 25, 2018 Share Posted September 25, 2018 Now you need to get logs. From our perspective, everything works. Have you perhaps update to Bionic? Link to comment Share on other sites More sharing options...
chwe Posted September 25, 2018 Share Posted September 25, 2018 5 minutes ago, freak said: Power cycling isn't solving it. well then set up serial console.. and provide some bootlog. I noticed that reboot after update of my tinker from 5.5x (don't remember exactly) to 5.60 took a way longer than normal. Maybe I should stick a serial console to it to get a clue why.. But if I remember correctly I didn't had to power-cycle it.. Link to comment Share on other sites More sharing options...
freak Posted September 25, 2018 Author Share Posted September 25, 2018 I don't have the usb/ttl adapter but I'll order one. In the meantime I don't mind upgrading to bionic. Can I do it from the command line from the build that I'm at? Link to comment Share on other sites More sharing options...
Igor Posted September 25, 2018 Share Posted September 25, 2018 33 minutes ago, freak said: I don't have the usb/ttl adapter but I'll order one. It's an amazing tool for the value It can save you a lot of time and money. 34 minutes ago, freak said: In the meantime I don't mind upgrading to bionic. That was more as a rhetorical question. Upgrade to Bionic might cause troubles but if you start with a clean image, you should be fine. That upgrade is out of our power to help you in case of troubles. Link to comment Share on other sites More sharing options...
freak Posted September 25, 2018 Author Share Posted September 25, 2018 Is whatever what was wrong fixed so I can run apt upgrade with bricking the rest of the units? Link to comment Share on other sites More sharing options...
Igor Posted September 25, 2018 Share Posted September 25, 2018 1 minute ago, freak said: Is whatever what was wrong fixed so I can run apt upgrade with bricking the rest of the units? No! We didn't find nor fix any problems. You have to get a serial console and show us what do you see when you power the bricked device. Perhaps the problem is in your configuration which is not our problem. I have no idea without seeing logs. Link to comment Share on other sites More sharing options...
chwe Posted September 25, 2018 Share Posted September 25, 2018 or... you SSH into a working SBC and use it's spare UART pins to ttl into the non-working SBC... Likely to be one of the most silly things I've ever did with an SBC.. But hey, it works... (at least @martinayotte will like it. ) 1 hour ago, Igor said: Pins: do we have UART2 and UART3? at least on mine (4.14) we use UART3... @TonyMac32? Or was it back then when we used UART2? Can't remember anymore.. 2 Link to comment Share on other sites More sharing options...
freak Posted September 25, 2018 Author Share Posted September 25, 2018 ...and I can re-image the affected boards to fix them. But for the working ones I'd like to upgrade to bionic if I can do that via the command line. Link to comment Share on other sites More sharing options...
Igor Posted September 25, 2018 Share Posted September 25, 2018 3 minutes ago, chwe said: do we have UART2 and UART3? at least on mine (4.14) we use UART3... This picture was taken on a working UART with kernel 4.14.y 1 minute ago, freak said: I'd like to upgrade to bionic if I can do that via the command line. Do (ask google), but in case of troubles don't cry You have been warned. Link to comment Share on other sites More sharing options...
TonyMac32 Posted September 25, 2018 Share Posted September 25, 2018 I'm at work so I can't check, but we should match the vendor, which is the UART at the bottom of the gpio header. REMEMBER 3.3V I/O Sent from my Pixel using Tapatalk Link to comment Share on other sites More sharing options...
freak Posted September 25, 2018 Author Share Posted September 25, 2018 Why "do-release-upgrade -d"? Isn't there a stable release? Link to comment Share on other sites More sharing options...
chwe Posted September 25, 2018 Share Posted September 25, 2018 1 minute ago, Igor said: 7 minutes ago, chwe said: do we have UART2 and UART3? at least on mine (4.14) we use UART3... This picture was taken on a working UART with kernel 4.14.y mine too.. See photo and that's UART3: 17 minutes ago, chwe said: even reboot works with ssh (notebook) --> ttl (OPi0) --> tinkerboard Link to comment Share on other sites More sharing options...
Igor Posted September 25, 2018 Share Posted September 25, 2018 9 minutes ago, freak said: Why "do-release-upgrade -d"? Isn't there a stable release? Google's first hit: https://linuxconfig.org/how-to-upgrade-to-ubuntu-18-04-lts-bionic-beaver Even Canonical recommends running distribution upgrade (old stable -> stable) from a serial console ... which you don't have. Anyway, we are getting off topic and out of free support for which you need to supply UART data. Now @chwe supplied alternative and working option. Link to comment Share on other sites More sharing options...
freak Posted September 25, 2018 Author Share Posted September 25, 2018 Update: This may have something to do with the resolution. These a mostly connected to 1920x1200 HP monitors often with hdmi to dvi cables (sometimes hdmi both ends) but quite often they only run at 1920x1080. But they work. This "bricked" unit boots fine when connected to a acer G215H 1920x1080 monitor via hdmi/dvi. Connected to an HP z27N via straight hdmi it will not boot. Link to comment Share on other sites More sharing options...
Igor Posted September 25, 2018 Share Posted September 25, 2018 1 minute ago, freak said: Update: This may have something to do with the resolution. These a mostly connected to 1920x1200 HP monitors often with hdmi to dvi cables (sometimes hdmi both ends) but quite often they only run at 1920x1080. But they work. This "bricked" unit boots fine when connected to a acer G215H 1920x1080 monitor via hdmi/dvi. Connected to an HP z27N via straight hdmi it will not boot. I was waiting for this. A prove that there is nothing wrong with our work. Thank you! Armbian does not provide support for 3rd party hardware, which are the root of this problem -> a few people can't provide FREE support for all kind of electronic junk on this planet in their spare time. I hope you do understand this? Without logs, problems are not even officially recognized. That's why I repeat myself like a parrot. Only logs might tell if we can solve the problem quickly or we will just file it. Link to comment Share on other sites More sharing options...
freak Posted September 25, 2018 Author Share Posted September 25, 2018 I was just giving you an update my friend. I will get your logs. 1 Link to comment Share on other sites More sharing options...
martinayotte Posted September 25, 2018 Share Posted September 25, 2018 1 hour ago, chwe said: (at least @martinayotte will like it. ) Link to comment Share on other sites More sharing options...
freak Posted September 25, 2018 Author Share Posted September 25, 2018 29 minutes ago, Igor said: I was waiting for this. A prove that there is nothing wrong with our work. Thank you! Armbian does not provide support for 3rd party hardware, which are the root of this problem -> a few people can't provide FREE support for all kind of electronic junk on this planet in their spare time. I hope you do understand this? Without logs, problems are not even officially recognized. That's why I repeat myself like a parrot. Only logs might tell if we can solve the problem quickly or we will just file it. But to be fair... all 85 of my boards were working fine till I ran apt upgrade. Link to comment Share on other sites More sharing options...
chwe Posted September 25, 2018 Share Posted September 25, 2018 5 minutes ago, freak said: But to be fair... all 85 of my boards were working fine till I ran apt upgrade. we don't willingly break hardware support just to upset someone... If it happens it's likely out of our control.. (e.g. kernel) and it's simply not possible to first: avoid all those possible scenarios of third party hardware and second provide you some help without even knowing what's going wrong. 1 hour ago, freak said: This "bricked" unit boots fine when connected to a acer G215H 1920x1080 monitor via hdmi/dvi. means bricked here upgraded power-cycled and then rebooted or with a freshly installed armbian? 3 hours ago, freak said: 5.41 xenial next 4.14.23. from the changelog: v5.41 / 10.2.2018 fixed LED driver on Helios4 bugfix update on sunxi/sunxi64 kernel. Updated to 4.14.18 kernel update for MVEBU next (4.14.18 and default 4.4.115) for Clearfog and Helios4. Upstream fixes,AUFS and Realtek 881yAU drivers update that's 8 months without doing any updates? Have fun to figure out when and which changes ended in 'doesn't work anymore'. IMO simply not possible. But git@tinkerboard:~$ find /dev | grep ttyS /dev/ttyS3 /dev/ttyS2 /dev/ttyS1 /dev/ttyS0 serial access is possible... You can even build circles.. SSH --> (opi) ttl --> (tinker) ttl --> opi with 85 tinkers, there should be one spare to debug one which is 'bricked' due to upgrade... Just connect UART1 to another boards UART3 (that's the one you want to debug) install picocom on the first one and then: picocom -b 115200 -r -l /dev/ttyS1 1 hour ago, martinayotte said: 2 hours ago, chwe said: one of the most silly things I've ever did with an SBC.. But hey, it works... (at least @martinayotte will like it. ) well.. obviously not true anymore.. Link to comment Share on other sites More sharing options...
TonyMac32 Posted September 25, 2018 Share Posted September 25, 2018 That's amazing. Now I want to Cascade login to as many as possible. #SBCeptionSent from my Pixel using Tapatalk 1 Link to comment Share on other sites More sharing options...
chwe Posted September 25, 2018 Share Posted September 25, 2018 22 minutes ago, TonyMac32 said: That's amazing. Now I want to Cascade login to as many as possible. #SBCeption so, enough off topic here... Link to comment Share on other sites More sharing options...
Recommended Posts