OneARMedBandit Posted June 12, 2016 Share Posted June 12, 2016 For some days now I have the board mentioned in the subject, the goal is to have a webserver, git server, and maybe lateron something like a mini-personal-cloud for my family. Yesterday me and my brother finally decided which OS to use - and dd-ed the SD card with the Armbian/Debian Vanilla Jessie Server image (Armbian_5.10_Lime2_Debian_jessie_4.5.2). Some hours later after installing apache2 and trying things out I thought it would be a good idea to do an apt-get upgrade for security reasons before exposing the server to "the world". Unfortunately it was not: The device is not booting any longer after the upgrade. This is the log output of /var/log/apt/term.log (I mounted the SD card at the Linux system on my notebook): Current default time zone: 'Europe/Ljubljana' Local time is now: Sat Jun 11 23:28:36 CEST 2016. Universal Time is now: Sat Jun 11 21:28:36 UTC 2016. Run 'dpkg-reconfigure tzdata' if you wish to change it. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 53096 files and directories currently installed.) Preparing to unpack .../libxapian22_1.2.19-1+deb8u1_armhf.deb ... Unpacking libxapian22 (1.2.19-1+deb8u1) over (1.2.19-1) ... Preparing to unpack .../armbian-hostapd_5.13_armhf.deb ... Unpacking armbian-hostapd (5.13) over (5.10) ... Preparing to unpack .../dpkg-dev_1.17.27_all.deb ... Unpacking dpkg-dev (1.17.27) over (1.17.26) ... Preparing to unpack .../libdpkg-perl_1.17.27_all.deb ... Unpacking libdpkg-perl (1.17.27) over (1.17.26) ... Preparing to unpack .../initramfs-tools_0.120+deb8u2_all.deb ... Unpacking initramfs-tools (0.120+deb8u2) over (0.120+deb8u1) ... Preparing to unpack .../linux-dtb-next-sunxi_5.11_armhf.deb ... Unpacking linux-dtb-next-sunxi (5.11) over (5.10) ... Preparing to unpack .../linux-firmware-image-next-sunxi_5.11_armhf.deb ... Unpacking linux-firmware-image-next-sunxi (5.11) over (5.10) ... Preparing to unpack .../linux-headers-next-sunxi_5.11_armhf.deb ... Unpacking linux-headers-next-sunxi (5.11) over (5.10) ... dpkg: warning: unable to delete old directory '/usr/src/linux-headers-4.5.2-sunxi/arch/arm/include/generated': Directory not empty dpkg: warning: unable to delete old directory '/usr/src/linux-headers-4.5.2-sunxi/arch/arm/include': Directory not empty dpkg: warning: unable to delete old directory '/usr/src/linux-headers-4.5.2-sunxi/arch/arm': Directory not empty dpkg: warning: unable to delete old directory '/usr/src/linux-headers-4.5.2-sunxi/arch': Directory not empty dpkg: warning: unable to delete old directory '/usr/src/linux-headers-4.5.2-sunxi/scripts/basic': Directory not empty dpkg: warning: unable to delete old directory '/usr/src/linux-headers-4.5.2-sunxi/scripts/mod': Directory not empty dpkg: warning: unable to delete old directory '/usr/src/linux-headers-4.5.2-sunxi/scripts/kconfig': Directory not empty dpkg: warning: unable to delete old directory '/usr/src/linux-headers-4.5.2-sunxi/scripts/dtc': Directory not empty dpkg: warning: unable to delete old directory '/usr/src/linux-headers-4.5.2-sunxi/scripts': Directory not empty dpkg: warning: unable to delete old directory '/usr/src/linux-headers-4.5.2-sunxi': Directory not empty Preparing to unpack .../linux-image-next-sunxi_5.11_armhf.deb ... Unpacking linux-image-next-sunxi (5.11) over (5.10) ... Preparing to unpack .../linux-jessie-root-next-lime2_5.11_armhf.deb ... Unpacking linux-jessie-root-next-lime2 (5.11) over (5.10) ... Preparing to unpack .../linux-u-boot-lime2-next_5.11_armhf.deb ... Unpacking linux-u-boot-lime2-next (5.11) over (5.10) ... Preparing to unpack .../openssl_1.0.1t-1+deb8u2_armhf.deb ... Unpacking openssl (1.0.1t-1+deb8u2) over (1.0.1k-3+deb8u4) ... Processing triggers for systemd (215-17+deb8u4) ... Processing triggers for man-db (2.7.0.2-5) ... Setting up perl-modules (5.20.2-3+deb8u5) ... Setting up perl (5.20.2-3+deb8u5) ... Setting up libssl1.0.0:armhf (1.0.1t-1+deb8u2) ... Setting up libssl-dev:armhf (1.0.1t-1+deb8u2) ... Setting up libidn11:armhf (1.29-1+deb8u1) ... Setting up libtasn1-6:armhf (4.2-3+deb8u2) ... Setting up libxml2:armhf (2.9.1+dfsg1-5+deb8u2) ... Setting up libarchive13:armhf (3.1.2-11+deb8u1) ... Setting up libexpat1:armhf (2.1.0-6+deb8u3) ... Setting up mysql-common (5.5.49-0+deb8u1) ... Setting up libmysqlclient18:armhf (5.5.49-0+deb8u1) ... Setting up libsvn1:armhf (1.8.10-6+deb8u4) ... Setting up subversion (1.8.10-6+deb8u4) ... Setting up libxapian22 (1.2.19-1+deb8u1) ... Setting up armbian-hostapd (5.13) ... Installing new version of config file /etc/hostapd/ifupdown.sh ... Installing new version of config file /etc/init.d/hostapd ... Setting up libdpkg-perl (1.17.27) ... Setting up dpkg-dev (1.17.27) ... Setting up initramfs-tools (0.120+deb8u2) ... update-initramfs: deferring update (trigger activated) Setting up linux-dtb-next-sunxi (5.11) ... Setting up linux-firmware-image-next-sunxi (5.11) ... Setting up linux-headers-next-sunxi (5.11) ... Compiling headers - please wait ... Setting up linux-image-next-sunxi (5.11) ... update-initramfs: Generating /boot/initrd.img-4.5.5-sunxi Setting up linux-jessie-root-next-lime2 (5.11) ... Setting up linux-u-boot-lime2-next (5.11) ... Setting up openssl (1.0.1t-1+deb8u2) ... Processing triggers for libc-bin (2.19-18+deb8u4) ... Processing triggers for systemd (215-17+deb8u4) ... Processing triggers for initramfs-tools (0.120+deb8u2) ... /boot/initrd.img-4.5.5-sunxi does not exist. Cannot update. Log ended: 2016-06-11 23:34:37 It seems the upgrade routine tried to create a new initrd image, but failed: When I look into the /boot/ directory on the SD card, there really is no file "initrd.img-4.5.5-sunxi" (and from my (admittedly not very deep) linux knowledge there *should* be such a file ^^) and also not the old version 4.5.2. Now 2 questions: 1. If my diagnosis is correct - does anyone know how to (re)create the file, when the board itself is not longer booting? Can I compile it on the Linux Mint system currently running on my NB? If yes, how? And is that the only thing to fix, or do I have to do more work to... well, get the system back to work? 2. I thought Debian Jessie has reached state 'stable'. So was this incident bad luck? Or is this supposed to happen more often? Since I guess this will not be the last upgrade I run on the server, I *really* don't want to rush to my families house (where the server will be located) every few weeks, only because an update script has blown up the boot files for the server and have to manually recreate them. Or is there maybe another way to keep the system up-to-date *except* the sensitive kernel/initrd stuff, something like "apt-get safe-upgrade-that-will-not-be-likely-to-touch-the-kernel-and-ruin-my-systems-bootability"? I would be glad if someone could help me with this issue. Link to comment Share on other sites More sharing options...
zador.blood.stained Posted June 12, 2016 Share Posted June 12, 2016 1. If my diagnosis is correct - does anyone know how to (re)create the file, when the board itself is not longer booting? Can I compile it on the Linux Mint system currently running on my NB? If yes, how? And is that the only thing to fix, or do I have to do more work to... well, get the system back to work? First possibility - you can boot manually without loading uInitrd (using UART-USB adapter or monitor and USB keyboard to abort default boot sequence) and then recreate ramdisk from a running system. Second possibility - you can mount your SD card on another system and edit boot script to skip uInitrd loading. 2. I thought Debian Jessie has reached state 'stable'. So was this incident bad luck? Or is this supposed to happen more often? Since I guess this will not be the last upgrade I run on the server, I *really* don't want to rush to my families house (where the server will be located) every few weeks, only because an update script has blown up the boot files for the server and have to manually recreate them. Or is there maybe another way to keep the system up-to-date *except* the sensitive kernel/initrd stuff, something like "apt-get safe-upgrade-that-will-not-be-likely-to-touch-the-kernel-and-ruin-my-systems-bootability"? Our kernel packaging scripts needs some adjustments, initrd handling code is not perfect. Most problems are related to backwards compatibility with installing system on NAND, where /boot partition is formatted to FAT filesystem. "apt-get safe-upgrade-that-will-not-be-likely-to-touch-the-kernel-and-ruin-my-systems-bootability" apt-mark hold linux-image-next-sunxi linux-dtb-next-sunxi linux-headers-next-sunxi and your kernel won't be upgraded in the future Link to comment Share on other sites More sharing options...
tkaiser Posted June 12, 2016 Share Posted June 12, 2016 Just in case: http://kaiser-edv.de/tmp/4U4tkD/Armbian_5.14_Lime2_Ubuntu_xenial_4.6.2.7z But this (the stuff in /boot/) is already based on 4.6.2 and confirmed to work (and shows one of the nasty Lime2 Ethernet bugs we have with either u-boot or mainline kernel) Link to comment Share on other sites More sharing options...
OneARMedBandit Posted June 12, 2016 Author Share Posted June 12, 2016 Wow, thank you both very much for the fast answers! First possibility - you can boot manually without loading uInitrd (using UART-USB adapter or monitor and USB keyboard to abort default boot sequence) and then recreate ramdisk from a running system.Second possibility - you can mount your SD card on another system and edit boot script to skip uInitrd loading. I have chosen the second possibility, because of... reasons ^^. I've modified the boot script now (replacing the line "bootz 0x4... 0x<initramfs-address> 0x4..." with "bootz 0x4... - 0x4..."), rebuilt it and the device is booting again. Now the only thing left I have to find out is how to recreate the initrd-image manually... apt-mark hold linux-image-next-sunxi linux-dtb-next-sunxi linux-headers-next-sunxi and your kernel won't be upgraded in the future Great, that sounds like a good solution! Just in case: http://kaiser-edv.de/tmp/4U4tkD/Armbian_5.14_Lime2_Ubuntu_xenial_4.6.2.7z But this (the stuff in /boot/) is already based on 4.6.2 and confirmed to work (and shows one of the nasty Lime2 Ethernet bugs we have with either u-boot or mainline kernel) Thanks. If I mess things completely up maybe I will consider using that image - but I think repairing the already working system is currently the better option (to not make our work from yesterday obsolete). Link to comment Share on other sites More sharing options...
zador.blood.stained Posted June 12, 2016 Share Posted June 12, 2016 Now the only thing left I have to find out is how to recreate the initrd-image manually... Something like # if initrd.img for current version does not exist update-initramfs -c -k $(uname -r) # or if it does exist - update update-initramfs -u -k $(uname -r) # and then pack it to uInitrd mkimage -A arm -O linux -T ramdisk -C gzip -n uInitrd -d /boot/initrd.img-$(uname -r) /boot/uInitrd Link to comment Share on other sites More sharing options...
OneARMedBandit Posted June 12, 2016 Author Share Posted June 12, 2016 Yup, already found out how to do the first part , only the mkimage command I was still searching for. Interestingly the system also booted without the mkimage step . This way I also found out what the problem with the apt-get upgrade kernel update script was: It seems it has not run "update-initramfs" with the -c switch, only with the -u switch. Thank you very much! Now the only remaining question is why the SSH fingerprint of the server got regenerated on every boot (so that ssh complained on reconnecting about a probable Man-in-the-middle-attack) before the system upgrade, and now that problem is gone. But that is 1) a completely different problem and 2) - as I said - it is gone, so I don't bother ^^. Link to comment Share on other sites More sharing options...
zador.blood.stained Posted June 12, 2016 Share Posted June 12, 2016 This way I also found out what the problem with the apt-get upgrade kernel update script was: It seems it has not run "update-initramfs" with the -c switch, only with the -u switch. Initramfs part is done automatically by scripts in "/etc/kernel/". But, compared to official Debian/Ubuntu, Armbian kernel is upgraded (same package name always), while Debian/Ubuntu use new package for each kernel version (different package names). Link to comment Share on other sites More sharing options...
Recommended Posts