Odroid HC-1 died after apt-update/upgrade

[jcorbet]

First I have to thank all maintainers for their work. I ran this server for 1.5 years before something broke. Since armbian is maintained by volunteers I want to cheer you up and thank you for your hard work.

 

However just today my little server would die after apt-update/upgrade. I use a SATA drive together with a boot SD-card (only containing /boot). I can observe the following behavior:

 

The system leds indicate that booting is sucessful (blue LED does blink regularly, I can detect HDD usage by observing the green LED), yet SSH login that I have been using for years now is impossible. The device is also not responsive when pinging it. From my router's dashboard I can not detect a sucessful network connection, yet the NIC's orange LED would blink regularly. Sadly I do not own an UART adapter, maybe I should have bought one sooner.

 

What I have already done:

 

Mirror the SD card using dd. I am uploading the file and will post the link here for review. // Upload finished: https://mega.nz/file/yYJwEIiA#zw4lBfeJ8wXyxS1EeGDpMGERbQquEoxvNaCh2CFwTbg

Notice that the focal images seem to be corrupt. I have added focal sources since I needed more modern packages for installing ntopng. Maybe this had something to do with it.

 

I am very glad if someone has some advice, well any advice. I would hate to setup everything again

[rosbeef]

I'm in the same trouble.

 

It boot well on my SDcard then i do a rsync of the root to the encrypted ssd change the boot.ini to boot to the ssd as everytime i did.

~ # sudo sed -e 's/root=[^ ]*/root=\/dev\/sdg\/root cryptopts=source=\/dev\/sda,target=sdc,lvm=sdg /' -i /boot/boot.ini
~ # sudo update-initramfs -u

then it boot first on the initramfs well and i have access to the busybox ssh.

then I execute cryptroot_unlock it seems to start initializing well but without network, so i can't access.

 

I have access to the ssd from the initramfs using /sbin/cryptsetup open ...

i can mount and view file on the ssd from the initramfs.

 

Manualy tried to init with root_switch. it kick me then it the same networking problem.

~ # /sbin/cryptsetup open --type luks /dev/sda sdc
Enter passphrase for /dev/sda: 
~ # /sbin/lvm
lvm> vgchange -ay
  3 logical volume(s) in volume group "sdg" now active
lvm> exit
  Exiting.
~ # mkdir /newroot
mkdir: can't create directory '/newroot': File exists
~ # mount /dev/sdg/root /newroot
~ # mount --move /sys /newroot/sys
~ # mount --move /proc /newroot/proc
~ # mount --move /dev /newroot/dev
~ # exec switch_root /newroot /sbin/init

I'm on HC1 Armbian_20.08.1_Odroidxu4_buster_current_5.4.61_minimal

 

What do you need to help me to debug ?

Edited September 16, 2020 by rosbeef
typing errors

[rosbeef]

just what i do after flashing armbian on my sd.

 

# $erver
mkdir /etc/dropbear-initramfs/

apt install armbian-config vim rsync lvm2 cryptsetup dropbear-initramfs busybox nmap thin-provisioning-tools lxc nftables ebtables libxml2-utils iproute2 bridge-utils uidmap unattended-upgrades zram-tools

# €lient
cat .ssh/id_*.pub
# $erver
vi /etc/dropbear-initramfs/authorized_keys

#change ssh ports
sed -e "s/#Port 22/Port 2222/g" -i /etc/ssh/sshd_config
sed -e "s/^#CRYPTSETUP=/CRYPTSETUP=y/" -i /etc/cryptsetup-initramfs/conf-hook
sed -e "s/^#DROPBEAR_OPTIONS=/DROPBEAR_OPTIONS=\"-p 22222\"/" -i /etc/dropbear-initramfs/config

armbian-config
# change specific hardware config hc1 - reboot

------------------------------------------------
reboot or physically reboot
---------------------------------------------


test connectivity and halt
make disk copy with the disk linux tool

#statick mac based ip addr
Debian
export ETHID=$(nmcli -t -f device c);
export UUID=$(nmcli -t -f uuid c);
sudo nmcli con add ifname brW type bridge con-name brW;
sudo nmcli con add type bridge-slave ifname $ETHID master brW;
sudo nmcli con modify brW bridge.stp no;
sudo nmcli con down $UUID;
sudo nmcli con up brW;
sudo nmcli con modify brW ipv6.addr-gen-mode stable-privacy;
sudo nmcli con show;


sudo cryptsetup luksFormat --type luks2 /dev/sda

sudo cryptsetup luksOpen /dev/sda sdc

sudo pvcreate --dataalignmentoffset 1s /dev/mapper/sdc
sudo vgcreate sdg /dev/mapper/sdc
sudo lvcreate -L 16G /dev/mapper/sdg -n root
sudo lvcreate -L 200G /dev/mapper/sdg -n data
sudo lvcreate -l 100%FREE /dev/mapper/sdg -n save

sudo mkfs.ext4 /dev/sdg/root -E discard
sudo mkfs.ext4 /dev/sdg/data -E discard
sudo mkfs.ext4 /dev/sdg/save -E discard

sudo cryptsetup luksHeaderBackup /dev/sda --header-backup-file /boot/luksHeaderBackup.hex

-----------------------
Copy the root partition to SSD
-----------------------

sudo mount /dev/sdg/root /mnt
sudo rm -rf /mnt/*

sudo mkdir /mnt/data
sudo mount /dev/sdg/data /mnt/data

sudo mkdir /mnt/data/home
sudo mkdir /mnt/data/lxc
sudo mkdir /mnt/home

sudo mount --bind /mnt/data/home /mnt/home

sudo rsync -ahPHAXx --delete --exclude={/boot/*,/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found} / /mnt

sudo touch /mnt/etc/default/dropbear
echo 'NO_START=1' | sudo tee /mnt/etc/default/dropbear
# CRYPTTAB
echo 'sdc /dev/sda none luks,initramfs,discard' | sudo tee -a /mnt/etc/crypttab

# FSTAB
# comment root based on disk uuid
sudo sed -e 's/UUID=/#UUID=/' -i /mnt/etc/fstab

# mount root on root logical volume
sudo sed -e '1i/dev/sdg/root / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1\' -i /mnt/etc/fstab

# create and mount at startup folder for boot files
sudo mkdir /mnt/media/sdcard
sudo sed -e '2i/dev/mmcblk0p1 /media/sdcard ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 2\' -i /mnt/etc/fstab
sudo sed -e '3i/media/sdcard/boot /boot none default,bind 0 2\' -i /mnt/etc/fstab

# create and mount at startup folders for lxc privileged and unprivileges containers on data logical volume
sudo sed -e '4i/dev/sdg/data /data ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 2\' -i /mnt/etc/fstab
sudo sed -e '5i/data/home /home none bind 0 0\' -i /mnt/etc/fstab
sudo sed -e '6i/data/lxc /var/lib/lxc none bind 0 0\' -i /mnt/etc/fstab

# remove root login permission
sudo sed -e 's/^PermitRootLogin yes/PermitRootLogin no #yes/' -i /mnt/etc/ssh/sshd_config
sudo cp /boot/boot.ini /boot/boot.ori
sudo rm /boot/boot.new

sudo sed -e 's/root=[^ ]*/root=\/dev\/sdg\/root cryptopts=source=\/dev\/sda,target=sdc,lvm=sdg /' -i /boot/boot.ini
sudo update-initramfs -u

#then reboot
sudo reboot

then ssh my server on root@ip -p 22222

on the server in initrramfs i run cryptroot_unlock

then it tell me that unlock successfull then kickme

Edited September 18, 2020 by rosbeef

[rosbeef]

I tried from a fresh image flashed on sdcard and just

sudo mkfs.ext4 /dev/sda -E discard
sudo mount /dev/sda /mnt
sudo rsync -ahPHAXx --delete --exclude={/boot/*,/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found} / /mnt
sudo sed -e 's/UUID=[^ ]*/UUID=MY_SDA_SSD_SPECIFIC_UUID"/' -i /boot/boot.ini

That does not change the problem

[rosbeef]

It seems that the mmcblk number change from 0 to 1 so my boot folder didn't mount, so automatic "sudo update-initramfs -u" doesn't work.

i started from scratch and replace

sudo sed -e '2i/dev/mmcblk0p1 /media/sdcard ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 2\' -i /mnt/etc/fstab

by

sudo sed -e '2i/dev/mmcblk1p1 ...

What should be the origin of this change ?

Edited September 21, 2020 by rosbeef