Armbian SD card backup

[rodolfo]

Hotcloning Armbian SDcard ( tested on OPI ONE/LITE  )

 

You need a USB card reader and Linux. With a running Armbian you've got all the Linux you'll ever need. The aim is to copy the entire system on SDcard including bootstuff and our personal belongings, roots, cats, dogs etc... to a new SDcard. The new card must only hold the data content of the old card and can be of different size (smaller, larger). The new SDcard contains one big partition of maximum size and is bootable.

 

Setup

 

1. Download the script and rename it to armbian_hotclone.sh

 

2. Start your OPI ONE with the SDcard you want to copy and connect to it via ssh ( or putty ).

 

3. The script needs to be run on the board ( in my case OPI ONE ). Copy the script to your board and make it executable ( chmod +x )

 

4. Attach a USB card reader to the OPI ONE. Make sure there is NO OTHER USB storage attached.

 

Running the script

 

5. Insert the target SDcard into the card reader and check it has been detected ( lsblk )

 

6. Run the script as root ( sudo ) and depending on the card reader and SDcards wait 10-60 minutes

 

Test the new SDcard

 

7. Shutdown the OPI ONE.

 

8. Replace the SDcard in the slot with the new copy

 

9. Make sure the OPI ONE boots correctly before you put the SDcard into the cookie jar for desaster recovery and worse.

 

Notes on usage

 

The script is just a skeleton to showcase the basics. A break needs to be added to prevent it from running accidentally and eating up disks.

 

Warning : Do NOT run the script on your host

 

armbian_hotclone.sh.txt

 

Enjoy !

[tkaiser]

The script is just a skeleton to showcase the basics. 

 

And it's absolutely necessary to get the idea that what you're trying to do can NOT work reliably in every situation (eg. running databases they will get corrupted with such rsync attempts -- applies to our nand_sata_install.sh script as well which is why I want execution of this script as soon after installation as possible).

 

Also your script doesn't take care of 2 partition Armbian installs and the partitioning stuff is prone to errors since fdisk behaviour might change depending on version number so you should add with which of the 4 distro variants Armbian supports you tested.

 

The only way to reliably clone an installation is by shutting the installation down before and then using a second installation to create an offline clone (using dd or rsync or whatever). Cloning a 'hot system' is not possible by design without taking care of several things. It might work sometimes (or seems to work after a first check just to realize later that some minor issues start to get bigger). It's also wrong to use a single rsync instance, always add a second one at the end otherwise you increase the chance of inconsistencies greatly (to get the idea: Start your clone process and then 1 minute later do an 'apt-get upgrade' -- that might happen unattended anyway to install security updates and chances are great that dpkg issues happen or even the installation is corrupted)

 

I know that people don't like these warnings and ignore them all the time since it looks inconvenient to do it correctly and hot cloning seems to work. It will not in every situation and you will realize that always way too late. You have been warned

[rodolfo]

And it's absolutely necessary to get the idea that what you're trying to do can NOT work reliably in every situation

 

Disclaimer : The script does not save the planet.  

[tkaiser]

Disclaimer : The script does not save the planet.  

 

Ok, I should have been more precise: It's broken by design since cloning an already running OS installation does not work reliably without taking care of some stuff (that's the reason filesystems supporting snapshots do exist, that's the reason shadow copy service on Windows has been invented -- without that stuff it's just luck if a 'hot cloned' installation behaves absolutely identical as before or not).

 

That being sad: A normal fresh Armbian installation without many apps running might be cloneable. But as soon as some background or foreground activity happens a single rsync attempt will fail for sure since inconsistencies are introduced. And most likely you'll realize that way too late. People thinking about using your script try to do some sort of 'backup' trying to prevent data loss. If that's the case an attempt that will always work should be preferred. And that means: Shutting down the running system before. Since only this works 100 percent reliably given that we're dealing with ext4 and rsync here.

 

But in the end I don't care that much since I do this partially for a living (helping companies recovering from such broken cloning / backup attempts)

[tkaiser]

BTW: It's pretty easy to check what's wrong with 'hot cloning' attempts. Armbian ships with lsof and iotop. Just check how many open files you have (especially when running a desktop installation with some apps open) and / or let iotop list what happens in the amount of time your cloning process runs.

 

If rsync needs 30 minutes for the first clone run, then try the following out (using the same amount of time -- 1800 seconds). Start this, then do the normal stuff you do and check what has changed within 30 minutes afterwards:

sudo iotop -P -b -o -t -d 1800 -q

Most of these changes will be missing at the destination so always run rsync another time directly after the first call. And if the Armbian installation is a desktop image and the user in question uses Firefox or Midori then the whole cloning can't work for sure. Since these browsers use several SQLite databases all the time that can't be cloned in a reliable way when they're open. Cloning should be done offline or at least in single user mode

[rodolfo]

That being sad: A normal fresh Armbian installation without many apps running might be cloneable.

 

Bingo !

 

You are of course perfectly right about "cloning". To properly backup/restore a system image you always need another system or proper virtualization.

People with Linux or any other X-ses background will not need explanations.

 

We are here in the world of tinker boards and some people might enjoy a "quick and dirty" way to replicate the SDcard of a $12 board

[tkaiser]

We are here in the world of tinker boards and some people might enjoy a "quick and dirty" way to replicate the SDcard of a $12 board

 

Sorry, but cheap hardware price does not legitimate a broken solution

 

We're here at Armbian and we spend the same amount of time/efforts on all the devices we support be it an OPi One or a Clearfog that costs 20 times more. Since users care and not the costs of their hardware.

 

I've already been a bit verbose in this thread in the hope someone (not me) would start thinking about adding stuff like an 'armbian-sync' tool. We already have a few bits of code here and there (firstrun -- partitioning that works everywhere -- or nand_sata_install.sh -- doubled rsync call to prevent data inconsistency). Why not trying to improve further? Such an 'armbian-sync' could be used in the following way:

• User connects his USB card reader with the target device
• Then 'sudo armbian-sync' will be started, does the user interaction (checking which device to use and then relying on the PARTUUID and not /dev/foo/bar), writes all the stuff to /etc/armbian-sync.conf, enables the execution of /etc/init.d/armbian-sync on next boot, asks the final questions and issues the reboot
• On the next reboot /etc/init.d/armbian-sync will do the syncing, deactivates itself (see firstrun) and ejects the card when done

This way the whole cloning approach can be used by anyone and it's guaranteed to work (if we add a hook to check for well known database packages that should all be stopped prior to starting any sync).

 

On a related note: nand_sata_install.sh should switch to this mode too since it's more reliable and in the end exactly the same task.

[wildcat_paris]

@tk

 

if we can also have the concept of SD/eMMC to USB HDD, it would be nice. thanks

 

https://github.com/golfromeo-fr/piclone

You can also create a dump image of a SDcard on a USB hard drive for archive1/ create an empty file on your harddrive about the size of your SD card (for example 32GB)	cd /mnt/myharddrive	dd if=/dev/zero of=PicloneSAVE_YYYY_MM_DD.bin bs=100M count=3202/ attach the file to a loopback device (example "/dev/loop0")	sudo losetup /dev/loop0 PicloneSAVE_YYYY_MM_DD.bin3/ use piclone with /dev/loop0 as target4/ detach loop device 	sudo losetup -d /dev/loop05/ you can use md5sum / sha1sum / sha256sum on the file PicloneSAVE_YYYY_MM_DD.bin as usual6/ you can zip the image using 7zip or another compression tool 

 

note: piclone is designed for Unix "runlevel 1" aka single user mode

[boobypi]

Hi guys,

Why just doing an backup in hot cloning mode?
Thnks for iotop tip,,, but put data in read only mode like linux on usb key is a secure way to clone entire partition : fstab edition or mount -o remount,NEWARGS filesystem.

Maybe it's possible to improve this tool.

I go back to my openwrt system (take lot of time to get a ready -saying faster- system : gcc, ffmpeg, nodejs). H3 is expensive in time

[tkaiser]

piclone is designed for Unix "runlevel 1" aka single user mode

 

No thanks. What's the purpose of writing stuff in C when the most critical parts rely on external tools: https://github.com/golfromeo-fr/piclone/blob/65fb5d95f84aa7ec63a93f8f944beaa0290e7c73/src/piclone.c#L140

 

The behaviour of tools like parted or fdisk might change when upgrading a package and then the whole thing might fail (and destroy your whole installation! That's why I asked @Rodolfo whether he tested his script on all 4 distro variants Armbian supports! Since it matters!). We just recently had to include a check which fdisk version we're using: https://github.com/igorpecovnik/lib/blob/2e284be77d62f7516901f7657f8e773c271aa953/scripts/firstrun#L207-220

 

So we wasted already hours (days to be honest) with testing, we made workarounds for this and that, we changed behaviour of partitioning for a reason (never ever use whole devices again since it might prevent our users from cloning a card to another card 'of the same size' since cards from different manufacturers/batches differ slightly in size so an 8GB card from Samsung isn't exactly the same size as one from Transcend) and we should not throw this stuff in the bin to replace it with tools that might work now but break with any package or OS update in the future (we do not only support Raspbian but 4 different Debian based distros).

 

The core stuff I was talking about is already there (contained in firstrun script and nand_sata_install.sh where some adoptions are IMO needed). Let's use this, maybe put the core logic in a library to be used from different tools later to achieve identical results and rely on simple scripts we can easily adopt if changes make it necessary (see above: util-linux version on Ubuntu Xenial has been increased to 2.27.1 and immediately partition resizing broke)

[wildcat_paris]

@tk

 

to please your eyes, bold & righteous TK

 

added feature requests.

https://github.com/raspberrypi/piclone/issues

[rodolfo]

And it's absolutely necessary to get the idea that what you're trying to do can NOT work reliably in every situation

 

There is no way to do a safe "hotclone" no matter what precautions ( double, triple, quadruple rsyncs etc.. ) you do and any efforts in that direction are a simple waste of time. I've seen enough sad hopeless hapless restore attempts by "backup-experts" on systems costing magnitudes more than thousands of ARM boards. It probably takes a bit of restore practice instead of all the academic backup concepts for the learned to realize what actually works

 

Working backup/restore/image solutions always depend on a complete freeze of the ressources envolved. The rest is either quick-and-dirty or the more common slow-and-dirty. In other words - you need an independent second rescue system. There are several ways to accomplish this task :

 

- Run everything virtualized ( silly joke - we are talking embedded )

 

- Rip the storage medium out of the board ( Only works with SDcard or other removable storage ) and handle it on another system ( External solution with ugly dependency ). Then dd,rsync,tar,zip... yadda,yadda,yadda.

 

- Run an  "Armbian rescue system" in ram on the board itself and fully access all storage on the board.  Then dd, rsync, tar, zip, <any-odd-way> .. yadda,yadda,yadda.

 

The reason we are already at post #42 in solving the simple tasks of backup/restore/image is that the discussion concentrates on solving the wrong problem. Once "Universal booting to rescue system in ram with mount/unmount read/write-access to all board storage" ( FEL-boot a live-system ) is solved, the rest is a simple matter of dd, rsync, tar, zip , <any-odd-way> ... yadda,yadda,yadda.

[zador.blood.stained]

The reason we are already at post #42 in solving the simple tasks of backup/restore/image is that the discussion concentrates on solving the wrong problem. Once "Universal booting to rescue system in ram with mount/unmount read/write-access to all board storage" ( FEL-boot a live-system ) is solved, the rest is a simple matter of dd, rsync, tar, zip , <any-odd-way> ... yadda,yadda,yadda.

What do you need to solve here? The "universal" part? If you prepare u-boot images and .bin files (H3) and .dtb file (A10/A20) for all available boards and give user a selection menu, this part is solved. The only requirements are

a ) Linux host (another SBC will work)

b ) network connection (wired Ethetnet or g_ether will work, wireless is significantly harder AFAIK due to shared libraries dependencies for wpasupplicant).

[rodolfo]

@zador

 

Sounds great ! .Is there a working example of a live-system ( RAM only ) for H3 ( OPI ONE/LITE)  I could play with ?  

[zador.blood.stained]

@zador

 

Sounds great ! .Is there a working example of a live-system ( RAM only ) for H3 ( OPI ONE/LITE)  ? 

It's not a "live system" exactly, it is "root on NFS" started from FEL: https://github.com/igorpecovnik/lib/blob/master/documentation/fel-boot.md

 

Setting up RAM only system is not wise because FEL transfer speed is too slow (less than 1 MB/s) to load anything close to full OS. If you don't want to use NFS at all, you can try to pack busybox and rsync into initrd and don't use rootfs at all.

[rodolfo]

@zador

 

Thanks - I'll have a look at it.

[rodolfo]

It's not a "live system" exactly, it is "root on NFS" started from FEL: https://github.com/igorpecovnik/lib/blob/master/documentation/fel-boot.md

 

Setting up RAM only system is not wise because FEL transfer speed is too slow (less than 1 MB/s) to load anything close to full OS. If you don't want to use NFS at all, you can try to pack busybox and rsync into initrd and don't use rootfs at all.

 

Thanks for the info. Trying to set that stuff up ended in dependency hell ( Stable Debian e basta ! ). Digging a bit deeper into the sunxi u-boot stuff I just realized how simple a real solution could be.

 

Working backup/restore/imaging solutions are simple, self-contained and independent

 

OPI ONE/LITE ( probably also other H3 boards ) can start from SDcard, USB storage, network and default to FEL-mode when started without SDcard. Network nfs, tftp-boot and FEL-mode have ugly external dependencies

on other systems clearly violating that principle. That leaves us basically with SDcard and USB storage. SDcard is eliminated by the fact that our "rescue" attempt should probably restore that SDcard. So we are at USB storage.

 

Here's the trick :

 

- Create a bootable USB-Drive/Partition from your working installation ( you can hotclone the sdcard to USB and adjust /boot/boot.cmd and /etc/fstab  or use the armbian utility nand-sata-install.sh )

- Properly label the USB-Rescue-Partition ( e.g set a mark /boot/xxrescuexxx for low-level u-boot to recognize it )

- Use a costumized boot.cmd that checks for the xxrescuexxx flag and boots from usb if found. Install it on SDcard and USB-storage

- et voilà : a completely independent rescue system running on the board accessing FROZEN ASSETS WITH 100% DATA CONSISTENCY.

 

I've tested the workability of this approach on OPI ONE. All USB ports ( in the case of OPI ONE 1 normal and 1 OTG ) can be fully used in host mode. All one needs for a fully working rescue mode is one ( additional or single ) partition on USB storage.

 

Now that an independent rescue system is solved on the basis of stock Armbian and requiring nothing but stock Armbian people can use their favourite brand of Linux, Windows, OS-X to flash their first Armbian SDcard and from then on simply carry on with Armbian.

 

Next step will be prototyping a simple versioned backup/restore/imaging setup for OPI ONE/LITE.

[ARMgunnaservu]

great read guys!,I have copied my opi emmc onto a usb stick, i wonder if i can use resize2fs -m on ubuntu(pc) and then be able to burn it onto smaller sd card. I'll be back to re-read...2..or 145 times.

[Ivan Voronov]

Hi All,

 

I am just trying to clone my armbian 5.25 to new SD using armbian_hotclone.sh but getting the following error on boot for the new SDcard:

 

[    7.599681] ALSA device list:
[    7.602722]   No soundcards found.
[    7.609298] Freeing unused kernel memory: 1024K (c0a00000 - c0b00000)
[    7.616632] mmc1: new high speed SDIO card at address 0001
Loading, please wait...
starting version 229
[    7.734173] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    7.742363] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    7.743141] random: udevadm: uninitialized urandom read (16 bytes read)
[    7.746219] random: udevadm: uninitialized urandom read (16 bytes read)
[    7.746928] random: udevadm: uninitialized urandom read (16 bytes read)
[    7.747665] random: udevadm: uninitialized urandom read (16 bytes read)
[    7.748380] random: udevadm: uninitialized urandom read (16 bytes read)
[    7.749136] random: udevadm: uninitialized urandom read (16 bytes read)
[    7.749908] random: udevadm: uninitialized urandom read (16 bytes read)
[    7.750786] random: udevadm: uninitialized urandom read (16 bytes read)
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
Begin: Waiting for root file system ... Begin: Running /scripts/local-block ... done.
[   20.657393] random: fast init done
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
done.
Gave up waiting for root device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  UUID=a0ad0452-614b-409e-b154-103d34996c13 does not exist.  Dropping to a shell!
Rebooting automatically due to panic= boot argument

 

Any thoughts on what is going wrong?

 

Thanks in advance!

[martinayotte]

1 hour ago, Ivan Voronov said:

ALERT! UUID=a0ad0452-614b-409e-b154-103d34996c13 does not exist.

Seems that you new SDCard doesn't have the same UUID...

Check its UUID using "blkid" and tweak /boot/armbianEnv.txt and /etc/fstab accordingly.

 

[Ivan Voronov]

1 hour ago, martinayotte said:

Seems that you new SDCard doesn't have the same UUID...

Check its UUID using "blkid" and tweak /boot/armbianEnv.txt and /etc/fstab accordingly.

 

After updating UUID done I can boot from the new SD card! Thanks!

 

The only issue on boot is

[FAILED] Failed to activate swap /var/swap.
See 'systemctl status var-swap.swap' for details.
[DEPEND] Dependency failed for Swap.

My fstab

UUID=65fa8bb4-f653-48e2-a453-76706142372f / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1
tmpfs /tmp tmpfs defaults,nosuid 0 0
/var/swap none swap sw 0 0

The only UUID has been updated there. Any suggestion?

[martinayotte]

Do you have that /var/swap file ?

 

[Ivan Voronov]

17 minutes ago, martinayotte said:

Do you have that /var/swap file ?

 

Yes,  I do have it on the original & cloned SD card
 

root@bcam:/var# ls -l | grep swap
-rw-------  1 root root   134217728 Feb  2  2017 swap

 

[Igor]

Try recreating swap.

[Ivan Voronov]

49 minutes ago, Igor said:

Try recreating swap.

swapon: swapfile has holes per journal.

Re-creating of swap file SOLVED the issue. 

THANKS!