Armbian SD card backup

[xwalter]

hello guys !!!

I want to make a backup of my sd card with armbian and other installed on it .

In windows not possible because the ext3-4 format of the card . In ubuntu I can see the armbian file system into the card but not copy because I don't have permission

So , I put again the card into my orange pi and I type sudo su and then password . I tyoe also sudo -s but nothing . Pratically some folders are not "visible" 

Do you have any suggestion?

ciao 

Walter

[wildcat_paris]

@xwalter

 

you can use the RPi SD card copier tool, it is a "community"-like tool

https://www.raspberrypi.org/blog/another-update-raspbian/

 

if you know some git / github / compilation tools, you can try my temporary mini-fork https://github.com/golfromeo-fr/piclone

 

or download the deb file (it may be faster)

https://share.orange.fr/#QULfXfE0CX974e72652

 

note: I had to change a line of code because Raspbian assumes Debian Jessie and {me,others} using Trusty, "ported" tool doesn't have the same output.

[xwalter]

@wildcat

How to install SD Card Copier on Armbian ? 

I don't have in accessories list ....

I have inserted my new SD card into an USB adapter and then plugged into my orange pi

With GParted I can see the new card ...and now ? 

[wildcat_paris]

@xwalter

 

download piclone_0.1_armhf.deb from my orange share

 

sudo dpkg -i piclone_0.1_armhf.deb

 

it is a live copy tool, so plug an empty USB key / USB microSD card reader on your board

The recipient has to be at least the size of your Armbian card.

 

note: let's hope you have all the needed packages installed. let's see.

[xwalter]

@wildcat

I have tryied but I don't have access ....

Then the new card is 8 Gb but armbian card is 16 .....uffff

[sooperior]

Are you root? you can do a backup with dd which is a tool usually included in all linuxes. But you need root or sudo to remove permissions problems, both for the filesystem on the card and for being able to read-write a boot sector

[xwalter]

ok ....could you please write  the command to do in my terminal ?

Thanks ! 

[xwalter]

I'm trying with 

dd if=/dev/mmcblk0p1 of=/dev/sdb1

...but from terminal I don't see nothing

[xwalter]

the files was copied successfully but the boot doesn't start ....uffff

[tkaiser]

the files was copied successfully but the boot doesn't start ....uffff

 

While I still don't know what you're trying to achieve just a small remark: If you try to clone just a single partition with dd then the resulting clone can not start. You would've to use

dd if=/dev/mmcblk0 of=/dev/sdb

instead to get a clone (this has nothing to do with backup of course)

[xwalter]

well. ....I would like to create an sd card as backup completed of system ,programs, and datas.

Try to image ....now I use an sd card to start armbian and I have already installed a lot of programs, services, netbeans project and else.

If the card crashes ?  I have to reinstall all from the beginning ....or not? 

But if I have my copy of the card I just change the sd card and start without any problem..... 

I hope now is clean what I'd like to do

[tkaiser]

I hope now is clean what I'd like to do

 

To make a probably unreliable clone from time to time the aforementioned approach might work (but you have to specify whole devices since otherwise bootloader and partition table and so on aren't written to the target). So using 2 identical SD cards this should work:

dd if=/dev/mmcblk0 of=/dev/sdb

(given sdb is the 2nd SD card). Cloning already running systems especially if databases and stuff like this is open might not work, so cloning an offline system (using a PC for example) is always the recommended way. And then still this is no backup, just a clone.

 

But since people don't care that much creating a clone from time to time is better than nothing. Some background information: http://forum.armbian.com/index.php/topic/897-backup-restore/

[xwalter]

yes is a clone , for example I have 1000 orange pi and I want to start all of then at same way .

I follow your link now ....

[xwalter]

I did the same job under Ubuntu desktop pc but nothing 

dd if=/dev/sdb1 of=/dev/sdg1

where 

sdb1= sds card that I use with orange pi 

sdg1= new sd card where I want to put the clone 

I don't want to believe that here you are a lot of wizard in linux and nobody know how to clone an sd card ..... 

[tkaiser]

I don't want to believe that here you are a lot of wizard in linux and nobody know how to clone an sd card ..... 

 

And I really don't want to believe that you're constantly ignoring the answers you get. You're doing it still 100 percent wrong since you're cloning partitions instead of devices. As already written: "but you have to specify whole devices since otherwise bootloader and partition table and so on aren't written to the target"

[xwalter]

ok but how ? ....If I know I dont ask you , right ?   

In windows I cannot use win32DiskImager because widows ask me to format the sd card before....

[jernej]

Just remove numbers at the end and you will copy whole disks instead partitions. Use for example /dev/sdb instead /dev/sdb1

[tkaiser]

ok but how ?

 

I gave you two times working examples already in this thread. Remove the numbers and you get the whole device (which is necessary since the bootloader lives before the 1st partition and without the installation on card is not able to boot your board).

[xwalter]

ok guys I'm going to do that  

Just a question ....Do I have to format the new sd card as ext4 or normally fat32 ?

[tkaiser]

Do I have to format the new sd card as ext4 or normally fat32 ?

 

Neither nor since you're cloning the whole device. The partition table and also all filesystem metadata will be cloned bit to bit identical (if the target card has at least the same amount of available sectors than the source otherwise you'll run into an error). You're not transferring files at all but this happens on the basis of block devices.

[martinayotte]

Absolutely ! (if both SDCards are the same size or if the new one is bigger than the old one.)

Otherwise, you should use the method described in the other thread.

[xwalter]

ok guys I have to follow other method because now I have the only one new card smaller then armbian card ....otherwise I get an error when I boot ....I have to order dome new 16Gb cards 

[tkaiser]

This thread lead to a new 'enhancement' discussion: https://github.com/igorpecovnik/lib/issues/343

 

So thanks for asking this.

 

BTW: Another approach is to do an offline clone into a compressed file. You would shutdown your device, then insert SD card into another Linux or OS X host (Windows should also work when using advanced tools like Rufus but it's 2016 now and I would believe no one is still using Windows any more?! ) and then create a compressed device image. Given the SD card is available as /dev/sdb it would look like the following to create a device image and to restore it to a new card:

dd if=/dev/sdb bs=10M | gzip -c >/path/to/Armbian-clone.img.gz
gunzip -c /path/to/Armbian-clone.img.gz | dd of=/dev/sdb bs=10M

This way you could clone your installation from time to time and would need to buy a new card only when your current card died. But as long as the aforementioned feature request isn't implemented you might struggle when trying to buy a new SD card of 'the same size' since exact capacity slightly differs.

[xwalter]

tkaiser, thanks for yours preciouses info.

I made this way 

dd if=/dev/sdh of=/dev/sdb

from 8 Gb --> 8 Gb , it seems to work well .But long time and then I did also 

dd if=/dev/sdh of=/home/walter/Armbian_backup/Armbian_bck.img

on Ubuntu ,and now I have the complete image in less time . I hope to restore I will type

dd if=/home/walter/Armbian_backup/Armbian_bck.img of=/dev/sdh 

or I copy the image on windows and I will use Win32ImageWriter 

 

Ok guys , now I'm cool .....

[tkaiser]

Well, I wrote the instructions above for a reason. When you don't provide the 'bs=' parameter then dd will use its defaults (1 block == 512 bytes) which slows down the cloning process like hell:

     bs=n     Set both input and output block size to n bytes, superseding the
              ibs and obs operands.  If no conversion values other than
              noerror, notrunc or sync are specified, then each input block is
              copied to the output as a single block without any aggregation
              of short blocks.

Also it's somewhat weird to not compress the image on the fly. So now just as a reference and as preparation for a hopefully improved Armbian documentation soon.

 

One way to do an offline clone from an Armbian installation with minimum size requirements would be to fill out all the unused space with zeros before (this really helps with compression afterwards if a lot of filesystem activity happened on the SD card before!) and then use a more efficient packer. So given the SD card is /dev/sdd you would do

mkdir /mnt/clone
mount /dev/sdd1 /mnt/clone
dd if=/dev/zero of=/mnt/clone/empty.file bs=10M || rm /mnt/clone/empty.file
umount /mnt/clone && rmdir /mnt/clone
dd if=/dev/sdd bs=10M | gzip -c >/path/to/gzipped.img.gz
dd if=/dev/sdd bs=10M | 7zr a -bd -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on -si /path/to/7zipped.img.7z

I did this with a quite normal Armbian desktop image on an 8 GB SD card that looks like this

/dev/sdd1                    7,3G    2,1G  5,2G   29% /mnt/clone

Instead of an uncompressed image that is 7.3GB in size the zeroed out 7z is 1.2GB in size and the gzip variant 1.7GB (please note that you could add '-9' then gzip would compress a little bit better).

 

But here comes a more intelligent approach: Only outlined as a stub since others should do the job and try to earn a REWARD for this:

 

When there's already a PC running Linux with a somewhat recent kernel where we can insert the SD card to create an offline clone then to combine the best of two worlds would be:

• Create a btrfs filesystem using maximum compression (compress=zlib)
• Configure keyless SSH authentication between your Linux host and the Armbian installation on the SBC to be able to run rsync later to copy filesystem's contents from the SBC to your Linux host
• Shut down your SBC, insert the SD card with the Armbian image into the PC, zero the unused space out and create an uncompressed dd image (the image will not be compressed internally since the filesystem does this job. So based on the aforementioned example the image we will create will show as being 7.3GB large but will only need approx. 1.3GB on disk since btrfs provides transparent file compression)
• Eject the SD card and start your SBC again

Now setup a script on the Linux host that uses the btrfs command, losetup and rsync to

• create a snapshot of the btrfs filesystem where the uncompressed image is hosted
• mount the 1st partition of this image
• use rsync to incrementally update the OS image on your Linux host with the contents of / on the SBC (/dev/mmcblk0p1) through the network
• set up a cron job that does this automagically

This way a whole bootable OS image only using the minimum amount of needed space will be stored on the Linux host and incrementally be updated. Since btrfs allows for transparent file compression and supports snapshots in case 100 MB have changed between two rsync runs (and given this data is compressable by 1:2) the whole storage requirements to store the new OS image variant will only be 50MB more. Yes, you have two independant OS images using this snapshot and compression technique that require not 8GB + 8 GB but just 1.3GB + 50MB.

 

If such a script has been set up then we're starting to talk about backup and not stupid cloning any more. Since then we're talking about

• periodically saving the filesystem's contents
• to a different location
• implementing versioning (old/deleted/changed stuff remains on the Linux host in the form of older snapshots)
• allowing also for desaster recovery (since we created a bootable OS image in the first place)

And the best news: The first step does also work through the network so in case we want to backup an internal eMMC then we boot the SBC with an SD card and transfer the dd image through the network to the Linux host. Also if you want to save a fleet of Armbian installations then simply also activate deduplication in btrfs and you'll end up with disk storage of below 2 GB to store eg. 8 Armbian desktop installation all being +2 GB in size

 

Now it's time for someone to write this up as a nice tutorial and start to code this armbian-clone script. And please remember: If you do start contributing to community efforts there's something you'll get in return: http://forum.armbian.com/index.php/topic/1325-claim-a-task-and-do-it/

[tkaiser]

Just as an addition since as usual stuff already exists on the internet:

• Regarding the combination of rsync with btrfs: https://github.com/oxplot/rsyncbtrfs
• And in case mainline kernel is used and a custom Armbian image not using ext4 but btrfs instead the whole approach will be even more efficient since then 'btrfs send/receive' could be used between SBC and backup host: https://github.com/yoshtec/snapbtrex

[xwalter]

[Masterchippy]

I am a convert to Linux and still coming to terms with command line and find dd command quite intimidating. Cannot remember where I read it but find the Disks utility useful to create and  restore images.

On Ubuntu Mate Disks is in Menu>Accessories.

With the SD card in the reader.

Click on the SD card entry on the left.

Click on the 3 bar settings icon and select  Create or restore.

If you are creating an image from your SD card give it a name and start. You will need to enter your root password.

if restoring find the image file and restore . again password may be required.

MAKE SURE YOU START BY SELECTING THE SD PARTITION it would be disastrous to write an image to your main linux partition

Not sure if you could do this from Armbian from your boot disk.

You could do this from a live distro.

 

Hope this helps someone

 

Oh this works for RaspberryPi and Open elec as well

[Ford Prefect]

I almost understood all this interesting subject.

I too am beginning to have lots of micro SD cards for testing .

And mainly 32GB ones with only a few GB used.

Is there a way to allow the copy to be restored on smaller ones to ease distribution and reduce time spent ?

Filling free space with zero is a good idea to save space anyway.

Could overwriting an original Armbian distribution with backups of all the files be a solution ?

[tkaiser]

Is there a way to allow the copy to be restored on smaller ones to ease distribution and reduce time spent ?

Filling free space with zero is a good idea to save space anyway.

 

Overwriting all unused space with zeros does only help when trying to create a compressed image from card (and when lots of data has changed before or has been deleted).

 

Currently a restore from a backup image to a smaller card is not possible since partition layout will exceed the size of the smaller card. To be honest: This might also be true for cards 'of the same size' since the actual capacity might slightly differ and if the original card was just one sector larger than the new card you run in the same problem (we fixed this 2 weeks ago and leave now some space unused to help with exactly this problem here -- but this only helps if you use a new Armbian image created within the last 2 weeks)

 

In case you know that you just want to test something on a 32 GB card and want to move this to an 8 GB card later, you could also create the file /root/.rootfs_resize containing '24%' prior to first boot (so you need a Linux host to adjust that). If you're creating your own Armbian images using our build system choosing images with a fixed size might be smarter: https://github.com/igorpecovnik/lib/blob/master/documentation/geek-faq.md#partitioning-of-the-sd-card