Creating a backup SD card (* .img) to another media or network

electric · September 16, 2015

Hello. Sorry for my English, I use google-translate.

Maybe someone already has experience and uses script / program to create a backup SD card entirely in the form of an image on an external storage (HDD, USB-flash), or over the network?

I am at the moment once a week turn off the system (I have a banana pi), remove the SD card and make a copy of the system image (using imagewriter.exe).

I want to automate the process, and without removing the SD card from the board. It's real?

Many thanks!

Igor · September 17, 2015

The smartest / perfect backup is if you backup only what's needed - your work on the top of the system.

I have a script to install all services that I need and a backup where are only configuration files + my data. Both data and configuration files are saving periodically. Restore can be done easy from scratch or only from older data.

This way of course requires deeper knowledge of the operating system.

Average way would be to save directories with your data and system configuration (/etc /boot). For this, some basic knowledge will be enough.

Saving the whole file-system periodically is possible but not recommended but if you insist - you can do it this way:

tar czpf backup.tgz --directory=/ --exclude=dev/* --exclude=proc/* --exclude=run/* --exclude=tmp/* --exclude=mnt/* .

Note that this backup is not boot-able image but the content of it - it's all but the u-boot.

ilu · December 20, 2015

What would be the best way to backup a full bootable image of the sd-card on my harddisk which can be restored to the SD in case I muck everything up? Of course I don't mind putting the SD into my main computer (which has Debian).

I know I could use dd but this always frightens me - one mistake and my (important) data could be gone.

Wolf2000 · December 20, 2015

Hi
Win32DiskImager me Read

Tido · December 20, 2015

create a backup SD card entirely

Hi, just recently I received a suggestion to do this. The bash script is written, but it needs some maintenance.

Have a look and any idea is welcome

Cheers

Tido

hardvk0 · February 2, 2016

Hi.

I just came from the world of RPi.

In raspbx there is a script that performs a full backup of the SD card.

In my case helped by crond, overnight, I mount an SMB route, and the script dumps the differences in the image.

The script is raspbx-backup.

I do not know how to adapt to this new image, because now there is only one partition and before had two, Boot and rootfs. I only know how to read between the lines.

For me this was a blessing because it could break anything and in five minutes was like new.

#!/bin/bash
# RasPBX backup utility, version 1.2
# Author: Gernot Bauer <gernot@raspbx.org>
#
# This file is part of RasPBX.
#
# RasPBX is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.
#
# RasPBX is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# files
MOUNTPATH=/media/bkup
BOOTMOUNT=/boot
DEFAULT_RPI_IMAGE_SIZE=3724
DEFAULT_BBB_IMAGE_SIZE=1828
DEFAULT_IMAGE_SIZE=$DEFAULT_RPI_IMAGE_SIZE

# subroutines
setup_lo() {
  DEVICE1=`/sbin/losetup -f`
  if [ -z "$DEVICE1" ]
  then
    echo "Error setting up loop device."
    exit 1
  fi
  /sbin/losetup -f -o 1048576 --sizelimit 78643200 $IMAGE_FILE
  if [ $? -ne 0 ]
  then
    echo "Error setting up loop device."
    exit 1
  fi

  DEVICE2=`/sbin/losetup -f`
  if [ -z "$DEVICE2" ]
  then
    echo "Error setting up loop device."
    /sbin/losetup -d $DEVICE1
    exit 1
  fi
  /sbin/losetup -f -o 80740352 $IMAGE_FILE
  if [ $? -ne 0 ]
  then
    echo "Error setting up loop device."
    /sbin/losetup -d $DEVICE1
    exit 1
  fi
}

delete_lo() {
  /sbin/losetup -d $DEVICE1
  if [ $? -ne 0 ]
  then
    echo "Error removing loop device."
    /sbin/losetup -d $DEVICE2
    exit 1
  fi

  /sbin/losetup -d $DEVICE2
  if [ $? -ne 0 ]
  then
    echo "Error removing loop device."
    exit 1
  fi
}

mount_image() {
  if [ ! -d $MOUNTPATH ]
  then
    mkdir $MOUNTPATH
  fi
  mount $DEVICE2 $MOUNTPATH
  if [ $? -ne 0 ]
  then
    echo "Error mounting root partition. Wrong image file?"
    delete_lo
    exit 1
  fi

  if [ ! -d ${MOUNTPATH}${BOOTMOUNT} ]
  then
    mkdir -p ${MOUNTPATH}${BOOTMOUNT}
  fi
  mount $DEVICE1 ${MOUNTPATH}${BOOTMOUNT}/
  if [ $? -ne 0 ]
  then
    echo "Error mounting boot partition. Wrong image file?"
    umount ${MOUNTPATH}
    delete_lo
    exit 1
  fi
}

unmount_image() {
  umount ${MOUNTPATH}${BOOTMOUNT} 
  if [ $? -ne 0 ]
  then
    echo "Error unmounting boot partition."
    umount -l ${MOUNTPATH}${BOOTMOUNT} 
    umount -l ${MOUNTPATH}
    delete_lo
    exit 1
  fi

  umount ${MOUNTPATH}
  if [ $? -ne 0 ]
  then
    echo "Error unmounting root partition."
    umount -l ${MOUNTPATH}
    delete_lo
    exit 1
  fi
}

create_image() {
  if [ -f $IMAGE_FILE ]
  then
    echo "Image file $IMAGE_FILE already exists!"
    echo "Please remove it manually or specify a different file."
    echo "Call \"raspbx-backup $IMAGE_FILE\" to make a backup to this file."
    exit 1
  fi
  
  apt-get -y install dosfstools rsync parted
  if [ $? -ne 0 ]
  then
    echo "Error installing required tools."
    exit 1
  fi
  
  echo "Creating image file. This will take a while..."
  SEEKMB=$(expr $IMAGE_SIZE - 1)
  dd bs=1M count=1 seek=$SEEKMB if=/dev/zero of=$IMAGE_FILE
  if [ $? -ne 0 ]
  then
    echo "Error creating image file."
    exit 1
  fi

  LASTSECTOR=$(expr $IMAGE_SIZE \* 2048 - 1)
  sync
  parted $IMAGE_FILE << EOF
mktable msdos
unit s
mkpart primary fat32 2048 155647
mkpart primary 157696 $LASTSECTOR
quit
EOF
  if [ $? -ne 0 ]
  then
    echo "Error creating partitions."
    exit 1
  fi

  setup_lo

  mkfs.vfat $DEVICE1
  if [ $? -ne 0 ]
  then
    echo "Error creating FAT partition."
    delete_lo
    exit 1
  fi

  mkfs.ext4 $DEVICE2
  if [ $? -ne 0 ]
  then
    echo "Error creating ext4 partition."
    delete_lo
    exit 1
  fi

  mount_image
  
  mkdir ${MOUNTPATH}/dev ${MOUNTPATH}/media ${MOUNTPATH}/mnt ${MOUNTPATH}/proc ${MOUNTPATH}/run ${MOUNTPATH}/sys ${MOUNTPATH}/tmp
  if [ $? -ne 0 ]
  then
    echo "Error creating directories."
    unmount_image
    delete_lo
    exit 1
  fi
  chmod a+rwxt ${MOUNTPATH}/tmp

  unmount_image
  delete_lo
}

do_backup () {
  if [ ! -f $IMAGE_FILE ]
  then
    echo "Image file $IMAGE_FILE does not exist."
    exit 1
  fi

  setup_lo
  mount_image

  sync
  rsync -aDH --partial --numeric-ids --delete --force --exclude '/dev' --exclude '/media' --exclude '/mnt' --exclude '/proc' --exclude '/run' --exclude '/sys' --exclude '/tmp' --exclude 'lost\+found' --exclude '/etc/udev/rules.d/70-persistent-net.rules' / ${MOUNTPATH}/
  if [ $? -ne 0 ]
  then
    echo "Error running backups."
    unmount_image
    delete_lo
    exit 1
  fi

  unmount_image
  delete_lo
}


# start
if [ `grep -c "/boot/uboot" /etc/fstab` -ne 0 ]
then
  BOOTMOUNT=/boot/uboot
  DEFAULT_IMAGE_SIZE=$DEFAULT_BBB_IMAGE_SIZE
fi

# command line options
for i in $*
do
  case $i in
  *)
    IMAGE_FILE=$i
    ;;
  esac
done

if [ ! -z "$IMAGE_FILE" ]
then
  do_backup
  exit 0
fi

echo "RasPBX backup utility. Read the documentation here: http://raspbx.org/backup"
echo
echo "In the following process a new image file will be created, mirroring your"
echo "complete system. The size of this file should be identical to the size"
echo "of your SD card. Put this file onto a mounted USB thumb/hard drive or"
echo "mounted network share. Don't put it on your SD card!"
echo "Once the image file is set up backups can be made to it in an automated"
echo "fashion with:"
echo
echo "raspbx-backup <path_to_image_file>"
echo

while [ -z "$IMAGE_FILE" ]
do
  echo -n "Image file including full path: "
  read IMAGE_FILE
done

ROOT_PART=`cat /proc/cmdline | sed 's/.*root=\([^ ]*\).*/\1/'`
ROOT_SIZE=0
SIZE_DETECTED=0
if [ ! -z "$ROOT_PART" ]
then
  ROOT_SIZE=`sfdisk -s $ROOT_PART`
fi
if [ $ROOT_SIZE -gt 102400 ]
then
  DEFAULT_IMAGE_SIZE=$(expr $ROOT_SIZE / 1024 + 65)
  SIZE_DETECTED=1
fi

echo
echo "Specify the size of the image file in MB. Defaults to $DEFAULT_IMAGE_SIZE MB, which"
if [ $SIZE_DETECTED -eq 1 ]
then
  echo "has been calculated from the actual size of your root partition."
else
  echo "is the original RasPBX image file size (keep the default if you did not"
  echo "resize your root partition)."
fi
echo -n "Size in MB [$DEFAULT_IMAGE_SIZE]: "
read IMAGE_SIZE
if [ -z "$IMAGE_SIZE" ]
then
  IMAGE_SIZE=$DEFAULT_IMAGE_SIZE
fi

echo
create_image
echo
echo "Running backup now..."
do_backup
echo "Finished."

exit 0

vlad · February 4, 2016

i started working sometime ago on a backup script for my armbian, it is not complete and it was built for my case but it give a starting point for anyone interested in making a backup.

some decisions i took before writing it:

decided to go with rsync + archive and not rsync + image, going with archive instead of image makes restoring the backup a bit harder but it doesn't stop you from using a different format for your partition or a different sd card size same .
archives are uploaded to a ftp server, i chose ftp and not ssh because the ftp service is provided by my ISP so super fast speeds and it cost almost nothing.
i have the rootfs on a mechanical hdd and the uboot/kernel on sdcard so i am making two archives boot and rootfs (but i could be easily changed to backup the whole thing)
the backup its suppose to run every midnight in a crontab so it doesn't output anything directly but its logged in a file, which i am planing to email after a successful backup.

backup.zip

After the backup in order to have a bootable sd card you need to write the u-boot to the sd card your using some information are provided here http://forum.armbian.com/index.php/topic/594-move-armbian-from-one-sd-to-another/- that i am hoping to fix in the future with a restore script.

Tido · February 6, 2016

it is not complete and it was built for my case but it give a starting point for anyone interested in making a backup.

Hi Vlad,

Thank you for your approach.

You explained your basic idea behind it, but do you also have a database or such running which you also want to back up?

In my post I link to a document, the last file I have taken from a Raspberry Pi - this what I thought is interesting:

# Shut down some services before starting backup process
echo "Stopping some services before backup." >> $DIR/backup.log
nginx -s stop
service apache2 stop
service couchdb stop
service mysql stop
service cron stop

tkaiser · February 7, 2016

service mysql stop

OMG. The requirement to backup databases exists since the day databases have been invented. That's what mysqldump is for (similar solutions/tools exist for every other database daemon in this world). And if you're smart then you use mainline kernel, btfs, snapshots and 'btrfs send/receive' to send consistent backups to another machine with minimal efforts.

Tido · February 7, 2016

(similar solutions/tools exist for every other database daemon in this world).

I know.

But I thought about the low power of the RPi. So I thought the person who wrote this script did this with intention, either because of CPU load or to keep the script simple.

Either way it is legitimate to me.

Beside, in OSS everybody can take it and change it to its need.

tkaiser · February 8, 2016

in OSS everybody can take it and change it to its need.

We're talking more about 'SBC world' than 'OSS world' here. And in the SBC world often things are simply the result of 'copy&paste gone wrong' especially when the origin was the Raspberry Pi. That applies not only to scripting stuff but to manuals or 'quick start' guides as well. For example: How moronic is a 'guide' that teaches you to partition/format your SD card just to overwrite partitions/filesystems one second later: http://www.lemaker.org/product-bananapro-guide.html

Using SD Formatter with large SD cards is necessary when you try to use the card with any Raspberry Pi (since there the boot process is totally different and the VideoCore GPU/VPU starts first and is only able to read from FAT and not from exFAT most large cards are formatted with). But this step is totally useless with any other ARM board. Why do we find SD Formatter mentioned in the 'guides' of so many vendors when it's completely useless here?

Think about it. Same applies to script snippets the net is flooded with...

Tido · February 8, 2016

For example: How moronic is a 'guide' that teaches you to partition/format your SD card just to overwrite partitions/filesystems one second later: http://www.lemaker.org/product-bananapro-guide.html

Think about it. Same applies to script snippets the net is flooded with...

I guess you meant the Linux section?

In a terminal, run the sudo fdisk -l command to check the MicroSD card node.

Run the sudo umount /dev/sdxx to unmount all the partitions of the MicroSD card.

Run the sudo fdisk /dev/sdx command. Use the o command to delete all partition of MicroSD card and use the n command to add one new partition. Use the w command to save change.

Run the sudo mkfs.vfat /dev/sdx1 command to format the new created partition of MicroSD card as FAT32.

(x should be replaced according to your MicroSD card node as discovered in point vi above)

You can also jump this step under Linux, because write image command dd under Linux will format the MicroSD card automatically.

The last sentence is badly placed, I agree. So first time you will walk all the steps, next time you jump.

Learning by doing :-)

Using SD Formatter with large SD cards is necessary

I just dd the image of Ubuntu Mate on a 16Gb SDcard (unformated) worked like a charm.

Me as example, I always delete the partition-table before I dd

Part III:

Message understood and accepted.

And still, if the person who wrote that script shuts down the DB because of the little capabilities of the RPi or just because of 'doesn't know it any better', it is both fine for me for this scenario -> Backup.

tkaiser · February 8, 2016

I guess you meant the Linux section?

It's as moronic to use fdisk in Linux to partition a device where you'll overwrite the partition table in the next step as it is when running Windows and doing the same useless step with SD Formatter.

This was meant as an example for a lousy 'guide' that was 'ported' from Raspberry Pi to Banana Pi. 'Copy&Paste gone wrong'

Same applies to backup adoptions that were made for RPi. Unless you try to understand how different any RPi boots compared to the way the boards Armbian supports do, you won't be able to use such backup scripts for Armbian.

It's a known fact that when using some kernel versions necessary hardware initialisation already happens early in u-boot. So if you don't take care of these first sectors of your SD card or NAND flash then desaster recovery will get troublesome.

This is the only real issue you should have in mind when speaking about Armbian + Backup: Take care of the bootloader otherwise you're lost when it's about desaster recovery. Everything else is not SBC specific but applies to Linux in general: And maybe the most important thing is that you can't create reliable/consistent backups of a running 'system as a whole' without snapshots (being it LVM based or using btrfs): Simply try it out using a Wheezy image, start 'apt-get dist-upgrade' and start either dd or rsync. With the dd approach you will end up with a corrupted filesystem and with rsync with inconsistent software.

The most interesting question regarding backup is why anyone wants to do it. The most common reasons

Feeling safe ("hey, at least I thought about implementing some sort of random data duplication that might be called backup")
Being able to recover partially/maybe from data loss/corruption -- under good weather conditions only
Defining time frames for both data loss deltas (the time between two backup runs) and restore durations, weighing the importance of data (base system, configuration, user data) and implementing a backup strategy where data loss doesn't hurt that much at reasonable costs
Desaster recovery (the ability to recover from a loss of the whole system or an important part like the SD card)

Most users care about 1) and maybe 2). So any implementation (even being just an ugly scripting hack with hardcoded paths that will fail from time to time) will do.

If you start to think about 3) and 4) it gets both interesting and time consuming. The quality of a specific backup implementation is defined only through the ability to restore the amount of defined data in the defined time frame. That means implementing, then testing, then fixing, then testing and so on... no one likes that and simply stays with a running script that does $something and trust in it.

vlad · February 9, 2016

Hi Vlad,

Thank you for your approach.

You explained your basic idea behind it, but do you also have a database or such running which you also want to back up?

In my post I link to a document, the last file I have taken from a Raspberry Pi - this what I thought is interesting:
# Shut down some services before starting backup process
echo "Stopping some services before backup." >> $DIR/backup.log
nginx -s stop
service apache2 stop
service couchdb stop
service mysql stop
service cron stop

I don't run a mysql server i also do not backup any data (cept what is in the home dir) - the backup was only meant for the OS - but for mysql backup i think you should take another route for backup as tkaiser suggested mysqldump would be the way to go.

also i see no reason to stop nginx or apache2 for that mater since they are simple web servers for serving files

Tido · February 9, 2016

Same applies to backup adoptions that were made for RPi.

I don't know what you guys are talking about but I want primarily save my data.

This can be a database, pictures, documents..

So, I can look at the script from RPi, copy the good ideas and adjust it to my needs.

Once the system is running I can ONCE make a copy of the OS. Done. No need for a script.

Vlad says: I want to backup my armbian - fine, why not.

In all my comments I never said anything about the OS - I don't see the reason to do that regularly.

As an R1 owner more important is to me to copy the OS between HDD and SDcard.

boobypi · April 27, 2016

Hi guy,

Once time i had datacorruption on my scard so i investigated a way to recovery entire sdcard or make an fsck at startup.
Problem,,,root partition is mountedand in use!

So you can edit fstab to put root fs in readonly mode, make an fsck and a dd like you want.

Inconvenient : very long process and server is down.
Avantage : you can reload original armbian OS into a new sdcard and after recovery your dd image directly on the OPPC (because I have some trouble with big 8GB dd transfert with sdcard usb adapter) from an external flash or hdd.

Here is my note to doing that (in french sorry) :

faire un dd de la sd:

1. editer fstab en ro
remplacer defaults par ro,suid,dev,exec,auto,nouser,async

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda7 during installation
UUID=33b870b8-a81e-4203-a4fd-7affa9f412fb    /               ext4    errors=remount-ro 0       1
# /boot was on /dev/sda5 during installation
UUID=c3cc32c0-b4bd-49f6-b23c-35fed37adea5    /boot           ext2    defaults        0       2
# /home was on /dev/sda8 during installation
UUID=c2d386a1-c2f9-4d2f-957a-65a5d9b4c4d7    /home           ext4    defaults        0       2

2. mettre pass Ã  1 pour un fsck au demarage

3. faire le dd vers un cle usb

4. restaurer fstab comme avant en remontant en rw :
mount -o remount,rw /dev/sda2
puis edition fstab

5. reinstaller lightdm

6. rebooter

Sign In

Creating a backup SD card (* .img) to another media or network

Recommended Posts

electric

Igor

ilu

Wolf2000

Tido

hardvk0

vlad

Tido

tkaiser

Tido

tkaiser

Tido

tkaiser

vlad

Tido

boobypi

Forums

My Activity Streams

Download

Store

Important Information