shrink-backup is a bash script for backing up your SBC:s into an img file
Version 1.2
Info updated: 20 Oct, 2024.
shrink-backup
I made this script because I wanted a universal method of backing up my SBC:s into img files as fast as possible (with rsync), no matter what os is in use.
Supports backing up root & boot (if existing) partitions. Data from other partitions will be written to root if not excluded (exception for btrfs, all existing subvolumes in /etc/fstab will be created).
Please see Info section.
Autoexpansion tested on Raspberry Pi os (bookworm and older), Armbian, Manjaro-arm, DietPi and ArchLinuxARM for rpi with ext4 or f2fs root partition.
(Now also experimental btrfs functionality, please read further down)
Full support for usage inside webmin (including "custom command" button).
Latest release: shrink-backup.v1.2
Testing branch if you want to have the absolute latest version. There might be bugs.
Very fast restore thanks to minimal size of img file.
Can back up any device as long as root is ext4 or f2fs (experimental btrfs)
Default device that will be backed up is determined by scanning what disk-device root resides on.
This means that if boot is a partition, that partition must be on the same device and before the root partition.
Backing up/restoring, to/from: usb-stick /dev/sdX with Raspberry pi os has been tested and works. Ie, writing an sd-card img to a usb-stick and vice versa works.
Ultra-fast incremental backups to existing img files.
See wiki for information about installation methods, usage and examples.
Ideas and feedback is always appreciated, whether it's positive or negative. Please just keep it civil.
If you find a bug or think something is missing in the script, please file a Bug report or Feature request
Don't forget to ensure the script is executable.
To restore a backup, simply "burn" the img file to a device using your favorite method.
When booting up a restored image with autoresize active, wait until the the reboot sequence has occured. The login prompt may very well become visible before the autoresize function has rebooted.
Usage:
shrink-backup -h
Script for creating an .img file and subsequently keeing it updated (-U), autoexpansion is enabled by default
Directory where .img file is created is automatically excluded in backup
########################################################################
Usage: sudo shrink-backup [-Uatyelhz] [--fix] [--loop] [--f2fs] imagefile.img [extra space (MiB)]
-U Update existing img file (rsync to existing img)
Optional [extra space] extends img size of root partition
-a Autoresize root partition (extra space is ignored)
When used in combination with -U:
Expand if partition is >=256MiB smaller than resize2fs recommended minimum
Shrink if partition is >=512MiB bigger than resize2fs recommended minimum
-t Use exclude.txt in same folder as script to set excluded directories
One directory per line: "/dir" or "/dir/*" to only exclude contents
-y Disable prompts in script (please use this option with care!)
-e DO NOT expand filesystem when image is booted
-l Write debug messages to logfile shrink-backup.log located in same directory as script
-z Make script zoom at light-speed, only question prompts might slow it down
Can be combined with -y for UNSAFE ultra mega superduper speed
--fix Try to fix the img file if -a fails with a "broken pipe" error
--loop [img] Loop img file and exit, works in combination with -l & -z
If optional [extra space] is defined, the img file will be extended with the amount before looping
NOTE that only the file gets truncated, no partitions
Useful if you for example want to manually manage the partitions
--f2fs Convert root filesystem on img from ext4 to f2fs
Only works on new img file, not in combination with -U
Will make backups of fstab & cmdline.txt to: fstab.shrink-backup.bak & cmdline.txt.shrink-backup.bak
Then change ext4 to f2fs in both files and add discard to options on root partition in fstab
--version Print version and exit
-h --help Show this help snippet
########################################################################
Examples:
sudo shrink-backup -a /path/to/backup.img (create img, resize2fs calcualtes size)
sudo shrink-backup -e -y /path/to/backup.img 1024 (create img, ignore prompts, do not autoexpand, add 1024MiB extra space)
sudo shrink-backup -Utl /path/to/backup.img (update img backup, use exclude.txt and write log to shrink-backup.log)
sudo shrink-backup -U /path/to/backup.img 1024 (update img backup, expand img size/root partition with 1024MiB)
sudo shrink-backup -Ua /path/to/backup.img (update img backup, resize2fs calculates and resizes img file if needed)
sudo shrink-backup -Ua --fix /path/to/backup.img 1024 (update img backup, automatically resizes img file if needed, fix img free space)
sudo shrink-backup -l --loop /path/to/backup.img 1024 (write to log file, expand IMG FILE (not partition) by 1024MiB and loop)
-t (exclude.txt)
The folder where the img file is created will ALWAYS be excluded in the backup.
If -t option is selected, exclude.txt MUST exist (but can be empty) within the directory where the script is located or the script will exit with an error.
Use one directory per line in exclude.txt.
/directory/* = create directory but exclude content.
/directory = exclude the directory completely.
If -t is NOT selected the following folders will be excluded:
/lost+found
/proc/*
/sys/*
/dev/*
/tmp/*
/run/*
/mnt/*
/media/*
/var/swap
Please see info section further down.
-l (Log file)
Use -l to write debug info into shrink-backup.log file in the same directory as the script.
-z (Zoom speed)
The -z "zoom" option simply removes the one second sleep at each prompt to give the user time to read.
By using the option, you save 15-25s when running the script.
When used in combination with -y warnings will also be bypassed! PLEASE use with care!
--fix (Broken pipe)
Add --fix to your options if a backup fails during rsync with a "broken pipe" error. You can also manually add [extra space] instead of using -a to solve this.
Example:
sudo shrink-backup -Ua --fix /path/to/backup.img
The reason it happens is because rsync normally deletes files during the backup, not creating a file-list > removing files from img before starting to copy.
So if you have removed and added new data on the system you backup from, there is a risk rsync tries to copy the new data before deleting data from the img, hence completely filling the img.
Using --fix makes rsync create a file-list and delete data before starting to transfer new data. This also means the backup takes a little longer.
Having a "broken pipe" error during backup has in my experience never broken an img backup after either using --fix (can be used in combination with -a) or adding [extra space] while updating the backup with -U.
--loop (Loop img file)
Use --loop to loop an img file to your /dev.
Example:
sudo shrink-backup --loop /path/to/backup.img
If used in combination with [extra space] the amount in MiB will be added to the IMG FILE NOT any partition.
With this you can run for example run: sudo gparted /dev/loop0 (if you have a graphical interface) to manually manage the img partitions in a graphical interface with gparted.
If you added [extra space] this will then show up as unpartitioned space at the end of the device where you can create partition(s) and manually copy data to by mounting the new loop partition that will become visible in lsblk.
If you do this, don't forget to create or update the img with -e (disable autoexpansion) first. Autoexpansion will not work since the space will be occupied by your manually managed partition.
Example:
sudo shrink-backup --loop /path/to/backup.img 1024
This functionality works on any linux system, just use the script on any img file anywhere available to the computer.
To remove the loop: sudo losetup -d /dev/loop0, change loop0 to the correct dev it got looped to.
To remind yourself: lsblk /dev/loop* (if you forgot the location after using --mount)
--f2fs (Convert ext4 into f2fs on img file)
ONLY use this for CONVERTING filesystem into img file, if you already have f2fs on your root, do not use this option.
The script will detect what filesystem is used on root and act accordingly.
Only supported with new backups, not when using -U.
Autoexpansion at boot is not supported for f2fs (there is no way of resizing a mounted f2fs filesystem, unlike with ext4) so resizing root partition have to be made manually after writing img to sd-card.
Resize operations (when updating backup with -U) is not available for f2fs as of now.
The script will make backups of fstab & cmdline.txt into fstab.shrink-backup.bak & cmdline.txt.shrink-backup.bak on the img.
It will then change from ext4 to f2fs in fstab & cmdline.txt and add discard to the options on the root partition in fstab.
Please read information about f2fs further down.
Info
Rsync WILL cross filesystem boundries, so make sure you exclude external drives unless you want them included in the backup. (separate /home for example)
The script will ONLY create boot (if exits) and root partitions on the img file.
The script will ONLY look at your root partition when calculating sizes.
Not excluding other partitions will copy the data to the img root partition, not create more partitions so make sure to manually add [extra space] if you do this.
Experimental btrfs is an exception to this, all subvolumes will be created.
Applications used in the script:
fdisk
sfdisk
dd
parted
e2fsck
truncate
mkfs.ext4
rsync
gdisk (sgdisk is needed if the partition table is GPT, the script will inform you)
To create a backup img using recomended size, use the -a option and the path to the img file.
Example:
sudo shrink-backup -a /path/to/backup.img
Theoretically the script should work on any device as long as root filesystem is ext4, f2fs or experimental btrfs.
Since the script uses lsblk to crosscheck with /etc/fstab to figure out where the root resides it does not matter what device it is on.
Even if you forget to disable autoexpansion on a non supported system, the backup will not fail.
Order of operations - image creation
Reads the block sizes of the partitions
Uses dd to create the boot part of the system + a few megabytes to include the filesystem on root (this can be a partition)
Removes and recreates the root partition, size depends on options used when starting the script
Creates a new ext4 filesystem with the same UUID and LABEL as the system you are backing up from
Uses rsync to sync both partitions (if more than one)
Uses lsblk & /etc/fstab to figure out the correct disk device to back up.
Reads the block sizes of the system's root (and boot if it exists) partition.
Uses dd to create the boot part of the system + a few megabytes to include the filesystem on root. (this can be a partition)
Uses df & resize2fs to calculate sizes by analyzing the system's root partition. (For btrfs: btrfs filesystem du + 192MiB is used instead of resize2fs)
Uses truncate to resize img file.
Loops the img file.
Removes and recreates the root partition on the loop of the img file.
Creates the root filesystem on loop of the img file with the same UUID and LABEL as the system you are backing up from.
Creates a temp directory and mounts img file root partition from loop.
Checks if boot partition exists, if true, checks fstab and creates directory on root and mounts accordingly from loop.
Uses rsync to sync filesystems.
Tries to create autoresize scripts if not disabled with -e.
Unmounts and removes temp directory and file (file created for rsync log output).
Added space is added on top of df reported “used space”, not the size of the partition. Added space is in MiB, so if you want to add 1GB, add 1024.
The script can be instructed to set the img size by requesting recommended minimum size from e2fsck by using the -a option.
This is not the absolute smallest size you can achieve bit is the “safest” way to create a “smallest possible” img file.
If you do not increase the size of the filesystem you are backing up too much, you can most likely keep it updated with the update function (-U) of the script.
By using -a in combination with -U the script will resize the img file if needed (not supported on f2fs).
Using combination -Ua on an img that has become overfilled works, if not add --fix and retry.
Please see --fix and image update sections for more information.
Smallest possible image
To get the absolute smallest img file possible, do NOT set -a option and set “extra space” to 0
Example:
sudo shrink-backup /path/to/backup.img 0
This will instruct the script to get the used space from df and adding 128MB “wiggle room”.
If you are like me, doing a lot of testing, rewriting the sd-card multiple times. The extra time it takes each time will add up pretty fast.
Example:
-rw-r--r-- 1 root root 3.7G Jul 22 21:27 test.img # file created with -a
-rw-r--r-- 1 root root 3.3G Jul 22 22:37 test0.img # file created with 0
Disclaimer:
Because of how filesystems work, df is never a true representation of what will actually fit on a created img file.
Each file, no matter the size, will take up one block of the filesystem, so if you have a LOT of very small files (running docker f.ex) the “0 added space method” might fail during rsync. Increase the 0 a little bit and retry.
This also means you have VERY little free space on the img file after creation.
If the filesystem you back up from increases in size, an update (-U) of the img file might fail.
By using -a in combination with -U the script will resize the img file if needed (not supported on f2fs).
Using combination -Ua on an img that has become overfilled works, if not add --fix and retry.
Please see --fix and Image update sections for more information.
Order or operations - image update
Loops the img file.
Probes the loop of the img file for information about partitions.
If -a is selected, calculates sizes by comparing root sizes on system and img file by using fdisk & resize2fs.
Expands filesystem on img file if needed or if manually added [extra space] is used.
Creates temp directory and mounts root partition from loop.
Checks if boot partition exists, if true, checks fstab and creates directory on root and mounts accordingly from loop.
Uses rsync to sync filesystems.
Shrinks filesystem on img file if -a was used and conditions were met in point 3.
Tries to create autoresize scripts if not disabled with -e.
Unmounts and removes temp directory and file (file created for rsync log output).
Resizing img file when updating
If -a is used in combination with -U, the script will compare the root partition on the img file to the size resize2fs recommends as minimum (or du calculations depending on filesystem).
The img file root partition needs to be >=256MB smaller than resize2fs recommended minimum to be expanded.
The img file root partition needs to be >=512MB bigger than resize2fs recommended minimum to be shrunk.
This is to protect from unessesary resizing operations most likely not needed.
If manually added [extra space] is used in combination with -U, the img file's root partition will be expanded by that amount. No checks are being performed to make sure the data you want to back up will actually fit.
Only expansion is possible with this method.
The script will detect f2fs on root automatically and act accordingly.
Do NOT USE --f2fs unless you are converting from a ext4 filesystem (on your system) into f2fs on the img file.
Autoexpansion at boot is not possible with f2fs. User will have to manually expand img to cover entire storage media (f.ex sd-card) when restoring.
Resizing of img root partition while updating img (-U) is not possible with f2fs as of now. User will have to create a new backup if img runs out of space.
This is something I am planning to implement further down the line.
btrfs
ALL testing has been done on Manjaro-arm
THIS IS NOT A CLONE, IT IS A BACKUP OF REQUIRED FILES FOR A BOOTABLE BTRFS SYSTEM!
All options in the script should work just as on ext4. The script will detect btrfs and act accordingly.
The script will treat snapshots as nested volumes, so make sure to exclude snapshots if you have any, or directories and nested volumes will be created on the img file. This can be done in exclude.txt, wildcards should work.
When starting the script, the initial report window will tell you what volumes will be created. Make sure these are correct before pressing Y.
As of now, top level subvolumes are checked for in /etc/fstab and mounted accordingly, mount options should be preseved (if you for example changed compression).
Autoresize function works on Manjaro-arm.
Yes, I am working on that.
Whiptail is what I am educating me in closer right now.
But it will most likely be it's own script. Trying to implement the script straight up I think would become messy pretty fast, and I don't think anybody is interested in that.
But yes, it is happening, sometime.. xD
I was honestly hoping it would already be a reality, but my health keeps putting up roadblocks for me.
However, there is one mistake. Instead of the last line
~ $ bash pack/usr/lib/u-boot/platform_install.sh pack/usr/lib/linux-u-boot-nanopineo2-current /dev/XXX # replace XXX with the actual device
you need to run
~ $ source pack/usr/lib/u-boot/platform_install.sh
~ $ write_uboot_platform pack/usr/lib/linux-u-boot-nanopineo2-current /dev/XXX # replace XXX with the actual device /dev/sdb
In addition it is important to start the partition at block 8192 (at least for my odroid HC2, may be a differnet number on other devices).
This way you can easily write your backups to an sd.
GPT is something the Armbian build framwork supports, not currently the default, but here is a list of boards that use it:
grep -i gpt *
armsom-sige7.csc:IMAGE_PARTITION_TABLE="gpt"
armsom-w3.csc:IMAGE_PARTITION_TABLE="gpt"
bananapir2pro.csc:IMAGE_PARTITION_TABLE="gpt"
fxblox-rk1.wip:IMAGE_PARTITION_TABLE="gpt"
hinlink-h28k.csc:IMAGE_PARTITION_TABLE="gpt"
hinlink-h88k.csc:IMAGE_PARTITION_TABLE="gpt"
hinlink-ht2.csc:IMAGE_PARTITION_TABLE="gpt"
indiedroid-nova.csc:declare -g IMAGE_PARTITION_TABLE="gpt"
jp-tvbox-3566.tvb:IMAGE_PARTITION_TABLE="gpt"
khadas-edge2.conf:declare -g IMAGE_PARTITION_TABLE="gpt"
mangopi-m28k.csc:IMAGE_PARTITION_TABLE="gpt"
mixtile-blade3.wip:declare -g IMAGE_PARTITION_TABLE="gpt"
nanopct6.wip:IMAGE_PARTITION_TABLE="gpt"
nanopi-r5c.csc:IMAGE_PARTITION_TABLE="gpt"
nanopi-r5s.wip:IMAGE_PARTITION_TABLE="gpt"
nanopi-r6s.conf:IMAGE_PARTITION_TABLE="gpt"
odroidm1.conf:IMAGE_PARTITION_TABLE="gpt"
orangepi3b.csc:IMAGE_PARTITION_TABLE="gpt"
orangepi5.conf:IMAGE_PARTITION_TABLE="gpt"
orangepi5-plus.conf:IMAGE_PARTITION_TABLE="gpt"
panther-x2.csc:IMAGE_PARTITION_TABLE="gpt"
quartz64a.csc:IMAGE_PARTITION_TABLE="gpt"
quartz64b.csc:IMAGE_PARTITION_TABLE="gpt"
radxa-e25.wip:IMAGE_PARTITION_TABLE="gpt"
rock-3a.conf:IMAGE_PARTITION_TABLE="gpt"
rock-5a.wip:IMAGE_PARTITION_TABLE="gpt"
rock-5b.conf:IMAGE_PARTITION_TABLE="gpt"
rock-5-cmio.csc:IMAGE_PARTITION_TABLE="gpt"
station-m2.csc:IMAGE_PARTITION_TABLE="gpt"
station-m3.csc:IMAGE_PARTITION_TABLE="gpt"
station-p2.csc:IMAGE_PARTITION_TABLE="gpt"
xiaomi-elish.conf:IMAGE_PARTITION_TABLE="gpt"
bedna got a reaction from laibsch in shrink-backup - a tool for backing up sbc:s
bedna got a reaction from Igor in shrink-backup - a tool for backing up sbc:s