bedna Posted July 28, 2023 Posted July 28, 2023 (edited) Updated to version 1.3. Please see Shrink-backup github for full information. Release notes: New options: -q|--quiet mode. Mute rsync output --no-color mode. No colorization in script --rsync. User will get presented with the rsync line that will be ran and can modify it --chroot option to access an image in a chroot environment (systemd-nspawn) to for example update/install packages or rebuild initramfs. Only works from the system you created the img Autoexpansion for operating systems: Kali-arm Ubuntu-server-arm (Ubuntu autoexpands by default, but that can be disabled with -e option) UI tweaks Removed check for .img extension while using --loop so it works on other img formats (for example .iso) Changed the dd line to use conv=fsync instead of sync and added a sync operation after dd is done to try to mitigate the "can not set loop paths" problem that some users experience User confirmation now require to also press enter after y/n Added /snap/* in exclude.txt and to be excluded as default if not using the exclude file Implemented a lock file that makes it impossible for boot partition to become unmounted during shrink-backup Nested btrfs subvolumes can now be added & removed in an update (-U) New way of excluding subvolumes for btrfs (exclude_btrfs.txt or shrink-backup_btrfs.conf depending on how you installed, see README for information) --fix now also adds option --fsync to rsync If you have made changes to exclude.txt and you used git to install, git pull will fail due to conflicts. The easiest way to resolve this is to make notes of the paths you added, then: git reset --hard git pull Then add your edited paths back into exclude.txt at the end of the file again. -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Short description: shrink-backup is a bash script for backing up your SBC:s into an img file Version 1.3 Info updated: 2 Aug, 2025. shrink-backup I made this script because I wanted a universal method of backing up my SBC:s into img files as fast as possible (with rsync), no matter what os is in use. shrink-backup is a very fast utility for backing up your SBC:s into minimal bootable img files for easy restore with autoexpansion at boot. Can backup any device with or without a boot partition as long as the filesystem is ext4, f2fs or btrfs (with subvolumes). Supports backing up root & boot (if existing) partitions. Data from other partitions will be written to root if not excluded. For btrfs, all existing top level 5 subvolumes in /etc/fstab will be created with new backups, nested subvolumes will be created and can also be removed/added in an update of the backup img. Please read Info section for more information. Autoexpansion tested & supported on following operating systems: - Raspberry Pi OS (bookworm and older) - Armbian - Manjaro-arm - DietPi - ArchLinux-arm - Kali-arm - Ubuntu-server-arm (Ubuntu autoexpands by default, but that can be disabled with -e option) Autoexpansion does not work on f2fs due to filesystem limitations. Other operating systems will most likely work too, but autoexpansion will not. The script will report the operating system as "unknown" but that does not mean the script will fail. Feel free to make a feature request if you use an operating system not on this list. Full functionality for usage inside webmin (including "custom command" button). Latest release: shrink-backup.v1.3 Testing branch if you want to have the absolute latest version. There might be bugs. Very fast restore thanks to minimal size of img file. Default device that will be backed up is determined by scanning what disk-device root resides on. This means that if boot is a partition, that partition must be on the same device and before the root partition. The script considers everything on the device before root as the bootsector. Backing up/restoring, to/from: usb-stick /dev/sdX with Raspberry pi os has been tested and works. Ie, writing an sd-card img to a usb-stick and vice versa works. Ultra-fast incremental backups to existing img files. See wiki for information about installation methods, usage and examples. Ideas and feedback is always appreciated, whether it's positive or negative. Please just keep it civil. If you find a bug or think something is missing in the script, please file a bug report or feature request To restore a backup, simply "burn" the img file to a device using your favorite method. When booting a restored image with autoresize active, on some operating systems a reboot will occur after resizing is made (you will be informed at the end of the script if your operating system is affected by this), please wait until the the reboot sequence has occurred. The login prompt may very well become visible before the autoresize function has rebooted. -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Usage: shrink-backup -h Script for creating an .img file and subsequently keeing it updated (-U), autoexpansion is enabled by default Directory where .img file is created is automatically excluded in backup ######################################################################## Usage: sudo shrink-backup [-Uatyelhz] [--fix] [--loop] [--f2fs] imagefile.img [extra space (MiB)] ------------------------------------------------------------------------------------------------------------------------------------------------ -U Update existing img file (rsync to existing img) Optional [extra space] extends img size of root partition -a Autoresize root partition, [extra space] is ignored. When used in combination with -U: Expand if partition is >=256MiB smaller than resize2fs recommended minimum Shrink if partition is >=512MiB bigger than resize2fs recommended minimum -t Use exclude.txt in same folder as script to set excluded directories One directory per line: "/dir" or "/dir/*" to only exclude contents -y Disable prompts in script (please use this option with care!) -e Disable autoexpansion of root filesystem when image is booted -l Write debug messages to logfile shrink-backup.log located in same directory as script -z Make script zoom at light-speed, only question prompts might slow it down Can be combined with -y for UNSAFE ultra mega superduper speed -q --quiet Do not print rsync copy process --no-color Run script without color formatted text --fix Try to fix the img file if -a fails with a "broken pipe" error Will activate rsync options --delete-before & --fsync --rsync Define custom rsync line manually. Will print rsync line for user to edit --loop [img] Loop img file and exit, works in combination with -l & -z If optional [extra space] is defined, the img file will be extended with the amount before looping NOTE that only the file gets truncated, no partitions Useful if you for example want to manually manage the partitions --chroot Use systemd-nspawn. Loop img file, mount to temp directory, enter chroot environment and drop to shell This will let you make changes in a chroot environment directly on the img file For example update with package manager or rebuild initramfs The script will keep running in the background --f2fs Convert root filesystem on img from ext4 to f2fs Only works on new img file, not in combination with -U Will make backups of fstab & cmdline.txt to: fstab.shrink-backup.bak & cmdline.txt.shrink-backup.bak Then change ext4 to f2fs in both files and add discard to options on root partition in fstab --version Print version and exit -h --help Show this help snippet ######################################################################## Examples: sudo shrink-backup -a /path/to/backup.img (create img, resize2fs calcualtes size) sudo shrink-backup -e -y /path/to/backup.img 1024 (create img, ignore prompts, do not autoexpand, add 1024MiB extra space) sudo shrink-backup -Utl /path/to/backup.img (update img backup, use exclude.txt and write log to shrink-backup.log) sudo shrink-backup -U /path/to/backup.img 1024 (update img backup, expand img size/root partition with 1024MiB) sudo shrink-backup -Ua /path/to/backup.img (update img backup, resize2fs calculates and resizes img file if needed) sudo shrink-backup -Ua --fix /path/to/backup.img 1024 (update img backup, automatically resizes img file if needed, fix img free space) sudo shrink-backup -l --loop /path/to/backup.img 1024 (write to log file, expand IMG FILE (not partition) by 1024MiB and loop) Thank you for using shrink-backup ❤️❤️ A backup is not really a backup until you have restored from it. Edited August 14 by bedna Updated information for v1.3 1 Quote
bedna Posted July 28, 2023 Author Posted July 28, 2023 (edited) A few changes introduced a bug that made armbian detect 2 partitions instead of one so If you have cloned it since op was made, you have to re clone it, the bug is now fixed. Sorry about that. Edited July 28, 2023 by bedna 0 Quote
bedna Posted August 1, 2023 Author Posted August 1, 2023 Good news everyone! Yesterday a new kernel update (5.15.93-sunxi64 > Linux 6.1.42-sunxi64) was pushed to my armbian on an opiPC2 and u-boot was triggered. Made tests by updating my old image instead of creating a new, then trying to boot from that image with GREAT success! I did not look deeper into what the u-boot actually did, but this time it worked at least. 0 Quote
bedna Posted August 14, 2023 Author Posted August 14, 2023 (edited) Attention, attention, version 0.9.1 released!! come get your free copy! xD A lot has happened: Device being backed up is what root resides on, ie works on any device now, not just mmcblk. No need to specify, script is smart enough to figure it out. ArchLinuxArm autoexpansion now supported without the need for any AUR downloads, only parted is needed. Tested to update with same image through kernel and boot changes on Armbian on orangePi-PC2 and no issues detected. Tested to update with same image through kernel and boot changes on ManjaroArm and no issues detected. Generally a more user friendly interface and the debug function is beyond explanatory. hint hint Hope you find it useful! Edit: Sadly I can no longer edit the original post to update with current information, but the link to github is still valid and all new info is available there. 😄 Edited August 14, 2023 by bedna 0 Quote
bedna Posted January 29, 2024 Author Posted January 29, 2024 Version 0.9.5 released. Support added for OrangePi 5. (gpt partition table) Experimental support for btrfs. For more information, please see https://github.com/UnconnectedBedna/shrink-backup 0 Quote
bedna Posted April 19, 2024 Author Posted April 19, 2024 ==> shrink-backup v1.0 <== I have made the decision to not deal with other partitions than boot and root for the 1.0 release. Instead I introduced the --loop function to let the user expand the img file using the [extra space] option and then manually create partitions by running for example: sudo gparted /dev/loop0 in a terminal to edit partitions in a graphical interface using gparted. I want to give the user freedom, but I also have to stay true to my initial plan with this script: a very fast utility to create a bootable img file from the system and subsequently keep it updated. I haven't dropped the idea of at least handling /home completely, but the script goes from "kinda basic functionality" to "advanced script" pretty fast when I start working on the feature. If I do this, I still want the script to be as easy as possible to use, but at the same time give power users the ability to fine tune, ie a lot of work. Features in the release: Introduction of --loop, --fix & -z (zoom speed) Now crosschecks fstab with lsblk for certain operations. Changed MB to MiB etc. Old habits die hard. Will now, if needed, check and/or ask for installing gdisk on debian and arch based systems. GPT partition table now supported Various bug fixes. I hope you find it useful! 0 Quote
bedna Posted June 26, 2024 Author Posted June 26, 2024 ==> shrink-backup 1.1 release <== With this release versioning is changed from x.x.x to x.x The most noticeable change is the UI with coloring. But small efficiency increases to the code has also been made. Support for dietPi and webmin. Also created a way to convert your systems ext4 filesystem into f2fs on the img file. Downside is f2fs can not be resized while mounted unlike with ext4 so the user have to manually expand the img to cover the entire storage medium manually before booting. Increasing size while updating the img is also not yet covered, but should be doable so this feature will be implemented in a future release. A loop function to retry 3 times after looping the img file within the script has been implemented because bug reports started coming in about the UUID on the loop not being found, therefore failing the backup. Giving the system some time seems to resolve the issue. This seems to be related to if img file is located on a network storage. Usually, but not always, wifi network. Features in the release: UI improvements in form of coloring and other formatting New funcionality: --f2fs convert ext4 on root into f2fs on img file Added support for f2fs Added support for DietPi Added support for webmin --version option added Added .gitignore to github repo for users that change exclude.txt and want to use git pull without issues. Thank you for reading. 0 Quote
basshunger Posted September 25, 2024 Posted September 25, 2024 wow nice read! thanks for your work! gonna test that out now on my x86 laptop running Armbian 24.8.3 0 Quote
robertoj Posted October 7, 2024 Posted October 7, 2024 Thank you for your work The concept sounds super useful to me... I will try it 0 Quote
robertoj Posted October 18, 2024 Posted October 18, 2024 I tried it... and I wished that the -d option was still there... I had to download an old version... however, it didn't work with -d /dev/mmcblk0 ... it started scanning my nvme root But I like the mechanics and the interface Now I am going to try it the normal way, with latest version. Update: I tried it from my orange pi zero 3, Armbian Linux 6.6 Spoiler roberto@myopiz3:/media/thumbdrive/shrink-backup$ sudo ./shrink-backup -a my_armbian_backup.img ## Zoom speed NOT requested... ## Scanning filesystem and calculating... ##################################################################################### # A backup will be created at /media/thumbdrive/shrink-backup/my_armbian_backup.img # # ext4 filesystem detected on root # # --------------------------------------------------------------------------------- # # Write to logfile: false # # Zoom speed requested: false # # Autocalculate img root partition size: true # # Autoexpand filesystem at boot: true # # Use exclude.txt: false # # Bootsector size: 3MiB # # Estemated root usage: 2176MiB # # Auto calculated size (root partition): 2578MiB # # Total img size: 2582MiB # ##################################################################################### ## Do you want to continue? [y/n] y ## Creating bootsector... 18431+0 records in 18431+0 records out 9436672 bytes (9.4 MB, 9.0 MiB) copied, 0.395813 s, 23.8 MB/s ## Resizing img file... ## Looping img file... ## Removing root partition... ## Recreating root partition... ## Formatting filesystem... mke2fs 1.47.0 (5-Feb-2023) Discarding device blocks: done Creating filesystem with 660106 4k blocks and 165312 inodes Filesystem UUID: 90b38738-deb4-4021-8044-9eddfda48c58 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done ## Checking img filesystem... armbi_root: 11/165312 files (0.0% non-contiguous), 28708/660106 blocks ## Creating temp directory... ## Mounting img root partition... ## Backing up files... sending incremental file list deleting lost+found/ ./ bin -> usr/bin initrd.img -> boot/initrd.img-6.6.16-current-sunxi64 initrd.img.old -> boot/initrd.img-6.6.16-current-sunxi64 lib -> usr/lib sbin -> usr/sbin vmlinuz -> boot/vmlinuz-6.6.16-current-sunxi64 vmlinuz.old -> boot/vmlinuz-6.6.16-current-sunxi64 boot/ boot/.next 0 0% 0.00kB/s 0:00:00 (xfr#1, ir-chk=1066/1105) boot/Image -> vmlinuz-6.6.16-current-sunxi64 boot/System.map-6.6.16-current-sunxi64 3.56M 6% 23.51MB/s 0:00:00 (xfr#2, ir-chk=1078/1105) boot/armbianEnv.txt 3.56M 6% 23.51MB/s 0:00:00 (xfr#3, ir-chk=1077/1105) boot/armbian_first_run.txt.template 3.56M 6% 23.36MB/s 0:00:00 (xfr#4, ir-chk=1076/1105) boot/boot.bmp 3.79M 7% 22.82MB/s 0:00:00 (xfr#5, ir-chk=1075/1105) ... thousands of lines later ... var/log/sysstat/sar09 1.99G 99% 4.53MB/s 0:06:58 (xfr#88868, to-chk=14/102914) var/log/sysstat/sar10 1.99G 99% 4.53MB/s 0:06:58 (xfr#88869, to-chk=13/102914) var/log/sysstat/sar19 1.99G 99% 4.53MB/s 0:06:58 (xfr#88870, to-chk=12/102914) var/mail/ var/opt/ var/spool/ var/spool/mail -> ../mail var/spool/cron/ var/spool/cron/crontabs/ var/spool/rsyslog/ var/tmp/ var/tmp/systemd-private-3643107540ea46368972fb8ab0e9a9ff-bluetooth.service-x9tTvV/ var/tmp/systemd-private-3643107540ea46368972fb8ab0e9a9ff-bluetooth.service-x9tTvV/tmp/ var/tmp/systemd-private-3643107540ea46368972fb8ab0e9a9ff-chrony.service-jC1xd8/ var/tmp/systemd-private-3643107540ea46368972fb8ab0e9a9ff-chrony.service-jC1xd8/tmp/ var/tmp/systemd-private-3643107540ea46368972fb8ab0e9a9ff-systemd-logind.service-CHH53d/ var/tmp/systemd-private-3643107540ea46368972fb8ab0e9a9ff-systemd-logind.service-CHH53d/tmp/ var/tmp/systemd-private-3643107540ea46368972fb8ab0e9a9ff-systemd-resolved.service-lnPMrV/ var/tmp/systemd-private-3643107540ea46368972fb8ab0e9a9ff-systemd-resolved.service-lnPMrV/tmp/ 1.99G 99% 4.53MB/s 0:06:59 (xfr#88870, to-chk=0/102914) Number of files: 102,914 (reg: 88,876, dir: 10,303, link: 3,735) Number of created files: 102,913 (reg: 88,876, dir: 10,302, link: 3,735) Number of deleted files: 1 (dir: 1) Number of regular files transferred: 88,870 Total file size: 2.00G bytes Total transferred file size: 1.99G bytes Literal data: 1.99G bytes Matched data: 0 bytes File list size: 2.36M File list generation time: 0.003 seconds File list transfer time: 0.000 seconds Total bytes sent: 2.00G Total bytes received: 1.76M sent 2.00G bytes received 1.76M bytes 4.76M bytes/sec total size is 2.00G speedup is 1.00 ## Rsync done... ## Please stand by... ## Finalizing filesystem... armbi_root: Inode 100609 extent tree (at level 1) could be shorter. IGNORED. armbi_root: /lost+found not found. CREATED. 102918 inodes used (62.26%, out of 165312) 88 non-contiguous files (0.1%) 30 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 99269/4 585744 blocks used (88.73%, out of 660106) 0 bad blocks 1 large file 88870 regular files 10303 directories 0 character device files 0 block device files 0 fifos 6 links 3735 symbolic links (3636 fast symbolic links) 0 sockets ------------ 102914 files ## Remounting for autoexpansion... ## Mounting img root partition... ## Enabling fs-autoexpand... ## Armbian OS detected... ## Armbian filesystem autoresizing at boot... ## Backup done. ##################################################################################### ## Write to logfile: false # ## Autoexpand filesystem at boot: true # ## Use exclude.txt: false # ## /media/thumbdrive/shrink-backup/my_amrbian_backup.img is 2582MiB with a root partition of 2578MiB # ## Please wait for the system to reboot after restoring an image with autoexpansion # ##################################################################################### ## Exiting and cleaning up... ## Please stand by... ## Done. ## Elapsed time: 08.04 roberto@my-opiz3:/media/thumbdrive/shrink-backup$ ls -l total 2325160 -rw-r--r-- 1 roberto roberto 83 Aug 1 2023 exclude.txt -rw-r--r-- 1 roberto roberto 1539 Jun 25 15:20 LICENSE -rw-r--r-- 1 root root 2707992064 Oct 18 18:52 my_armbian_backup.img -rw-r--r-- 1 roberto roberto 18192 Jun 25 15:00 README.md -rwxr-xr-x 1 roberto roberto 99071 Jun 25 14:58 shrink-backup roberto@my-opiz3:/media/thumbdrive/shrink-backup$ Now I have a 2.7GB IMG file in a USB drive which I bring to my Linux Mint laptop Balena etcher has just finished flashing, and is estimating 30 minutes to validate 2.7 GB completed validating quickly too (I was using the laptop sd writer and that is not as reliable as my very cheap USB microSD writer). The Armbian Image booted just fine in a newer Orange Pi Zero 3. THANK YOU FOR MAKING THIS TOOL. I WILL USE IT MANY TIMES 0 Quote
bedna Posted October 20, 2024 Author Posted October 20, 2024 I am not sure why I do not get email notifications when posts are made here, I thought I fixed that... (lmao, I just realized I had for some reason unclicked "follow topic") Thank you so much, both of you for your feedback! ------------------------------------------------------------------------------------------------------- shrink-backup v1.2 released As per usual, I can not change OP so for full and updated information please visit https://github.com/UnconnectedBedna/shrink-backup A smaller release this time. Added curl install method for users that prefer to have application and configs files in a static location, please see wiki Various QOL updates regarding the interface Bug fixes Thank you for using shrink-backup! 0 Quote
Werner Posted October 20, 2024 Posted October 20, 2024 1 hour ago, bedna said: As per usual, I can not change OP Try again 0 Quote
bedna Posted October 20, 2024 Author Posted October 20, 2024 2 hours ago, Werner said: 3 hours ago, bedna said: Try again Thank you! Op now updated to reflect current version. 0 Quote
Igor Posted October 21, 2024 Posted October 21, 2024 22 hours ago, bedna said: Op now updated to reflect current version. Welcome to try with integration to https://github.com/armbian/configng We didn't set rules how to include tools from external repo yet, so this is a bit wild ... - there are unit tests in place for each command - automatic documentation generation is WIP 0 Quote
bedna Posted October 22, 2024 Author Posted October 22, 2024 (edited) Yes, I am working on that. Whiptail is what I am educating me in closer right now. But it will most likely be it's own script. Trying to implement the script straight up I think would become messy pretty fast, and I don't think anybody is interested in that. But yes, it is happening, sometime.. xD I was honestly hoping it would already be a reality, but my health keeps putting up roadblocks for me. Edited October 22, 2024 by bedna 1 Quote
Meestor_X Posted January 2 Posted January 2 Is there a way I can use this in the same way that armbian-install works? I want to automate the copying a complete image from an SD card to the eMMC, exactly the way armbian-install does, but via commandline so I can automate it. Using your tool, I tried creating an image file and then dd-ing it to the eMMC, but it wouldn't boot. I assume there's some fstab stuff that would need to be done when sending the image to another disk? 0 Quote
bedna Posted January 2 Author Posted January 2 Found the login credentials, lets continue on the github where we started. Might as well make the disclaimer here too. The "original" Bedna has passed, I as an extremely close relative have taken over. (Bedna actually came from "Big Edna" a character in the Weird Al movie UHF and we kinda shared that username, twins you know) All credentials for his accounts has been passed on to me so his spirit will live on in shrink-backup. There are TONS of documentation left behind. Please do not make a thing about it. I just wanted to let ppl know that this is still in good hands and will continue to be supported if bugs are found and if changes are needed to be made. But as for getting it into armbian as armbian-backup will have to take a step back. He started with things, but if I do that, it is better to start from scratch again, and my time is a bit limited... 0 Quote
DeadlineX Posted January 13 Posted January 13 (edited) Hi, I've found a problem. I've installed V2Ray, and assigned its logs to /var/log.hdd/v2ray/. This sub-directory is missing in the image file, so V2Ray hangs on startup. Is there any way to add this (or any other) sub-directory to the image file explicitly? I only see explicit exceptions file. Or should I customize the script myself? Thank you. P.S. Further investigations show, that /var/log.hdd/v2ray exists on the written SD (when root is ~1,8 GB) yet before SD insertion into the slot and reboot. But after image shrinking this sub-directory is absent. Moreover, /var/log/journal (which is a link to /var/log.hdd/journal sub-directory), also is absent in the shrunk partition. I used -az option for image creation. -a option leads to the same result. Edited January 13 by DeadlineX 0 Quote
bedna Posted January 13 Author Posted January 13 (edited) Please create an issue on the github providing the logfile and describe your workflow in detail. Make sure the script is up to date, because I think in the past, hdd.log WAS excluded by default. Also provide an example run and show the file is missing by creating a fresh backup, use the `--loop` function of the script and then mount the loop and run ls to show the file is missing. Even if the directory is in ram or similar, and not excluded, it should be in the backup img. Edit. Oh, I see the claim is the file is missing AFTER the autoexpansion, well.. Please still do the original request so there are logs to study. (run the script and include the -l option) If it actually gets removed in the expansion, we have to look deeper into it, because the autoexpansion is made by using armbians systemd autoexpansion. Edit 2 Could also be interesting to see if the file "survives" if you include the -e option, ie do not autoexpand on boot. Edited January 13 by bedna 0 Quote
DeadlineX Posted January 13 Posted January 13 2 часа назад, bedna сказал: Could also be interesting to see if the file "survives" if you include the -e option, ie do not autoexpand on boot. Oh, That's a thought! I didn't try such option. Will do it. What I have now: I've created both /var/log/v2ray and /var/log.hdd/v2ray sub-directories. As a result, both of them survive shrinking. What is most interesting, V2ray starts to write to /var//log/v2ray, though '/var/log.hdd/v2ray" is set in the config, as the log directory. After few seconds (or tens of seconds) it stops to write lod there and starts to write to "log.hdd". I'm puzzled. But of course this is not your script problem, as I believe. As for "journal" missing, please look at the screenshot attached. This is an obvious mistake. In fact we have a link instead of the real directory, and have nothing instead of link. You see the "log.hdd" sub-directory. And once again, everything is OK in the partition before SD card insertion into the boot slot. I.e. we have directory in the "log.hdd" and the link to it in the "log". Also, I'd like to ask, if there is an option do not show all those fast scrolling lines during the process. 0 Quote
bedna Posted January 13 Author Posted January 13 I'm no expert, but I think what you are talking about here is a function to keep the log in memory and write to disk at shutdown or at timed intervals to minimize writes to disk. I would assume it involves some kind of mount --bind at boot, but I honestly don't know. Please create an issue on the github and continue conversation there if you still think this is something the script is responsible for, and by doing so, please follow the instructions in my previous post here on the forum. As for the "keep the script quiet", sure, make a feature request on the ghub and I'll look into it, does not sound to complicated to fix for you tbh. 0 Quote
robertoj Posted February 15 Posted February 15 Today, I used your shrink-backup tool again It works as expected! VERY VERY USEFUL! 0 Quote
Geoffrey Schaller Posted June 30 Posted June 30 Hello, I am running into an issue with a Broken Pipe error on one of my two Orange Pi devices, even with the --fix option. The two Orange Pis (dns1 and dns2) are nearly identical, both are running PiHole. The only difference is dns1 has an extra package (Nebula, and its dependencies) to sync PiHole information over to dns2. When I run shrink-backup on dns2, and back it up to /mnt/backups/PhHole (an NFS mount back to my NAS), it works perfectly. When I run the same command on dns1, it terminates with a Broken Pipe error: root@dns1:/home/orangepi# /root/shrink-backup/shrink-backup --fix /mnt/backups/PiHole/dns1.img (Snip lots of unneeded output) var/lib/snapd/cache/ var/lib/snapd/cache/027a50a18054f65e1f36334c4246c44f5f2f7d037608419191b2c83a58e5afda831a4a1096e878f6b91e5cbb3d49d08b 4.03G 75% 10.82MB/s 0:05:55 (xfr#120880, to-chk=1043/148245) var/lib/snapd/cache/56bd390cd391c0d702651d6489bfa8b61ed4c096b604c14e5d10fd99020e3cdf8fb4467b6f9703aacf0609dcd55b26b0 4.11G 77% 10.92MB/s 0:05:58 (xfr#120881, to-chk=1041/148245) var/lib/snapd/cache/c1c16377ccf539e6292263cd0a579cdf04956110104909a72c492677c30b017542f1f4683a220fd895b68d600e14bceb 4.18G 78% 11.00MB/s 0:06:02 (xfr#120882, to-chk=1037/148245) var/lib/snapd/cache/f60dc4c9c0a8f6f7ddefb04334e5eb74d9bc31305a09911f71deee520569742e40edd137c876b8be67ddfe94afb97266 4.19G 78% 19.53MB/s 0:00:56 rsync: [receiver] write failed on "/tmp/backup-SbW/var/lib/snapd/cache/f60dc4c9c0a8f6f7ddefb04334e5eb74d9bc31305a09911f71deee520569742e40edd137c876b8be67ddfe94afb97266": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(381) [receiver=3.2.7] rsync: [sender] write error: Broken pipe (32) var/lib/snapd/assertions/asserts-v0/snap-revision/wcFjd8z1OeYpImPNClec3wSVYRAQSQmnLEkmd8MLAXVC8fRoOiIP2JW2jWAOFLzr/active 3.96G 74% 10.72MB/s 0:05:52 (xfr#120879, to-chk=1047/148245) var/lib/snapd/assertions/private-keys-v1/ var/lib/snapd/auto-import/ var/lib/snapd/cache/ var/lib/snapd/cache/027a50a18054f65e1f36334c4246c44f5f2f7d037608419191b2c83a58e5afda831a4a1096e878f6b91e5cbb3d49d08b 4.03G 75% 10.82MB/s 0:05:55 (xfr#120880, to-chk=1043/148245) var/lib/snapd/cache/56bd390cd391c0d702651d6489bfa8b61ed4c096b604c14e5d10fd99020e3cdf8fb4467b6f9703aacf0609dcd55b26b0 4.11G 77% 10.92MB/s 0:05:58 (xfr#120881, to-chk=1041/148245) var/lib/snapd/cache/c1c16377ccf539e6292263cd0a579cdf04956110104909a72c492677c30b017542f1f4683a220fd895b68d600e14bceb 4.18G 78% 11.00MB/s 0:06:02 (xfr#120882, to-chk=1037/148245) var/lib/snapd/cache/f60dc4c9c0a8f6f7ddefb04334e5eb74d9bc31305a09911f71deee520569742e40edd137c876b8be67ddfe94afb97266 4.19G 78% 19.53MB/s 0:00:56 rsync: [receiver] write failed on "/tmp/backup-SbW/var/lib/snapd/cache/f60dc4c9c0a8f6f7ddefb04334e5eb74d9bc31305a09911f71deee520569742e40edd137c876b8be67ddfe94afb97266": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(381) [receiver=3.2.7] rsync: [sender] write error: Broken pipe (32) !! RSYNC FAILED!!! !! Cleanup function called with non zero exit code, something went wrong!!! ## Exiting and cleaning up... ## Please stand by... ## Done. ## Elapsed time: 06.56 root@dns1:/home/orangepi# Are you able to help me troubleshoot / diagnose this issue, please? The SD card is 32 GB, there should be plenty of free space, and the NAS itself has plenty of space as well. Thank you! 0 Quote
eselarm Posted June 30 Posted June 30 (edited) @Geoffrey Schaller error is "No space left on device (28)", from rsync The github info mentions: Rsync WILL cross filesystem boundries, and you have Ubuntu snapd active it seems, so my guess is the script copies double; bot snapd image file and mounted content. mount command on your OPI3 will show that. -x option of rsync shall be used, e.g. test it with mkdir -p /mnt/backups/PiHole/test/ rsync -avx / /mnt/backups/PiHole/test/ Edited June 30 by eselarm 0 Quote
bedna Posted June 30 Author Posted June 30 (edited) Please use github for issues: https://github.com/UnconnectedBedna/shrink-backup/issues Don't forget to run shrink-backup with -l option and provide the log with the report. Also please try testing branch first, see below. I want to point out there are a ton of reasons for a broken pipe error with rsync that are presented as "no space left", memory on the machine running for example even thought it sounds strange, short network disconnections is another, so most likely these errors are not related to shrink-backup. There have been some changes made to the rsync operation on the testing branch though, primarily an extended --timeout in case it is network related, so please try that out first before creating an issue on github. cd <directory where you git cloned shrink-backup> # switch to testing branch git checkout testing # run shrink-backup sudo ./shrink-backup <whatever options you use> # if you want to switch back to main git checkout main If you used other method than git to acquire the application you can find the testing branch here: https://github.com/UnconnectedBedna/shrink-backup/tree/testing Edit: Quote and you have Ubuntu snapd active it seems, so my guess is the script copies double; bot snapd image file and mounted content. mount command on your OPI3 will show that. -x option of rsync shall be used, e.g. test it with mkdir -p /mnt/backups/PiHole/test/ rsync -avx / /mnt/backups/PiHole/test/ A solution in a situation like that would be to edit exclude.txt and add paths to what should be excluded and then run shrink-backup with -t option Please see: https://github.com/UnconnectedBedna/shrink-backup/tree/main?tab=readme-ov-file#-t-excludetxt Edited June 30 by bedna 0 Quote
Geoffrey Schaller Posted June 30 Posted June 30 I'll run with the logs now, and open an issue with GitHub. Wasn't sure if this qualifies as a "bug" or not. 0 Quote
bedna Posted June 30 Author Posted June 30 (edited) Well that depends, have you tried adding more space to exclude the possibility that the img actually can fit all the data? If you are using snaps without excluding the containerized mounts on one of them and not the other, it seems like a very logical conclusion that the img actually runs out of space and the "not enough space" is accurate in this situation. Edit: I have never used snaps so I don't know, but I think snaps create containerized mounts every time you boot, and you probably don't want that data on your backup anyway since it will never be used. Others can surely put more light into this than my wild guess about this. Edit 2: This made me a bit curious about how snaps actually work, if they were to randomize the mountpoints and such, but after a tiny bit of searching I found this: https://www.howtogeek.com/660193/how-to-work-with-snap-packages-on-linux/#installing-snap-packages Quote Each file system is mounted on a directory within the /snap directory. So just editing exclude.txt and adding /snap/* and then add -t option when running shrink-backup should solve this issue. As a matter of fact, I should probably add that as default in the exclude.txt file on the repo and in the script if not using the exclude file, because I don't think you ever want that data on the image... Feedback about this would be highly appreciated before I implement that change, does snap create the directory for each SquashFS within /snap if it does not exist or will it break if it is excluded like I suggest above? Edit 3: According to this each SquashFS is mounted directly to /snap so excluding /snap/* (keeping the actual /snap directory but excluding all content) should work right? Quote The snap file is stored in /var/lib/snaps/snaps. It is then loop mounted to /snap from where its contents become readable by the system. Feedback please, I know there are some of you who know how this work, @eselarm maybe? Edited June 30 by bedna 0 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.