Remote backup of SD card for an Orange Pi?

[Geoffrey Schaller]

I have a pair of Orange Pi 3's acting as redundant PiHoles on my home network. After a bit of tinkering and configuring, I have them set up the way I want, and would like to keep them that way.

 

Of course, a good backup plan is essential to this.  Looking online, I found Image-Utils for Raspberry Pi, but that is limited in support for only the Raspbian OS.

 

Is there a good system out there to make a cloneable image of an SD card in an Orange Pi, either from the Pi itself to a remote source (NFS mount, etc.), or from a remote source pulling from the Pi?  I'd like to schedule this as a CRON job, so that in the event of a hardware failure, I can just restore the image to a new SD card and deploy it with minimal loss.

 

Thank you!

[Werner]

Maybe this helps? https://forum.armbian.com/topic/29427-shrink-backup-a-tool-for-backing-up-sbcs/

[eselarm]

Most SBC backup tools I have seen are based on rsync and they operate then on/with Ext4 filesystem. Is fine when you don't have complex servers and databases running. I am not sure about pihole, but I saw that by default it keeps a 1 year history so a database file of about 1GB. I am not sure what happens to integrity of such a file on the target if during copy/rsync the source file also changes. AFAIK databases have their ways to handle it, when power-loss or so, but not sure.

If you want to be sure, use Btrfs as filesystem, then you can make atomic snapshots and use those for source of rsync or use:  btrfs send -p <old snapshotnumber> <snapshot number>| ssh <remote_host> btrfs receive <backup folder>. That is the basic manual option. Complete tool is btrbk, it is standard in Debian repo, see https://digint.ch/btrbk/index.html docs of author. You still need some partition and bootloader handling I think. Or assume that a total crash is rare so only backup to NAS or so and reconstruct manually if you need a new SD-card or so or want to copy thing so eMMC of the OPi3.

[bedna]

On 6/28/2025 at 6:58 AM, eselarm said:

Most SBC backup tools I have seen are based on rsync and they operate then on/with Ext4 filesystem. Is fine when you don't have complex servers and databases running. I am not sure about pihole, but I saw that by default it keeps a 1 year history so a database file of about 1GB. I am not sure what happens to integrity of such a file on the target if during copy/rsync the source file also changes. AFAIK databases have their ways to handle it, when power-loss or so, but not sure.

 

I have a pihole on one of my devices that I backup using shrink-backup. When pihole released 6.0 there was a bug that rendered my system unstable so I found it easier to revert back to 5.x until they fixed the bug. The restore worked without any issues whatsoever.

The pihole database is only a db file and the records between the backup and to date is obv lost, but as for integrity of the system it's only a query log so it has no impact on pihole itself. As a matter of fact, you can run pihole without any db history at all. And IIRC the database history is 90 days by default, not one year.

 

On 6/28/2025 at 6:58 AM, eselarm said:

If you want to be sure, use Btrfs as filesystem, then you can make atomic snapshots and use those for source of rsync or use:  btrfs send -p <old snapshotnumber> <snapshot number>| ssh <remote_host> btrfs receive <backup folder>. That is the basic manual option.

 

A btrfs send|recieve is only a file backup, it will never handle boot process or anything like that. You will also need to do it on every subvolume individually since send|recieve only takes one subvol, not even nested volumes. You will also have to make the source subvolume read only before issuing the send, but I guess that is what you mean by saying "make atomic snapshots". It's also very slow compared to rsync.

Besides, using btrfs send|recieve does not solve the issue that copying a database that is in use might corrupt it, shutting down the application using the database before making a backup is. As a matter of fact, it might just make things even more complicated since you have to make the database (subvolume) read only during the send so the application might freak out from not being able to write to the database (unless you snapshot it first like you mention, but then you still have the same problem that an open database might break after being backed up).

btrbk solves this (I think) by creating a snapshot (snapshots become read only by default) from the subvolume first and then send that, but you can't btrfs send|recieve a read/write subvolume, but again, does not solve the issue that the database is open.

 

https://btrfs.readthedocs.io/en/latest/btrfs-send.html

Quote

All snapshots involved in one send command must be read-only, and this status cannot be changed as long as there’s a running send operation that uses the snapshot. Read-only mount of the subvolume is not sufficient, there’s no way to guarantee that there won’t be any other writable mount of the same subvolume that would potentially write while send would be running.

 

Fun fact: shrink-backup can backup btrfs devices.

https://github.com/UnconnectedBedna/shrink-backup?tab=readme-ov-file#btrfs

Edited Thursday at 04:04 PM by bedna

[eselarm]

I just mention the very basic, Btrfs has way to many features and options to make a generic image creating script. I see already 2 issues in the shrink-backup scrip, simple test:

root@rock5b:~ # btrfs filesystem du -s --raw /
     Total   Exclusive  Set shared  Filename

ERROR: not a btrfs filesystem: /boot/efi
WARNING: cannot access 'efi': Invalid argument

WARNING: cannot access 'boot': Invalid argument

ERROR: cannot check space of '/': Invalid argument

This is on Opensuse Tumbleweed, but distro should not matter as I have a custom simple subvolume scheme. It is after 10+ years that I have a fully automated methods that just backup the structure I use in every Linux system so that it also can be 'played back' to a (sparse) image file or just raw device. I don't do any resizing, that is not essential to backup; I just re-create that same partition table (also backed up) so I have the same machine after a failed SD-card or HDD with too many bad sectors or just a clone template for a extra new SBC.

 

I use a tool called snapper to create snapshots, both automatic and ad-hoc. It is also integrated in apt and zypper. So before the upgrade to pihole v6, a (read-only) snapshot is created. Also I was bothered by buggy v6, so I just take a read-write snapshot of the last know good/running read-only snapshot and set that that to default, then reboot, or even sync and power cycle. Of course that snapshot (UUID) is also as a differential created tree/folder on my NAS (in case the SD-card or so would fail).

 

Long time ago installers created a lot of subvolumes, like for /home /var/log and many more. That indeed gets into big problems. Just use 1 for the root and snapshot that. That works fine for a workstation like machine with no real data stored locally (all is on NAS).

 

Note that since kernel 6.1, btrfs send can send compressed data, so as it is stored on storage device. That saves time (and money) on mobile links for example. You can compress and decompress via rsync or ssh or vpn, but it costs extra battery energy. 

 

Also how long does "make atomic snapshots" take on a 4TB HDD of a subvolume of 2TB you think?

 

[bedna]

Thanks for the feedback, If that was the intention. If you try the version on the testing branch, that won't be an issue because 'btrfs du' (it is just slow and unnecessary) is no longer used. xD

There are actually more changes coming to btrfs in the next release. Even looking into conversion from ext4 to btrfs.

 

Oddly enough, I can backup my arch system with boot mounted at /boot/efi using the script on main branch, wonder why yours doesn't, so clearly distro DOES matter. But it soon won't matter for the problem you had, because it will be changed anyway.

Besides, shrink-backup is not really intended to be used on a desktop computer, it's for sbc:s that usually are way less convoluted than a desktop setup.

For ext4 and f2fs shrink-backup only covers root and boot partitions (if boot partition exists), btrfs is an exception where the script looks up existing subvolumes and creates them. As of now, the script also assumes all subvolumes starting with @ is a top volume, witch does not actually have to be the case, but changes are coming.

 

Interesting to see you use /boot/efi (discouraged by linux kernel standards), and not /boot as boot partition, wonder why?

I do the same, because I use timeshift that is built upon snapper. Timeshift also creates entries directly on my grub menu letting me boot into the ro snapshots and restore from there if needed.

For others reading this: This is because if you roll back a snapshot where you have changed things in boot process and files inside /boot, you kinda want files inside /boot to be rolled back too or your system won't boot, hence letting /boot be part of the root subvolume and mount boot partition on /boot/efi instead.

 

My point was, that the problem you mentioned, the database might become corrupted is not solved by using snapshots and btrfs, a database has to be closed and locked to be 100% sure it won't become corrupted in ANY copy process. (beside the fact that the database in the case of pihole is only a log and has nothing to do with the integrity of the application itself)

 

7 hours ago, eselarm said:

Long time ago installers created a lot of subvolumes, like for /home /var/log and many more. That indeed gets into big problems. Just use 1 for the root and snapshot that. That works fine for a workstation like machine with no real data stored locally (all is on NAS).

 

I completely disagree with that. If I revert a snapshot because I messed something up on my system or something went wrong in an update for example, witch happens once in a while, I do NOT want my home to be over-written by an old snapshot. I also want to be able to read old log-files to figure out WHAT went wrong, so /home & /var/log in separate snapshots is absolutely a good thing for me.

On a sbc I might agree that separate snapshots is a bit overkill though, but not in any way a bad idea. But if you actually use snapshots as a restore points on an sbc I argue /var/log is a good thing to have on separate snapshot.

 

7 hours ago, eselarm said:

Also how long does "make atomic snapshots" take on a 4TB HDD of a subvolume of 2TB you think?

 

A snapshot takes no time at all, 1s I would guess, but what about the send|receive?

Because a snapshot is completely worthless if the filesystem you took the snapshot from becomes corrupt, COW (copy on write) means you use the exact same data. That is why it's (or should be) general knowledge that a snapshot is NOT A BACKUP, but it becomes a file-backup if you send|receive it to a completely different filesystem.

So the interesting part here is the send|receive because op asked about backups, and that takes about as long as a dd would take, while rsync probably does the file transfer in about 10% of that time.

Also, again, shrink-backup takes everything, including boot process parts and makes an img file at small size (for two reasons, saving space and ability to use the img file on smaller storage device, moving from a 64GiB to a 32GiB sd-card for example) that can be written directly instead of also having to do a bunch of pretty complicated restore processes. And quite a few armbian images utilizes u-boot without a boot partition with header data written to the disk before root partition, shrink-backup covers that as-well.

 

Generally, I treat snapshots as they are intended, as snapshots that can be used as RESTORE POINTS, and I use a backup application to make regular BACKUPS. On my desktop computer I use clonezilla that I have added a menu for in grub to easily boot into (you can make custom menus and boot directly into img/iso files in grub), and on all my sbc:s I obviously use shrink-backup. (for those who don't know, I am the maintainer of shrink-backup)

I also have 260 "individual" incremental file backups of my home (and some other locations like /etc and my "file disk"), one every 6h using urbackup running on a rpi4 with a 6TB disk connected to it. It utilizes btrfs so the "individual" means they are just snapshots and only changed files actually take up more space.

 

Lastly, if you don't actually utilize the functionality of running a system with multiple subvolumes, you don't even need to have a subvolume for /, you can just write files directly to the btrfs filesystem.

Edited 16 hours ago by bedna

[eselarm]

42 minutes ago, bedna said:

Interesting to see you use /boot/efi (discouraged by linux kernel standards), and not /boot as boot partition, wonder why?

I know, but who cares if it is impractical IMO and various distros by default use that path. And RPiOS needs /boot/firmware, I myself use also /boot/uboot so I am more flexible w.r.t bootloaders. What is on boot FAT or in SPI-flash or tftpboot  or else is secondary, the master (from package manager) is in rootfs.

 

55 minutes ago, bedna said:

A snapshot takes no time at all, 1s I would guess, but what about the send|receive?

Because a snapshot is completely worthless if the filesystem you took the snapshot from becomes corrupt, COW (copy on write) means you use the exact same data. That is why it's (or should be) general knowledge that a snapshot is NOT A BACKUP, but it kinda becomes a backup if you send|receive it to a completely different filesystem.

Read back my 1st message; there is option -p

A snapshot is identified by its UUID, whether local or remote/other/foreign filesystem. So my backup tool/script figures that out. If an SD-card would be end of life or damaged scrub will notice it and then I regenerate the whole thing on a new SD-card from the latest transferred snapshot.

 

I do not use /home for any meaningful data. All is on NAS. Some have no local storage except SPI for bootloader.  I also treat all Linux computers the same, except those that are NAS or a clone of it. For rare Windows I indeed use 'imaging' although there is WinBTRFS. 

[bedna]

8 hours ago, eselarm said:

Read back my 1st message; there is option -p

 

That does not change anything about the FIRST send you do, it will take just as long as a dd does while rsync does it in 10% of that time.

-p does the same thing as an rsync does to files that has already been rsync:d, the difference is rsync only looks at the data in the files while a send|receive also syncs the entire structure of the filesystem, that's the whole point with the COW snapshot method. So if parent is corrupted without you knowing it (witch is very often the case), so will the child.

And as a rule of thumb in backups, you should have multiple backups, the 3-2-1 model is a good guidance.

The downside with snapshots is they inherit the flaws of the parent, that is why it is common knowledge (or should be) that snapshots should NOT be treated as backups but ONLY as restore points.

 

8 hours ago, eselarm said:

If an SD-card would be end of life or damaged scrub will notice it

 

No, a scrub will not notice anything until it's too late. A scrub is just a check of the integrity of the filesystem, so if a scrub notices something is wrong the filesystem is already corrupted.

So unless you also do a scrub EVERY time on the filesystem you are about to send from before sending, the receive system will also become corrupted. A complete scrub of 2TB takes a while, probably a few hours even if on an nvme.

 

8 hours ago, eselarm said:

I do not use /home for any meaningful data

 

We are all different, I have all my userspace data on home, seems odd to me to rely on a network to be able to access my userspace. Would be really annoying if something were to happen to networking during for example an update or if the network interface stopped working on my desktop for some reason, maybe broken hardware in between like a switch or something.

I also do a bit of AI stuff, and some models can be pretty darn big (for example some flux models are around 12GiB) so I want to be able to access them at the speed of an nvme.

Other files like movies and such I keep on a network drive, I just have an NFS share on a rpi with a 4TiB spinning drive connected to it, that works just fine for me. That drive is ofc also backed up to another device with 260 incremental versions using urbackup.

 

I'm not trying to convince you to use anything specific, I am simply trying to explain why relying soly on snapshots is not a great idea.

I also think that having a restore via an img file is way easier than having only file backups in case of disaster, therefore I maintain shrink-backup. If for example my pihole dies, let's say the sd-card becomes completely unusable, I want as little downtime as possible. Writing an img file to another sd-card only takes a few minutes and the system is restored.

Or even worse, if I were to utilize your system relying on a nas. If that nas dies and all my backups are on that nas, what do I do then?

 

So as some friendly advice, I think you should reconsider your backup strategy if you want to be safe in case of complete disaster, you should always plan for worst case scenario.

Here is a pretty good read on why snapshots are not considered a good strategy for backups: https://community.veeam.com/blogs-and-podcasts-57/data-backup-basics-i-why-snapshots-are-not-backups-understanding-the-differences-6074

I also rely on snapshots as restore points, but I would never rely on them alone.

 

As an example, by using shrink-backup on my sbc:s I create an img file, I compress that img file and that file then becomes backed up with urbackup to another device (I actually exclude the img file itself in the backup), so when I update that img with shrink-backup, I rename the last compressed file with an "-old" suffix and create a new compressed file. That way I always have 2 backups accessible directly on my desktop computer. Those files are then again backed up by urbackup and are kept at least 3 months (260 backups is the actual threshold so it depends on how many backups is done per day, but minimum 3 months if my computer were to run 24/7). I also store one of those urbackup backups per month for 3 years at a location outside my house, sent encrypted via the internet to another physical location at a friends house (this could also be a cloud storage). In case of for example a fire, no matter how many backups I have at the house, they will all burn and I loose everything. A disaster only has to occur once, and by then it's too late.

3-2-1 backup strategy.

Edited 6 hours ago by bedna