m11k

Hi,
 
I wanted to convert my root filesystem to btrfs.  This allows me to use snapper to take regular snapshops.  I also use btrbk to send my snapshots to my secondary storage (also a btrfs array).  My root filesystems is installed to eMMC, but there isn't any reason you couldn't do this with an sd card.  These instructions are done with a laptop running Linux.  I'm using Debian in this case, although any distro with a recent kernel and btrfs tools installed should work.
 
CAVEATS:
The current u-boot image (2020.08.21) doesn't support booting natively to btrfs.  u-boot *does* support btrfs though.  Hopefully a future build of u-boot will add btrfs support.  To work around this, we have to create an ext4 /boot partition, and then a btrfs root partition.
 
1. To start, I booted the helios64 via sd card to the USB mass storage image (https://wiki.kobol.io/helios64/install/emmc/).  I then connected a laptop to the helios64 via the USB-C cable, and the emmc appeared as /dev/sdb.  The rest of these instructions will assume the eMMC is /dev/sdb, so please adjust accordingly for your system.
 
2. Mount the eMMC root filesystem on the host:
# mkdir /mnt/nasrootext4 # mount /dev/sdb1 -o ro /mnt/nasrootext4  
3. Back up the contents of the root filesystem:
# tar -C /mnt/nasrootext4 --acls --xattrs -cf root_backup.tar .  
4. Make a backup of the partition layout (in case you decide to revert this modification):
# fdisk -l /dev/sdb | tee nas_fdisk.txt  
5. Unmount the filesystem, and run fdisk on it.  Make note of the start sector.  It should be 32768.
# umount /dev/sdb1 # fdisk /dev/sdb Command (m for help): p Disk /dev/sdb: 14.6 GiB, 15634268160 bytes, 30535680 sectors Disk model: UMS disk 0 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x794023b5 Device Boot Start End Sectors Size Id Type /dev/sdb1 32768 30230303 30197536 14.4G 83 Linux  
6. Delete the existing partition ('d')
7. Create a new partition ('n')
8. Select 'primary' partition type
9. Select partition number 1
10. Select first sector: 32768 (This is very important to get right, and should match the start sector of the partition that was just deleted)
11. Select last sector: +512M (this create a 512MB /boot partition)
12. You will see a message "Partition #1 contains a ext4 signature.".  This is okay.  I selected 'n' (don't remove signature) but it doesn't matter.  You should probably remove the signature since we're going to create a new ext4 filesystem anyway.
13. Create new partition ('n')
14. Select 'primary' partition type
15. Select partition number 2
16. First sector: 1081344 (this is one more than the end sector of the other partition)
17. Last sector: (leave the default as the full drive, 30535679 in my case)
18. Print the layout to confirm:
Disk /dev/sdb: 14.6 GiB, 15634268160 bytes, 30535680 sectors Disk model: UMS disk 0 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x794023b5 Device Boot Start End Sectors Size Id Type /dev/sdb1 32768 1081343 1048576 512M 83 Linux /dev/sdb2 1081344 30535679 29454336 14G 83 Linux  
19. Write the partition table to disk and exit ('w')
20. Create a new ext4 filesystem on the first partition: 'mkfs.ext4 /dev/sdb1' (you may need to add -F if it complains about an existing ext4 signature)
21. Create a new btrfs filesystem on the second partition: 'mkfs.btrfs /dev/sdb2'
22. Mount the new root filesystem:
# mkdir /mnt/nasrootbtrfs # sudo mount -o compress=zstd /dev/sdb2 /mnt/nasrootbtrfs  
23. Make some initial subvolumes (note that '@' will be the root subvolume, as this is preferred by 'snapper'):
# cd /mnt/nasrootbtrfs # btrfs subvol create @ # btrfs subvol create @home # btrfs subvol create @var_log  
24. Mount the boot partition, and the @home and @var_log subvolumes
# cd @ # mkdir boot # mount /dev/sdb1 boot/ # mkdir home # mount /dev/sdb2 -o noatime,compress=zstd,subvol=/@home home # mkdir -p var/log # mount /dev/sdb2 -o noatime,compress=zstd,subvol=/@var_log var/log # mount | grep sdb /dev/sdb2 on /mnt/nasrootbtrfs type btrfs (rw,relatime,compress=zstd,space_cache,subvolid=5,subvol=/) /dev/sdb1 on /mnt/nasrootbtrfs/@/boot type ext4 (rw,relatime) /dev/sdb2 on /mnt/nasrootbtrfs/@/home type btrfs (rw,noatime,compress=zstd,space_cache,subvolid=258,subvol=/@home) /dev/sdb2 on /mnt/nasrootbtrfs/@/var/log type btrfs (rw,noatime,compress=zstd,space_cache,subvolid=260,subvol=/@var_log)  
25. Note that our current directory is '/mnt/nasrootbtrfs/@'
26. Untar the old root filesystem into the new root directory.  Since boot, home, and var/log are mounted, tar will extract the relevant data into these mounted filesystems.
# tar -xvf /root/helios64_root.tar --acls --xattrs --numeric-owner |@ tee /root/extract_log.txt This is the only spot where things did not go as expected.  I received a handful of errors about "No space left on device" while untarring.  These seem to be transient btrfs errors. I use btrfs pretty extensively at work, and I've seen cases in the past on older kernels where btrfs would falsely report ENOSPC when the filesystem was under heavy load.  I'm not sure if that is what is happening here, but I would guess it's a bug.  It could be a bug in the kernel on my laptop, or it could be a bug in the usb-c mass storage export on the helios64.  To recover from this, I ran the tar command a couple of times.  I continued to receive these errors on different files.  I did this three or four times.  I also tried removing --acls and --xattrs, but it didn't seem to make much of a difference.  On my last attempt to untar, all of the files that failed happened to be from usr/lib/python3, so I untarred just that directory: 'tar -xvf /root/helios64_root.tar --numeric-owner ./usr/lib/python3'.  That extracted without error.  I was satisfied that everything made it back on disk.  Perhaps if you only extracted one top-level directory at a time, it would not produce any errors.  Maybe there is some way to slow down the 'tar' to avoid these ENOSPC errors.
 
27. After extracting the root filesystem, I made a snapshot of it: 'btrfs subvol snap -r . root_post_copy'
28. Update /mnt/nasrootbtrfs/@/etc/fstab.  This was my old fstab:
Old version: UUID=dabb3dbf-8631-4051-a032-c0b97eb285bd / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1 tmpfs /tmp tmpfs defaults,nosuid 0 0 # >>> [openmediavault] /dev/disk/by-label/helios64_btrfs_raid1 /srv/dev-disk-by-label-helios64_btrfs_raid1 btrfs defaults,nofail,noatime,nodiratime,compress=zstd 0 2 # <<< [openmediavault]  
Look up the new UUID for /dev/sdb2.  I did this by running 'ls -l /dev/disk/by-uuid').  Updated /etc/fstab:
UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f / btrfs defaults,noatime,compress=zstd,ssd,subvol=@ 0 0 UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /home btrfs defaults,noatime,compress=zstd,ssd,subvol=@home 0 0 UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /var/log/ btrfs defaults,noatime,compress=zstd,ssd,subvol=@var_log 0 0 UUID=fd5b620c-57e2-4031-b56c-3c64ce7c7d5f /boot ext4 defaults,noatime 0 2 tmpfs /tmp tmpfs defaults,nosuid 0 0 # >>> [openmediavault] /dev/disk/by-label/helios64_btrfs_raid1 /srv/dev-disk-by-label-helios64_btrfs_raid1 btrfs defaults,nofail,noatime,nodiratime,compress=zstd 0 2 # <<< [openmediavault]  
29. Modify /mnt/nasrootbtrfs/@/boot/armbianEnv.txt.  In particular modify rootdev, rootfstype, and add 'extraargs'.  Old version:
verbosity=1 bootlogo=false overlay_prefix=rockchip rootdev=UUID=dabb3dbf-8631-4051-a032-c0b97eb285bd rootfstype=ext4 usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u  
New version:
verbosity=1 bootlogo=false overlay_prefix=rockchip rootdev=UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f rootfstype=btrfs usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u extraargs=rootflags=subvol=@  
30. Make a 'boot' symlink.  I'm not sure if this is necessary, but u-boot expects the boot files to be in the '/boot' directory on the root filesystem, but they are now in the root directory of the boot partition.  This symlink makes sure that if u-boot tries to access /boot/foo, it will actually access /foo.
# cd /mnt/nasrootbtrfs/@/boot # ln -s . boot  
31.  Unmount everything:
# umount /mnt/nasrootbtrfs/@/var/log # umount /mnt/nasrootbtrfs/@/home # umount /mnt/nasrootbtrfs/@/boot # umount /mnt/nasrootbtrfs/  
32. Power off helios64
33. Remove sd card
34. Start picocom on your laptop ('picocom -b 1500000 /dev/ttyUSB0')
35. Turn on helios 64 and look for startup.
 
TROUBLESHOOTING:
Of course this did not work on the first attempt for me.  After rebooting the helios64, it never successfully booted into Linux.  I didn't see any errors from u-boot.  To troubleshoot, I wrote the original helios64 debian buster image to sd card and booted the helios64 to that.  I mounted /dev/mmcblk1p1 to /mnt, and changed the 'verbosity' to 7 in armbianEnv.txt.  I eventually discovered that I had the UUID wrong in /boot/armbianEnv.txt.  Correcting that value fixed the issue, and the helios64 booted successfully from eMMC.
 
SETTING UP AUTOMATED SNAPSHOTS AND BACKUPS:
Enabling 'snapper' for automated root snapshots was very easy:
1. sudo apt install snapper
2. sudo snapper -c root create-config /
 
The debian snapper automatically take snapshots before and after apt upgrades.  However since /boot is a separate filesystem, this is not included in these snapshots.  As a workaround, I added an apt hook which rsyncs /boot to /.bootbackup before the snapshots are made.  Note that the 'snapper' hook is /etc/apt/apt.conf.d/80snapper, so I created /etc/apt/apt.conf.d/79-snapper-boot-backup:
# /etc/apt/apt.conf.d/79-snapper-boot-backup DPkg::Pre-Invoke { "if mountpoint -q /boot && [ -e /.bootbackup ] ; then rsync -a --delete /boot /.bootbackup || true ; fi"; }; DPkg::Post-Invoke { "if mountpoint -q /boot && [ -e /.bootbackup ] ; then rsync -a --delete /boot /.bootbackup || true ; fi"; };  
This takes care of automated snapshots.  I still want snapshots stored on another drive (or another system) for redundancy.  I'm using 'btrbk' to do this.  btrbk wants the btrfs top-level subvolume mounted somewhere.  To do so, I created the directory /mnt/btr_pool, and added the following line to /etc/fstab:
UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /mnt/btr_pool/ btrfs defaults,noatime,compress=zstd,ssd,subvolid=5 0 0  
My spinning drive array on the helios64 is a btrfs filesystem, and I have it mounted on /srv/dev-disk-by-label-helios64_btrfs_raid1.  I created a '/srv/.../backups/helios64/btrfs/volumes' directory.  I then created the following three files:
 
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/btrbk.conf:
timestamp_format long volume /mnt/btr_pool snapshot_dir btrbk snapshot_preserve 30d snapshot_preserve_min latest target_preserve 7d 8w 6m target_preserve_min latest target send-receive /srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/volumes subvolume @ subvolume @home subvolume @var_log  
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/rsync_boot.sh:
#!/bin/sh set -e -x mountpoint /boot [ -e /.bootbackup ] rsync -a --delete /boot /.bootbackup  
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/run.sh
#!/bin/bash set -e -x CDIR=$( dirname ${BASH_SOURCE[0]} ) ${CDIR}/rsync_boot.sh btrbk -c ${CDIR}/btrbk.conf run  
I then added a daily cron job which runs the 'run.sh' script.
 
Well, that was a longer post than I had imagined.  I hope someone finds it helpful.

I haven't been able to find any specs on the eMMC module in the helios64.  I've found some posts online indicating that it is a samsung KLMAG1JETD-B041, but I haven't been able to find any info on the write endurance of this module (not sure if it is standard for eMMC vendors to provide that information like SSD vendors).
 
I have been able to run `mmc extcsd read /dev/mmcblk1`, which shows that the emmc life time estimation is between 0 and 10%, but since this value is in 10% increments, its not very helpful.
 
I've had issues with the default ramlog-based /var/log, so I had to turn it off.  I've increased the size to 250MB but it regularly fills up because the 'sysstat', 'pcp' (for cockpit-pcp), and 'atop' packages all write persistent logs into /var/log, which 'armbian-truncate-logs' doesn't clean up.  I could probably update the armbian-truncate-logs script to support these tools if that was the only issue.  However I've also enabled the systemd persistent journal, and although I can see that armbian-truncate-logs is calling journalctl, I still get messages about corrupt journals.  Also `journalctl` only reads logs from /var/log/journal, not /var/log.hdd/journal, so it is only able to show the current day's logs, which isn't very useful.
 
So I'm trying to determine if I should be safe to have /var/log on disk without the ramlog overlay.  Any recommendations?  I do have my root filesystem on btrfs, and /var/log in a separate subvolume.  Maybe there's some way to tweak how frequently to sync /var/log to disk, but I don't know of any mechanism off the top of my head.
 
Thanks,
Mike

I gave this method a try, and it worked successfully without any of the btrfs "no space left on device" errors.  The process was largely the same, except instead of the eMMC device being /dev/sdb, it was /dev/mmcblk1.  I also mounted the storage array and used that to hold the tarball of the old root filesystem.

Sure, I'd be happy to contribute this.  Rewriting it in markdown would be a pleasure, as I found all the clicking required in the forum a little frustrating for a long post.  When I get a few hours free, I'd like to try this process again using the helios itself booted to armbian on sd-card to do the conversion.  That way a second linux system wouldn't be required, and it might fix the random ENOSPC errors I encountered.

Hi,
 
I wanted to convert my root filesystem to btrfs.  This allows me to use snapper to take regular snapshops.  I also use btrbk to send my snapshots to my secondary storage (also a btrfs array).  My root filesystems is installed to eMMC, but there isn't any reason you couldn't do this with an sd card.  These instructions are done with a laptop running Linux.  I'm using Debian in this case, although any distro with a recent kernel and btrfs tools installed should work.
 
CAVEATS:
The current u-boot image (2020.08.21) doesn't support booting natively to btrfs.  u-boot *does* support btrfs though.  Hopefully a future build of u-boot will add btrfs support.  To work around this, we have to create an ext4 /boot partition, and then a btrfs root partition.
 
1. To start, I booted the helios64 via sd card to the USB mass storage image (https://wiki.kobol.io/helios64/install/emmc/).  I then connected a laptop to the helios64 via the USB-C cable, and the emmc appeared as /dev/sdb.  The rest of these instructions will assume the eMMC is /dev/sdb, so please adjust accordingly for your system.
 
2. Mount the eMMC root filesystem on the host:
# mkdir /mnt/nasrootext4 # mount /dev/sdb1 -o ro /mnt/nasrootext4  
3. Back up the contents of the root filesystem:
# tar -C /mnt/nasrootext4 --acls --xattrs -cf root_backup.tar .  
4. Make a backup of the partition layout (in case you decide to revert this modification):
# fdisk -l /dev/sdb | tee nas_fdisk.txt  
5. Unmount the filesystem, and run fdisk on it.  Make note of the start sector.  It should be 32768.
# umount /dev/sdb1 # fdisk /dev/sdb Command (m for help): p Disk /dev/sdb: 14.6 GiB, 15634268160 bytes, 30535680 sectors Disk model: UMS disk 0 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x794023b5 Device Boot Start End Sectors Size Id Type /dev/sdb1 32768 30230303 30197536 14.4G 83 Linux  
6. Delete the existing partition ('d')
7. Create a new partition ('n')
8. Select 'primary' partition type
9. Select partition number 1
10. Select first sector: 32768 (This is very important to get right, and should match the start sector of the partition that was just deleted)
11. Select last sector: +512M (this create a 512MB /boot partition)
12. You will see a message "Partition #1 contains a ext4 signature.".  This is okay.  I selected 'n' (don't remove signature) but it doesn't matter.  You should probably remove the signature since we're going to create a new ext4 filesystem anyway.
13. Create new partition ('n')
14. Select 'primary' partition type
15. Select partition number 2
16. First sector: 1081344 (this is one more than the end sector of the other partition)
17. Last sector: (leave the default as the full drive, 30535679 in my case)
18. Print the layout to confirm:
Disk /dev/sdb: 14.6 GiB, 15634268160 bytes, 30535680 sectors Disk model: UMS disk 0 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x794023b5 Device Boot Start End Sectors Size Id Type /dev/sdb1 32768 1081343 1048576 512M 83 Linux /dev/sdb2 1081344 30535679 29454336 14G 83 Linux  
19. Write the partition table to disk and exit ('w')
20. Create a new ext4 filesystem on the first partition: 'mkfs.ext4 /dev/sdb1' (you may need to add -F if it complains about an existing ext4 signature)
21. Create a new btrfs filesystem on the second partition: 'mkfs.btrfs /dev/sdb2'
22. Mount the new root filesystem:
# mkdir /mnt/nasrootbtrfs # sudo mount -o compress=zstd /dev/sdb2 /mnt/nasrootbtrfs  
23. Make some initial subvolumes (note that '@' will be the root subvolume, as this is preferred by 'snapper'):
# cd /mnt/nasrootbtrfs # btrfs subvol create @ # btrfs subvol create @home # btrfs subvol create @var_log  
24. Mount the boot partition, and the @home and @var_log subvolumes
# cd @ # mkdir boot # mount /dev/sdb1 boot/ # mkdir home # mount /dev/sdb2 -o noatime,compress=zstd,subvol=/@home home # mkdir -p var/log # mount /dev/sdb2 -o noatime,compress=zstd,subvol=/@var_log var/log # mount | grep sdb /dev/sdb2 on /mnt/nasrootbtrfs type btrfs (rw,relatime,compress=zstd,space_cache,subvolid=5,subvol=/) /dev/sdb1 on /mnt/nasrootbtrfs/@/boot type ext4 (rw,relatime) /dev/sdb2 on /mnt/nasrootbtrfs/@/home type btrfs (rw,noatime,compress=zstd,space_cache,subvolid=258,subvol=/@home) /dev/sdb2 on /mnt/nasrootbtrfs/@/var/log type btrfs (rw,noatime,compress=zstd,space_cache,subvolid=260,subvol=/@var_log)  
25. Note that our current directory is '/mnt/nasrootbtrfs/@'
26. Untar the old root filesystem into the new root directory.  Since boot, home, and var/log are mounted, tar will extract the relevant data into these mounted filesystems.
# tar -xvf /root/helios64_root.tar --acls --xattrs --numeric-owner |@ tee /root/extract_log.txt This is the only spot where things did not go as expected.  I received a handful of errors about "No space left on device" while untarring.  These seem to be transient btrfs errors. I use btrfs pretty extensively at work, and I've seen cases in the past on older kernels where btrfs would falsely report ENOSPC when the filesystem was under heavy load.  I'm not sure if that is what is happening here, but I would guess it's a bug.  It could be a bug in the kernel on my laptop, or it could be a bug in the usb-c mass storage export on the helios64.  To recover from this, I ran the tar command a couple of times.  I continued to receive these errors on different files.  I did this three or four times.  I also tried removing --acls and --xattrs, but it didn't seem to make much of a difference.  On my last attempt to untar, all of the files that failed happened to be from usr/lib/python3, so I untarred just that directory: 'tar -xvf /root/helios64_root.tar --numeric-owner ./usr/lib/python3'.  That extracted without error.  I was satisfied that everything made it back on disk.  Perhaps if you only extracted one top-level directory at a time, it would not produce any errors.  Maybe there is some way to slow down the 'tar' to avoid these ENOSPC errors.
 
27. After extracting the root filesystem, I made a snapshot of it: 'btrfs subvol snap -r . root_post_copy'
28. Update /mnt/nasrootbtrfs/@/etc/fstab.  This was my old fstab:
Old version: UUID=dabb3dbf-8631-4051-a032-c0b97eb285bd / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1 tmpfs /tmp tmpfs defaults,nosuid 0 0 # >>> [openmediavault] /dev/disk/by-label/helios64_btrfs_raid1 /srv/dev-disk-by-label-helios64_btrfs_raid1 btrfs defaults,nofail,noatime,nodiratime,compress=zstd 0 2 # <<< [openmediavault]  
Look up the new UUID for /dev/sdb2.  I did this by running 'ls -l /dev/disk/by-uuid').  Updated /etc/fstab:
UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f / btrfs defaults,noatime,compress=zstd,ssd,subvol=@ 0 0 UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /home btrfs defaults,noatime,compress=zstd,ssd,subvol=@home 0 0 UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /var/log/ btrfs defaults,noatime,compress=zstd,ssd,subvol=@var_log 0 0 UUID=fd5b620c-57e2-4031-b56c-3c64ce7c7d5f /boot ext4 defaults,noatime 0 2 tmpfs /tmp tmpfs defaults,nosuid 0 0 # >>> [openmediavault] /dev/disk/by-label/helios64_btrfs_raid1 /srv/dev-disk-by-label-helios64_btrfs_raid1 btrfs defaults,nofail,noatime,nodiratime,compress=zstd 0 2 # <<< [openmediavault]  
29. Modify /mnt/nasrootbtrfs/@/boot/armbianEnv.txt.  In particular modify rootdev, rootfstype, and add 'extraargs'.  Old version:
verbosity=1 bootlogo=false overlay_prefix=rockchip rootdev=UUID=dabb3dbf-8631-4051-a032-c0b97eb285bd rootfstype=ext4 usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u  
New version:
verbosity=1 bootlogo=false overlay_prefix=rockchip rootdev=UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f rootfstype=btrfs usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u extraargs=rootflags=subvol=@  
30. Make a 'boot' symlink.  I'm not sure if this is necessary, but u-boot expects the boot files to be in the '/boot' directory on the root filesystem, but they are now in the root directory of the boot partition.  This symlink makes sure that if u-boot tries to access /boot/foo, it will actually access /foo.
# cd /mnt/nasrootbtrfs/@/boot # ln -s . boot  
31.  Unmount everything:
# umount /mnt/nasrootbtrfs/@/var/log # umount /mnt/nasrootbtrfs/@/home # umount /mnt/nasrootbtrfs/@/boot # umount /mnt/nasrootbtrfs/  
32. Power off helios64
33. Remove sd card
34. Start picocom on your laptop ('picocom -b 1500000 /dev/ttyUSB0')
35. Turn on helios 64 and look for startup.
 
TROUBLESHOOTING:
Of course this did not work on the first attempt for me.  After rebooting the helios64, it never successfully booted into Linux.  I didn't see any errors from u-boot.  To troubleshoot, I wrote the original helios64 debian buster image to sd card and booted the helios64 to that.  I mounted /dev/mmcblk1p1 to /mnt, and changed the 'verbosity' to 7 in armbianEnv.txt.  I eventually discovered that I had the UUID wrong in /boot/armbianEnv.txt.  Correcting that value fixed the issue, and the helios64 booted successfully from eMMC.
 
SETTING UP AUTOMATED SNAPSHOTS AND BACKUPS:
Enabling 'snapper' for automated root snapshots was very easy:
1. sudo apt install snapper
2. sudo snapper -c root create-config /
 
The debian snapper automatically take snapshots before and after apt upgrades.  However since /boot is a separate filesystem, this is not included in these snapshots.  As a workaround, I added an apt hook which rsyncs /boot to /.bootbackup before the snapshots are made.  Note that the 'snapper' hook is /etc/apt/apt.conf.d/80snapper, so I created /etc/apt/apt.conf.d/79-snapper-boot-backup:
# /etc/apt/apt.conf.d/79-snapper-boot-backup DPkg::Pre-Invoke { "if mountpoint -q /boot && [ -e /.bootbackup ] ; then rsync -a --delete /boot /.bootbackup || true ; fi"; }; DPkg::Post-Invoke { "if mountpoint -q /boot && [ -e /.bootbackup ] ; then rsync -a --delete /boot /.bootbackup || true ; fi"; };  
This takes care of automated snapshots.  I still want snapshots stored on another drive (or another system) for redundancy.  I'm using 'btrbk' to do this.  btrbk wants the btrfs top-level subvolume mounted somewhere.  To do so, I created the directory /mnt/btr_pool, and added the following line to /etc/fstab:
UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /mnt/btr_pool/ btrfs defaults,noatime,compress=zstd,ssd,subvolid=5 0 0  
My spinning drive array on the helios64 is a btrfs filesystem, and I have it mounted on /srv/dev-disk-by-label-helios64_btrfs_raid1.  I created a '/srv/.../backups/helios64/btrfs/volumes' directory.  I then created the following three files:
 
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/btrbk.conf:
timestamp_format long volume /mnt/btr_pool snapshot_dir btrbk snapshot_preserve 30d snapshot_preserve_min latest target_preserve 7d 8w 6m target_preserve_min latest target send-receive /srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/volumes subvolume @ subvolume @home subvolume @var_log  
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/rsync_boot.sh:
#!/bin/sh set -e -x mountpoint /boot [ -e /.bootbackup ] rsync -a --delete /boot /.bootbackup  
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/run.sh
#!/bin/bash set -e -x CDIR=$( dirname ${BASH_SOURCE[0]} ) ${CDIR}/rsync_boot.sh btrbk -c ${CDIR}/btrbk.conf run  
I then added a daily cron job which runs the 'run.sh' script.
 
Well, that was a longer post than I had imagined.  I hope someone finds it helpful.

Another approach I thought of after the fact is booting the helios64 to the stock armbian image on a sd-card, and doing all of the filesystem operations connected over SSH.  You'll still need a place to copy the old root filesystem to.  You could use the sd card (which would be slow), a USB storage device, or you could mount the 3.5" storage array from the helios.  I would be curious to try this to see if this avoids the "no space left on device" errors I was getting when un-tarring the root filesystem back onto the new btrfs filesystem.

Hi,
 
I wanted to convert my root filesystem to btrfs.  This allows me to use snapper to take regular snapshops.  I also use btrbk to send my snapshots to my secondary storage (also a btrfs array).  My root filesystems is installed to eMMC, but there isn't any reason you couldn't do this with an sd card.  These instructions are done with a laptop running Linux.  I'm using Debian in this case, although any distro with a recent kernel and btrfs tools installed should work.
 
CAVEATS:
The current u-boot image (2020.08.21) doesn't support booting natively to btrfs.  u-boot *does* support btrfs though.  Hopefully a future build of u-boot will add btrfs support.  To work around this, we have to create an ext4 /boot partition, and then a btrfs root partition.
 
1. To start, I booted the helios64 via sd card to the USB mass storage image (https://wiki.kobol.io/helios64/install/emmc/).  I then connected a laptop to the helios64 via the USB-C cable, and the emmc appeared as /dev/sdb.  The rest of these instructions will assume the eMMC is /dev/sdb, so please adjust accordingly for your system.
 
2. Mount the eMMC root filesystem on the host:
# mkdir /mnt/nasrootext4 # mount /dev/sdb1 -o ro /mnt/nasrootext4  
3. Back up the contents of the root filesystem:
# tar -C /mnt/nasrootext4 --acls --xattrs -cf root_backup.tar .  
4. Make a backup of the partition layout (in case you decide to revert this modification):
# fdisk -l /dev/sdb | tee nas_fdisk.txt  
5. Unmount the filesystem, and run fdisk on it.  Make note of the start sector.  It should be 32768.
# umount /dev/sdb1 # fdisk /dev/sdb Command (m for help): p Disk /dev/sdb: 14.6 GiB, 15634268160 bytes, 30535680 sectors Disk model: UMS disk 0 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x794023b5 Device Boot Start End Sectors Size Id Type /dev/sdb1 32768 30230303 30197536 14.4G 83 Linux  
6. Delete the existing partition ('d')
7. Create a new partition ('n')
8. Select 'primary' partition type
9. Select partition number 1
10. Select first sector: 32768 (This is very important to get right, and should match the start sector of the partition that was just deleted)
11. Select last sector: +512M (this create a 512MB /boot partition)
12. You will see a message "Partition #1 contains a ext4 signature.".  This is okay.  I selected 'n' (don't remove signature) but it doesn't matter.  You should probably remove the signature since we're going to create a new ext4 filesystem anyway.
13. Create new partition ('n')
14. Select 'primary' partition type
15. Select partition number 2
16. First sector: 1081344 (this is one more than the end sector of the other partition)
17. Last sector: (leave the default as the full drive, 30535679 in my case)
18. Print the layout to confirm:
Disk /dev/sdb: 14.6 GiB, 15634268160 bytes, 30535680 sectors Disk model: UMS disk 0 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x794023b5 Device Boot Start End Sectors Size Id Type /dev/sdb1 32768 1081343 1048576 512M 83 Linux /dev/sdb2 1081344 30535679 29454336 14G 83 Linux  
19. Write the partition table to disk and exit ('w')
20. Create a new ext4 filesystem on the first partition: 'mkfs.ext4 /dev/sdb1' (you may need to add -F if it complains about an existing ext4 signature)
21. Create a new btrfs filesystem on the second partition: 'mkfs.btrfs /dev/sdb2'
22. Mount the new root filesystem:
# mkdir /mnt/nasrootbtrfs # sudo mount -o compress=zstd /dev/sdb2 /mnt/nasrootbtrfs  
23. Make some initial subvolumes (note that '@' will be the root subvolume, as this is preferred by 'snapper'):
# cd /mnt/nasrootbtrfs # btrfs subvol create @ # btrfs subvol create @home # btrfs subvol create @var_log  
24. Mount the boot partition, and the @home and @var_log subvolumes
# cd @ # mkdir boot # mount /dev/sdb1 boot/ # mkdir home # mount /dev/sdb2 -o noatime,compress=zstd,subvol=/@home home # mkdir -p var/log # mount /dev/sdb2 -o noatime,compress=zstd,subvol=/@var_log var/log # mount | grep sdb /dev/sdb2 on /mnt/nasrootbtrfs type btrfs (rw,relatime,compress=zstd,space_cache,subvolid=5,subvol=/) /dev/sdb1 on /mnt/nasrootbtrfs/@/boot type ext4 (rw,relatime) /dev/sdb2 on /mnt/nasrootbtrfs/@/home type btrfs (rw,noatime,compress=zstd,space_cache,subvolid=258,subvol=/@home) /dev/sdb2 on /mnt/nasrootbtrfs/@/var/log type btrfs (rw,noatime,compress=zstd,space_cache,subvolid=260,subvol=/@var_log)  
25. Note that our current directory is '/mnt/nasrootbtrfs/@'
26. Untar the old root filesystem into the new root directory.  Since boot, home, and var/log are mounted, tar will extract the relevant data into these mounted filesystems.
# tar -xvf /root/helios64_root.tar --acls --xattrs --numeric-owner |@ tee /root/extract_log.txt This is the only spot where things did not go as expected.  I received a handful of errors about "No space left on device" while untarring.  These seem to be transient btrfs errors. I use btrfs pretty extensively at work, and I've seen cases in the past on older kernels where btrfs would falsely report ENOSPC when the filesystem was under heavy load.  I'm not sure if that is what is happening here, but I would guess it's a bug.  It could be a bug in the kernel on my laptop, or it could be a bug in the usb-c mass storage export on the helios64.  To recover from this, I ran the tar command a couple of times.  I continued to receive these errors on different files.  I did this three or four times.  I also tried removing --acls and --xattrs, but it didn't seem to make much of a difference.  On my last attempt to untar, all of the files that failed happened to be from usr/lib/python3, so I untarred just that directory: 'tar -xvf /root/helios64_root.tar --numeric-owner ./usr/lib/python3'.  That extracted without error.  I was satisfied that everything made it back on disk.  Perhaps if you only extracted one top-level directory at a time, it would not produce any errors.  Maybe there is some way to slow down the 'tar' to avoid these ENOSPC errors.
 
27. After extracting the root filesystem, I made a snapshot of it: 'btrfs subvol snap -r . root_post_copy'
28. Update /mnt/nasrootbtrfs/@/etc/fstab.  This was my old fstab:
Old version: UUID=dabb3dbf-8631-4051-a032-c0b97eb285bd / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1 tmpfs /tmp tmpfs defaults,nosuid 0 0 # >>> [openmediavault] /dev/disk/by-label/helios64_btrfs_raid1 /srv/dev-disk-by-label-helios64_btrfs_raid1 btrfs defaults,nofail,noatime,nodiratime,compress=zstd 0 2 # <<< [openmediavault]  
Look up the new UUID for /dev/sdb2.  I did this by running 'ls -l /dev/disk/by-uuid').  Updated /etc/fstab:
UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f / btrfs defaults,noatime,compress=zstd,ssd,subvol=@ 0 0 UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /home btrfs defaults,noatime,compress=zstd,ssd,subvol=@home 0 0 UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /var/log/ btrfs defaults,noatime,compress=zstd,ssd,subvol=@var_log 0 0 UUID=fd5b620c-57e2-4031-b56c-3c64ce7c7d5f /boot ext4 defaults,noatime 0 2 tmpfs /tmp tmpfs defaults,nosuid 0 0 # >>> [openmediavault] /dev/disk/by-label/helios64_btrfs_raid1 /srv/dev-disk-by-label-helios64_btrfs_raid1 btrfs defaults,nofail,noatime,nodiratime,compress=zstd 0 2 # <<< [openmediavault]  
29. Modify /mnt/nasrootbtrfs/@/boot/armbianEnv.txt.  In particular modify rootdev, rootfstype, and add 'extraargs'.  Old version:
verbosity=1 bootlogo=false overlay_prefix=rockchip rootdev=UUID=dabb3dbf-8631-4051-a032-c0b97eb285bd rootfstype=ext4 usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u  
New version:
verbosity=1 bootlogo=false overlay_prefix=rockchip rootdev=UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f rootfstype=btrfs usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u extraargs=rootflags=subvol=@  
30. Make a 'boot' symlink.  I'm not sure if this is necessary, but u-boot expects the boot files to be in the '/boot' directory on the root filesystem, but they are now in the root directory of the boot partition.  This symlink makes sure that if u-boot tries to access /boot/foo, it will actually access /foo.
# cd /mnt/nasrootbtrfs/@/boot # ln -s . boot  
31.  Unmount everything:
# umount /mnt/nasrootbtrfs/@/var/log # umount /mnt/nasrootbtrfs/@/home # umount /mnt/nasrootbtrfs/@/boot # umount /mnt/nasrootbtrfs/  
32. Power off helios64
33. Remove sd card
34. Start picocom on your laptop ('picocom -b 1500000 /dev/ttyUSB0')
35. Turn on helios 64 and look for startup.
 
TROUBLESHOOTING:
Of course this did not work on the first attempt for me.  After rebooting the helios64, it never successfully booted into Linux.  I didn't see any errors from u-boot.  To troubleshoot, I wrote the original helios64 debian buster image to sd card and booted the helios64 to that.  I mounted /dev/mmcblk1p1 to /mnt, and changed the 'verbosity' to 7 in armbianEnv.txt.  I eventually discovered that I had the UUID wrong in /boot/armbianEnv.txt.  Correcting that value fixed the issue, and the helios64 booted successfully from eMMC.
 
SETTING UP AUTOMATED SNAPSHOTS AND BACKUPS:
Enabling 'snapper' for automated root snapshots was very easy:
1. sudo apt install snapper
2. sudo snapper -c root create-config /
 
The debian snapper automatically take snapshots before and after apt upgrades.  However since /boot is a separate filesystem, this is not included in these snapshots.  As a workaround, I added an apt hook which rsyncs /boot to /.bootbackup before the snapshots are made.  Note that the 'snapper' hook is /etc/apt/apt.conf.d/80snapper, so I created /etc/apt/apt.conf.d/79-snapper-boot-backup:
# /etc/apt/apt.conf.d/79-snapper-boot-backup DPkg::Pre-Invoke { "if mountpoint -q /boot && [ -e /.bootbackup ] ; then rsync -a --delete /boot /.bootbackup || true ; fi"; }; DPkg::Post-Invoke { "if mountpoint -q /boot && [ -e /.bootbackup ] ; then rsync -a --delete /boot /.bootbackup || true ; fi"; };  
This takes care of automated snapshots.  I still want snapshots stored on another drive (or another system) for redundancy.  I'm using 'btrbk' to do this.  btrbk wants the btrfs top-level subvolume mounted somewhere.  To do so, I created the directory /mnt/btr_pool, and added the following line to /etc/fstab:
UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /mnt/btr_pool/ btrfs defaults,noatime,compress=zstd,ssd,subvolid=5 0 0  
My spinning drive array on the helios64 is a btrfs filesystem, and I have it mounted on /srv/dev-disk-by-label-helios64_btrfs_raid1.  I created a '/srv/.../backups/helios64/btrfs/volumes' directory.  I then created the following three files:
 
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/btrbk.conf:
timestamp_format long volume /mnt/btr_pool snapshot_dir btrbk snapshot_preserve 30d snapshot_preserve_min latest target_preserve 7d 8w 6m target_preserve_min latest target send-receive /srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/volumes subvolume @ subvolume @home subvolume @var_log  
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/rsync_boot.sh:
#!/bin/sh set -e -x mountpoint /boot [ -e /.bootbackup ] rsync -a --delete /boot /.bootbackup  
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/run.sh
#!/bin/bash set -e -x CDIR=$( dirname ${BASH_SOURCE[0]} ) ${CDIR}/rsync_boot.sh btrbk -c ${CDIR}/btrbk.conf run  
I then added a daily cron job which runs the 'run.sh' script.
 
Well, that was a longer post than I had imagined.  I hope someone finds it helpful.

Hi,
 
I wanted to convert my root filesystem to btrfs.  This allows me to use snapper to take regular snapshops.  I also use btrbk to send my snapshots to my secondary storage (also a btrfs array).  My root filesystems is installed to eMMC, but there isn't any reason you couldn't do this with an sd card.  These instructions are done with a laptop running Linux.  I'm using Debian in this case, although any distro with a recent kernel and btrfs tools installed should work.
 
CAVEATS:
The current u-boot image (2020.08.21) doesn't support booting natively to btrfs.  u-boot *does* support btrfs though.  Hopefully a future build of u-boot will add btrfs support.  To work around this, we have to create an ext4 /boot partition, and then a btrfs root partition.
 
1. To start, I booted the helios64 via sd card to the USB mass storage image (https://wiki.kobol.io/helios64/install/emmc/).  I then connected a laptop to the helios64 via the USB-C cable, and the emmc appeared as /dev/sdb.  The rest of these instructions will assume the eMMC is /dev/sdb, so please adjust accordingly for your system.
 
2. Mount the eMMC root filesystem on the host:
# mkdir /mnt/nasrootext4 # mount /dev/sdb1 -o ro /mnt/nasrootext4  
3. Back up the contents of the root filesystem:
# tar -C /mnt/nasrootext4 --acls --xattrs -cf root_backup.tar .  
4. Make a backup of the partition layout (in case you decide to revert this modification):
# fdisk -l /dev/sdb | tee nas_fdisk.txt  
5. Unmount the filesystem, and run fdisk on it.  Make note of the start sector.  It should be 32768.
# umount /dev/sdb1 # fdisk /dev/sdb Command (m for help): p Disk /dev/sdb: 14.6 GiB, 15634268160 bytes, 30535680 sectors Disk model: UMS disk 0 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x794023b5 Device Boot Start End Sectors Size Id Type /dev/sdb1 32768 30230303 30197536 14.4G 83 Linux  
6. Delete the existing partition ('d')
7. Create a new partition ('n')
8. Select 'primary' partition type
9. Select partition number 1
10. Select first sector: 32768 (This is very important to get right, and should match the start sector of the partition that was just deleted)
11. Select last sector: +512M (this create a 512MB /boot partition)
12. You will see a message "Partition #1 contains a ext4 signature.".  This is okay.  I selected 'n' (don't remove signature) but it doesn't matter.  You should probably remove the signature since we're going to create a new ext4 filesystem anyway.
13. Create new partition ('n')
14. Select 'primary' partition type
15. Select partition number 2
16. First sector: 1081344 (this is one more than the end sector of the other partition)
17. Last sector: (leave the default as the full drive, 30535679 in my case)
18. Print the layout to confirm:
Disk /dev/sdb: 14.6 GiB, 15634268160 bytes, 30535680 sectors Disk model: UMS disk 0 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x794023b5 Device Boot Start End Sectors Size Id Type /dev/sdb1 32768 1081343 1048576 512M 83 Linux /dev/sdb2 1081344 30535679 29454336 14G 83 Linux  
19. Write the partition table to disk and exit ('w')
20. Create a new ext4 filesystem on the first partition: 'mkfs.ext4 /dev/sdb1' (you may need to add -F if it complains about an existing ext4 signature)
21. Create a new btrfs filesystem on the second partition: 'mkfs.btrfs /dev/sdb2'
22. Mount the new root filesystem:
# mkdir /mnt/nasrootbtrfs # sudo mount -o compress=zstd /dev/sdb2 /mnt/nasrootbtrfs  
23. Make some initial subvolumes (note that '@' will be the root subvolume, as this is preferred by 'snapper'):
# cd /mnt/nasrootbtrfs # btrfs subvol create @ # btrfs subvol create @home # btrfs subvol create @var_log  
24. Mount the boot partition, and the @home and @var_log subvolumes
# cd @ # mkdir boot # mount /dev/sdb1 boot/ # mkdir home # mount /dev/sdb2 -o noatime,compress=zstd,subvol=/@home home # mkdir -p var/log # mount /dev/sdb2 -o noatime,compress=zstd,subvol=/@var_log var/log # mount | grep sdb /dev/sdb2 on /mnt/nasrootbtrfs type btrfs (rw,relatime,compress=zstd,space_cache,subvolid=5,subvol=/) /dev/sdb1 on /mnt/nasrootbtrfs/@/boot type ext4 (rw,relatime) /dev/sdb2 on /mnt/nasrootbtrfs/@/home type btrfs (rw,noatime,compress=zstd,space_cache,subvolid=258,subvol=/@home) /dev/sdb2 on /mnt/nasrootbtrfs/@/var/log type btrfs (rw,noatime,compress=zstd,space_cache,subvolid=260,subvol=/@var_log)  
25. Note that our current directory is '/mnt/nasrootbtrfs/@'
26. Untar the old root filesystem into the new root directory.  Since boot, home, and var/log are mounted, tar will extract the relevant data into these mounted filesystems.
# tar -xvf /root/helios64_root.tar --acls --xattrs --numeric-owner |@ tee /root/extract_log.txt This is the only spot where things did not go as expected.  I received a handful of errors about "No space left on device" while untarring.  These seem to be transient btrfs errors. I use btrfs pretty extensively at work, and I've seen cases in the past on older kernels where btrfs would falsely report ENOSPC when the filesystem was under heavy load.  I'm not sure if that is what is happening here, but I would guess it's a bug.  It could be a bug in the kernel on my laptop, or it could be a bug in the usb-c mass storage export on the helios64.  To recover from this, I ran the tar command a couple of times.  I continued to receive these errors on different files.  I did this three or four times.  I also tried removing --acls and --xattrs, but it didn't seem to make much of a difference.  On my last attempt to untar, all of the files that failed happened to be from usr/lib/python3, so I untarred just that directory: 'tar -xvf /root/helios64_root.tar --numeric-owner ./usr/lib/python3'.  That extracted without error.  I was satisfied that everything made it back on disk.  Perhaps if you only extracted one top-level directory at a time, it would not produce any errors.  Maybe there is some way to slow down the 'tar' to avoid these ENOSPC errors.
 
27. After extracting the root filesystem, I made a snapshot of it: 'btrfs subvol snap -r . root_post_copy'
28. Update /mnt/nasrootbtrfs/@/etc/fstab.  This was my old fstab:
Old version: UUID=dabb3dbf-8631-4051-a032-c0b97eb285bd / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1 tmpfs /tmp tmpfs defaults,nosuid 0 0 # >>> [openmediavault] /dev/disk/by-label/helios64_btrfs_raid1 /srv/dev-disk-by-label-helios64_btrfs_raid1 btrfs defaults,nofail,noatime,nodiratime,compress=zstd 0 2 # <<< [openmediavault]  
Look up the new UUID for /dev/sdb2.  I did this by running 'ls -l /dev/disk/by-uuid').  Updated /etc/fstab:
UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f / btrfs defaults,noatime,compress=zstd,ssd,subvol=@ 0 0 UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /home btrfs defaults,noatime,compress=zstd,ssd,subvol=@home 0 0 UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /var/log/ btrfs defaults,noatime,compress=zstd,ssd,subvol=@var_log 0 0 UUID=fd5b620c-57e2-4031-b56c-3c64ce7c7d5f /boot ext4 defaults,noatime 0 2 tmpfs /tmp tmpfs defaults,nosuid 0 0 # >>> [openmediavault] /dev/disk/by-label/helios64_btrfs_raid1 /srv/dev-disk-by-label-helios64_btrfs_raid1 btrfs defaults,nofail,noatime,nodiratime,compress=zstd 0 2 # <<< [openmediavault]  
29. Modify /mnt/nasrootbtrfs/@/boot/armbianEnv.txt.  In particular modify rootdev, rootfstype, and add 'extraargs'.  Old version:
verbosity=1 bootlogo=false overlay_prefix=rockchip rootdev=UUID=dabb3dbf-8631-4051-a032-c0b97eb285bd rootfstype=ext4 usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u  
New version:
verbosity=1 bootlogo=false overlay_prefix=rockchip rootdev=UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f rootfstype=btrfs usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u extraargs=rootflags=subvol=@  
30. Make a 'boot' symlink.  I'm not sure if this is necessary, but u-boot expects the boot files to be in the '/boot' directory on the root filesystem, but they are now in the root directory of the boot partition.  This symlink makes sure that if u-boot tries to access /boot/foo, it will actually access /foo.
# cd /mnt/nasrootbtrfs/@/boot # ln -s . boot  
31.  Unmount everything:
# umount /mnt/nasrootbtrfs/@/var/log # umount /mnt/nasrootbtrfs/@/home # umount /mnt/nasrootbtrfs/@/boot # umount /mnt/nasrootbtrfs/  
32. Power off helios64
33. Remove sd card
34. Start picocom on your laptop ('picocom -b 1500000 /dev/ttyUSB0')
35. Turn on helios 64 and look for startup.
 
TROUBLESHOOTING:
Of course this did not work on the first attempt for me.  After rebooting the helios64, it never successfully booted into Linux.  I didn't see any errors from u-boot.  To troubleshoot, I wrote the original helios64 debian buster image to sd card and booted the helios64 to that.  I mounted /dev/mmcblk1p1 to /mnt, and changed the 'verbosity' to 7 in armbianEnv.txt.  I eventually discovered that I had the UUID wrong in /boot/armbianEnv.txt.  Correcting that value fixed the issue, and the helios64 booted successfully from eMMC.
 
SETTING UP AUTOMATED SNAPSHOTS AND BACKUPS:
Enabling 'snapper' for automated root snapshots was very easy:
1. sudo apt install snapper
2. sudo snapper -c root create-config /
 
The debian snapper automatically take snapshots before and after apt upgrades.  However since /boot is a separate filesystem, this is not included in these snapshots.  As a workaround, I added an apt hook which rsyncs /boot to /.bootbackup before the snapshots are made.  Note that the 'snapper' hook is /etc/apt/apt.conf.d/80snapper, so I created /etc/apt/apt.conf.d/79-snapper-boot-backup:
# /etc/apt/apt.conf.d/79-snapper-boot-backup DPkg::Pre-Invoke { "if mountpoint -q /boot && [ -e /.bootbackup ] ; then rsync -a --delete /boot /.bootbackup || true ; fi"; }; DPkg::Post-Invoke { "if mountpoint -q /boot && [ -e /.bootbackup ] ; then rsync -a --delete /boot /.bootbackup || true ; fi"; };  
This takes care of automated snapshots.  I still want snapshots stored on another drive (or another system) for redundancy.  I'm using 'btrbk' to do this.  btrbk wants the btrfs top-level subvolume mounted somewhere.  To do so, I created the directory /mnt/btr_pool, and added the following line to /etc/fstab:
UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /mnt/btr_pool/ btrfs defaults,noatime,compress=zstd,ssd,subvolid=5 0 0  
My spinning drive array on the helios64 is a btrfs filesystem, and I have it mounted on /srv/dev-disk-by-label-helios64_btrfs_raid1.  I created a '/srv/.../backups/helios64/btrfs/volumes' directory.  I then created the following three files:
 
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/btrbk.conf:
timestamp_format long volume /mnt/btr_pool snapshot_dir btrbk snapshot_preserve 30d snapshot_preserve_min latest target_preserve 7d 8w 6m target_preserve_min latest target send-receive /srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/volumes subvolume @ subvolume @home subvolume @var_log  
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/rsync_boot.sh:
#!/bin/sh set -e -x mountpoint /boot [ -e /.bootbackup ] rsync -a --delete /boot /.bootbackup  
/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/run.sh
#!/bin/bash set -e -x CDIR=$( dirname ${BASH_SOURCE[0]} ) ${CDIR}/rsync_boot.sh btrbk -c ${CDIR}/btrbk.conf run  
I then added a daily cron job which runs the 'run.sh' script.
 
Well, that was a longer post than I had imagined.  I hope someone finds it helpful.