Jump to content

HOWTO: btrfs root filesystem


m11k

Recommended Posts

Hi,

 

I wanted to convert my root filesystem to btrfs.  This allows me to use snapper to take regular snapshops.  I also use btrbk to send my snapshots to my secondary storage (also a btrfs array).  My root filesystems is installed to eMMC, but there isn't any reason you couldn't do this with an sd card.  These instructions are done with a laptop running Linux.  I'm using Debian in this case, although any distro with a recent kernel and btrfs tools installed should work.

 

CAVEATS:

The current u-boot image (2020.08.21) doesn't support booting natively to btrfs.  u-boot *does* support btrfs though.  Hopefully a future build of u-boot will add btrfs support.  To work around this, we have to create an ext4 /boot partition, and then a btrfs root partition.

 

1. To start, I booted the helios64 via sd card to the USB mass storage image (https://wiki.kobol.io/helios64/install/emmc/).  I then connected a laptop to the helios64 via the USB-C cable, and the emmc appeared as /dev/sdb.  The rest of these instructions will assume the eMMC is /dev/sdb, so please adjust accordingly for your system.

 

2. Mount the eMMC root filesystem on the host:

# mkdir /mnt/nasrootext4
# mount /dev/sdb1 -o ro /mnt/nasrootext4

 

3. Back up the contents of the root filesystem:

# tar -C /mnt/nasrootext4 --acls --xattrs -cf root_backup.tar .

 

4. Make a backup of the partition layout (in case you decide to revert this modification):

# fdisk -l /dev/sdb | tee nas_fdisk.txt

 

5. Unmount the filesystem, and run fdisk on it.  Make note of the start sector.  It should be 32768.

# umount /dev/sdb1
# fdisk /dev/sdb
Command (m for help): p
Disk /dev/sdb: 14.6 GiB, 15634268160 bytes, 30535680 sectors
Disk model: UMS disk 0      
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x794023b5

Device     Boot Start      End  Sectors  Size Id Type
/dev/sdb1       32768 30230303 30197536 14.4G 83 Linux

 

6. Delete the existing partition ('d')

7. Create a new partition ('n')

8. Select 'primary' partition type

9. Select partition number 1

10. Select first sector: 32768 (This is very important to get right, and should match the start sector of the partition that was just deleted)

11. Select last sector: +512M (this create a 512MB /boot partition)

12. You will see a message "Partition #1 contains a ext4 signature.".  This is okay.  I selected 'n' (don't remove signature) but it doesn't matter.  You should probably remove the signature since we're going to create a new ext4 filesystem anyway.

13. Create new partition ('n')

14. Select 'primary' partition type

15. Select partition number 2

16. First sector: 1081344 (this is one more than the end sector of the other partition)

17. Last sector: (leave the default as the full drive, 30535679 in my case)

18. Print the layout to confirm:

Disk /dev/sdb: 14.6 GiB, 15634268160 bytes, 30535680 sectors
Disk model: UMS disk 0      
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x794023b5

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sdb1         32768  1081343  1048576  512M 83 Linux
/dev/sdb2       1081344 30535679 29454336   14G 83 Linux

 

19. Write the partition table to disk and exit ('w')

20. Create a new ext4 filesystem on the first partition: 'mkfs.ext4 /dev/sdb1' (you may need to add -F if it complains about an existing ext4 signature)

21. Create a new btrfs filesystem on the second partition: 'mkfs.btrfs /dev/sdb2'

22. Mount the new root filesystem:

# mkdir /mnt/nasrootbtrfs
# sudo mount -o compress=zstd /dev/sdb2 /mnt/nasrootbtrfs

 

23. Make some initial subvolumes (note that '@' will be the root subvolume, as this is preferred by 'snapper'):

# cd /mnt/nasrootbtrfs
# btrfs subvol create @
# btrfs subvol create @home
# btrfs subvol create @var_log

 

24. Mount the boot partition, and the @home and @var_log subvolumes

# cd @
# mkdir boot
# mount /dev/sdb1 boot/
# mkdir home
# mount /dev/sdb2 -o noatime,compress=zstd,subvol=/@home home
# mkdir -p var/log
# mount /dev/sdb2 -o noatime,compress=zstd,subvol=/@var_log var/log

# mount | grep sdb
/dev/sdb2 on /mnt/nasrootbtrfs type btrfs (rw,relatime,compress=zstd,space_cache,subvolid=5,subvol=/)
/dev/sdb1 on /mnt/nasrootbtrfs/@/boot type ext4 (rw,relatime)
/dev/sdb2 on /mnt/nasrootbtrfs/@/home type btrfs (rw,noatime,compress=zstd,space_cache,subvolid=258,subvol=/@home)
/dev/sdb2 on /mnt/nasrootbtrfs/@/var/log type btrfs (rw,noatime,compress=zstd,space_cache,subvolid=260,subvol=/@var_log)

 

25. Note that our current directory is '/mnt/nasrootbtrfs/@'

26. Untar the old root filesystem into the new root directory.  Since boot, home, and var/log are mounted, tar will extract the relevant data into these mounted filesystems.

# tar -xvf /root/helios64_root.tar --acls --xattrs --numeric-owner |@ tee /root/extract_log.txt

This is the only spot where things did not go as expected.  I received a handful of errors about "No space left on device" while untarring.  These seem to be transient btrfs errors. I use btrfs pretty extensively at work, and I've seen cases in the past on older kernels where btrfs would falsely report ENOSPC when the filesystem was under heavy load.  I'm not sure if that is what is happening here, but I would guess it's a bug.  It could be a bug in the kernel on my laptop, or it could be a bug in the usb-c mass storage export on the helios64.  To recover from this, I ran the tar command a couple of times.  I continued to receive these errors on different files.  I did this three or four times.  I also tried removing --acls and --xattrs, but it didn't seem to make much of a difference.  On my last attempt to untar, all of the files that failed happened to be from usr/lib/python3, so I untarred just that directory: 'tar -xvf /root/helios64_root.tar --numeric-owner ./usr/lib/python3'.  That extracted without error.  I was satisfied that everything made it back on disk.  Perhaps if you only extracted one top-level directory at a time, it would not produce any errors.  Maybe there is some way to slow down the 'tar' to avoid these ENOSPC errors.

 

27. After extracting the root filesystem, I made a snapshot of it: 'btrfs subvol snap -r . root_post_copy'

28. Update /mnt/nasrootbtrfs/@/etc/fstab.  This was my old fstab:

Old version:
UUID=dabb3dbf-8631-4051-a032-c0b97eb285bd / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1

tmpfs /tmp tmpfs defaults,nosuid 0 0
# >>> [openmediavault]
/dev/disk/by-label/helios64_btrfs_raid1         /srv/dev-disk-by-label-helios64_btrfs_raid1     btrfs   defaults,nofail,noatime,nodiratime,compress=zstd        0 2
# <<< [openmediavault]

 

Look up the new UUID for /dev/sdb2.  I did this by running 'ls -l /dev/disk/by-uuid').  Updated /etc/fstab:

UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f / btrfs defaults,noatime,compress=zstd,ssd,subvol=@ 0 0
UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /home btrfs defaults,noatime,compress=zstd,ssd,subvol=@home 0 0
UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /var/log/ btrfs defaults,noatime,compress=zstd,ssd,subvol=@var_log 0 0
UUID=fd5b620c-57e2-4031-b56c-3c64ce7c7d5f /boot ext4 defaults,noatime 0 2
tmpfs /tmp tmpfs defaults,nosuid 0 0
# >>> [openmediavault]
/dev/disk/by-label/helios64_btrfs_raid1         /srv/dev-disk-by-label-helios64_btrfs_raid1     btrfs   defaults,nofail,noatime,nodiratime,compress=zstd        0 2
# <<< [openmediavault]

 

29. Modify /mnt/nasrootbtrfs/@/boot/armbianEnv.txt.  In particular modify rootdev, rootfstype, and add 'extraargs'.  Old version:

verbosity=1
bootlogo=false
overlay_prefix=rockchip
rootdev=UUID=dabb3dbf-8631-4051-a032-c0b97eb285bd
rootfstype=ext4
usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u

 

New version:

verbosity=1
bootlogo=false
overlay_prefix=rockchip
rootdev=UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f
rootfstype=btrfs
usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u
extraargs=rootflags=subvol=@

 

30. Make a 'boot' symlink.  I'm not sure if this is necessary, but u-boot expects the boot files to be in the '/boot' directory on the root filesystem, but they are now in the root directory of the boot partition.  This symlink makes sure that if u-boot tries to access /boot/foo, it will actually access /foo.

# cd /mnt/nasrootbtrfs/@/boot
# ln -s . boot

 

31.  Unmount everything:

# umount /mnt/nasrootbtrfs/@/var/log
# umount /mnt/nasrootbtrfs/@/home
# umount /mnt/nasrootbtrfs/@/boot
# umount /mnt/nasrootbtrfs/

 

32. Power off helios64

33. Remove sd card

34. Start picocom on your laptop ('picocom -b 1500000 /dev/ttyUSB0')

35. Turn on helios 64 and look for startup.

 

TROUBLESHOOTING:

Of course this did not work on the first attempt for me.  After rebooting the helios64, it never successfully booted into Linux.  I didn't see any errors from u-boot.  To troubleshoot, I wrote the original helios64 debian buster image to sd card and booted the helios64 to that.  I mounted /dev/mmcblk1p1 to /mnt, and changed the 'verbosity' to 7 in armbianEnv.txt.  I eventually discovered that I had the UUID wrong in /boot/armbianEnv.txt.  Correcting that value fixed the issue, and the helios64 booted successfully from eMMC.

 

SETTING UP AUTOMATED SNAPSHOTS AND BACKUPS:

Enabling 'snapper' for automated root snapshots was very easy:

1. sudo apt install snapper

2. sudo snapper -c root create-config /

 

The debian snapper automatically take snapshots before and after apt upgrades.  However since /boot is a separate filesystem, this is not included in these snapshots.  As a workaround, I added an apt hook which rsyncs /boot to /.bootbackup before the snapshots are made.  Note that the 'snapper' hook is /etc/apt/apt.conf.d/80snapper, so I created /etc/apt/apt.conf.d/79-snapper-boot-backup:

# /etc/apt/apt.conf.d/79-snapper-boot-backup
  DPkg::Pre-Invoke  { "if mountpoint -q /boot && [ -e /.bootbackup ] ; then rsync -a --delete /boot /.bootbackup || true ; fi"; };
  DPkg::Post-Invoke  { "if mountpoint -q /boot && [ -e /.bootbackup ] ; then rsync -a --delete /boot /.bootbackup || true ; fi"; };

 

This takes care of automated snapshots.  I still want snapshots stored on another drive (or another system) for redundancy.  I'm using 'btrbk' to do this.  btrbk wants the btrfs top-level subvolume mounted somewhere.  To do so, I created the directory /mnt/btr_pool, and added the following line to /etc/fstab:

UUID=240bab7d-1b6c-48f4-898d-ba12abcecb3f /mnt/btr_pool/ btrfs defaults,noatime,compress=zstd,ssd,subvolid=5 0 0

 

My spinning drive array on the helios64 is a btrfs filesystem, and I have it mounted on /srv/dev-disk-by-label-helios64_btrfs_raid1.  I created a '/srv/.../backups/helios64/btrfs/volumes' directory.  I then created the following three files:

 

/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/btrbk.conf:

timestamp_format long

volume /mnt/btr_pool
  snapshot_dir btrbk
  snapshot_preserve 30d
  snapshot_preserve_min latest
  target_preserve 7d 8w 6m
  target_preserve_min latest
  target send-receive /srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/volumes
    subvolume @
    subvolume @home
    subvolume @var_log

 

/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/rsync_boot.sh:

#!/bin/sh

set -e -x

mountpoint /boot
[ -e /.bootbackup ]
rsync -a --delete /boot /.bootbackup

 

/srv/dev-disk-by-label-helios64_btrfs_raid1/backups/helios64/btrfs/run.sh

#!/bin/bash

set -e -x
CDIR=$( dirname ${BASH_SOURCE[0]} )
${CDIR}/rsync_boot.sh
btrbk -c ${CDIR}/btrbk.conf run

 

I then added a daily cron job which runs the 'run.sh' script.

 

Well, that was a longer post than I had imagined.  I hope someone finds it helpful.

Edited by m11k
fixes
Link to comment
Share on other sites

Another approach I thought of after the fact is booting the helios64 to the stock armbian image on a sd-card, and doing all of the filesystem operations connected over SSH.  You'll still need a place to copy the old root filesystem to.  You could use the sd card (which would be slow), a USB storage device, or you could mount the 3.5" storage array from the helios.  I would be curious to try this to see if this avoids the "no space left on device" errors I was getting when un-tarring the root filesystem back onto the new btrfs filesystem.

Link to comment
Share on other sites

7 hours ago, gprovost said:

@m11k Thanks for your tutorial. This could be a page on our wiki if you wish to contribute.   https://github.com/kobol-io/wiki

We are using mkdocs. So maybe once btrfs supports is added to u-boot, this instruction could be updated and put into a wiki page :P

 

Sure, I'd be happy to contribute this.  Rewriting it in markdown would be a pleasure, as I found all the clicking required in the forum a little frustrating for a long post.  When I get a few hours free, I'd like to try this process again using the helios itself booted to armbian on sd-card to do the conversion.  That way a second linux system wouldn't be required, and it might fix the random ENOSPC errors I encountered.

Link to comment
Share on other sites

On 11/25/2020 at 5:03 PM, m11k said:

Another approach I thought of after the fact is booting the helios64 to the stock armbian image on a sd-card, and doing all of the filesystem operations connected over SSH.  You'll still need a place to copy the old root filesystem to.  You could use the sd card (which would be slow), a USB storage device, or you could mount the 3.5" storage array from the helios.  I would be curious to try this to see if this avoids the "no space left on device" errors I was getting when un-tarring the root filesystem back onto the new btrfs filesystem.

 

I gave this method a try, and it worked successfully without any of the btrfs "no space left on device" errors.  The process was largely the same, except instead of the eMMC device being /dev/sdb, it was /dev/mmcblk1.  I also mounted the storage array and used that to hold the tarball of the old root filesystem.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines