Jump to content

Btrfs scrub causes crash.

Tonat

By Tonat
in Rockchip

 Share

Recommended Posts

Tonat

Tonat

Posted
Posted (edited)

I have the Helios with two 10T disks mounted as a BTRFS raid 1 array. This setup is stable, although I have not stressed it a lot, except for when I do a btrfs scrub.

It is using the 20.11.1 Buster image on SDcard. The disks have about 900GB of files.

A btrfs scrub causes a crash with the following kernel output. Is this a known bug?

 

  Reveal hidden contents

 

 

 

Edited by TRS-80
put long output inside code block inside spoiler
TRS-80

TRS-80

Posted
Posted

Today is your lucky day.  :D

 

Jokes aside, I have been meaning to write this for some time.  So finally I made time just now to make (my own, separate) thread about why I am not a fan of btrfs.  This has nothing to do with you personally, and I hope you don't think I am singling you out because I am not.

 

Maybe that has something to do with your issue, or maybe it doesn't.  In either case, I really do wish you the best of luck sorting it out.  Also, for what I hope are now obvious reasons, I am afraid I don't know enough about btrfs to help in any further detail.  Hopefully someone else will.

 

Cheers and happy Friday!  :beer:

m11k

m11k

Posted
Posted

Huh, that's strange.  I've been using btrfs on 5 14TB drives in a RAID-1.  I've done several scrubs without issue, including one where I purposefully corrupted data on one of the drives while it was offline, to test the ability to recover from this.  I think I might have done that first test on the 5.8 kernel, but I've definitely had the scrub run from cron since upgrading to 5.9 (I'm currently on 5.9.11).

 

The only difference in my scenario is that my OS is installed to the eMMC, but I can't imagine how that would change anything.

 

I have also converted the root filesystem to btrfs.  I haven't done a scrub on it (there's not much point in a scrub without a RAID configuration), but it's been working fine for a couple of weeks.

 

Any btrfs messages in the kernel log?  Try running 'dmesg | grep -i btrfs'.  What are your btrfs mount options?  Here is what I have:

  

$ cat /etc/fstab
...
/dev/disk/by-label/helios64_btrfs_raid1         /srv/dev-disk-by-label-helios64_btrfs_raid1     btrfs   defaults,nofail,noatime,nodiratime,compress=zstd        0 2

 

Some info about my btrfs filesystem:

  

$ lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda            8:0    0 12.8T  0 disk 
└─sda1         8:1    0 12.8T  0 part /srv/dev-disk-by-label-helios64_btrfs_raid1
sdb            8:16   0 12.8T  0 disk 
└─sdb1         8:17   0 12.8T  0 part 
sdc            8:32   0 12.8T  0 disk 
└─sdc1         8:33   0 12.8T  0 part 
sdd            8:48   0 12.8T  0 disk 
└─sdd1         8:49   0 12.8T  0 part 
sde            8:64   0 12.8T  0 disk 
└─sde1         8:65   0 12.8T  0 part 
mmcblk1      179:0    0 14.6G  0 disk 
├─mmcblk1p1  179:1    0  512M  0 part /boot
└─mmcblk1p2  179:2    0   14G  0 part /mnt/btr_pool
mmcblk1boot0 179:32   0    4M  1 disk 
mmcblk1boot1 179:64   0    4M  1 disk

  

$ sudo btrfs fi usage /srv/dev-disk-by-label-helios64_btrfs_raid1/
Overall:
    Device size:                  63.67TiB
    Device allocated:             18.19TiB
    Device unallocated:           45.47TiB
    Device missing:                  0.00B
    Used:                         18.19TiB
    Free (estimated):             22.74TiB      (min: 22.74TiB)
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,RAID1: Size:9.08TiB, Used:9.08TiB (99.97%)
   /dev/sda1       3.63TiB
   /dev/sdb1       3.63TiB
   /dev/sdc1       3.63TiB
   /dev/sdd1       3.64TiB
   /dev/sde1       3.63TiB

Metadata,RAID1: Size:12.00GiB, Used:11.61GiB (96.75%)
   /dev/sda1       6.00GiB
   /dev/sdb1       7.00GiB
   /dev/sdc1       4.00GiB
   /dev/sdd1       3.00GiB
   /dev/sde1       4.00GiB

System,RAID1: Size:8.00MiB, Used:1.25MiB (15.62%)
   /dev/sdd1       8.00MiB
   /dev/sde1       8.00MiB

Unallocated:
   /dev/sda1       9.09TiB
   /dev/sdb1       9.09TiB
   /dev/sdc1       9.09TiB
   /dev/sdd1       9.09TiB
   /dev/sde1       9.09TiB

  

$ dmesg | grep -i btrfs
[    0.000000] Kernel command line: root=UUID=db73782d-ba22-47ed-bc7d-60deedc591ec rootwait rootfstype=btrfs earlycon console=ttyS2,1500000 console=tty1 consoleblank=0 loglevel=7 ubootpart=794023b5-01 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u rootflags=subvol=@  cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1
[    3.297371] Btrfs loaded, crc32c=crc32c-generic
[   14.920125] BTRFS: device label helios64_btrfs_raid1 devid 5 transid 7810 /dev/sde1 scanned by btrfs (362)
[   14.922109] BTRFS: device label helios64_btrfs_raid1 devid 4 transid 7810 /dev/sdd1 scanned by btrfs (362)
[   14.924064] BTRFS: device label helios64_btrfs_raid1 devid 3 transid 7810 /dev/sdc1 scanned by btrfs (362)
[   14.926025] BTRFS: device label helios64_btrfs_raid1 devid 2 transid 7810 /dev/sdb1 scanned by btrfs (362)
[   14.927975] BTRFS: device label helios64_btrfs_raid1 devid 1 transid 7810 /dev/sda1 scanned by btrfs (362)
[   14.929590] BTRFS: device fsid db73782d-ba22-47ed-bc7d-60deedc591ec devid 1 transid 10019 /dev/mmcblk1p2 scanned by btrfs (362)
[   15.002086] BTRFS info (device mmcblk1p2): disk space caching is enabled
[   15.002715] BTRFS info (device mmcblk1p2): has skinny extents
[   15.013757] BTRFS info (device mmcblk1p2): enabling ssd optimizations
[   16.477703] BTRFS info (device mmcblk1p2): use zstd compression, level 3
[   16.477726] BTRFS info (device mmcblk1p2): disk space caching is enabled
[   18.389379] BTRFS info (device sda1): use zstd compression, level 3
[   18.389985] BTRFS info (device sda1): disk space caching is enabled
[   18.390539] BTRFS info (device sda1): has skinny extents

  

$ sudo btrfs scrub status /srv/dev-disk-by-label-helios64_btrfs_raid1/
UUID:             41ded890-363f-4b35-9b75-b2048662bb38
Scrub started:    Tue Dec  1 23:15:01 2020
Status:           finished
Duration:         7:01:13
Total to scrub:   18.19TiB
Rate:             748.36MiB/s
Error summary:    no errors found

 

The only other difference is that I installed 'btrfs-progs' from the 'buster-backports' repository so that I would get the version matching the 5.9 kernel, rather than the 4.19 version.  This was only a relatively recent change for me though and I'm pretty sure my initial testing was with the 4.19 version of btrfs-progs.  It shouldn't matter, since the scrub is done by the kernel, and 'btrfs-progs' simply tells the kernel to start it.

TRS-80

TRS-80

Posted
Posted

In the meantime I received a quite interesting reply on my above linked thread from none other than tkaiser himself.  It's quite long and full of lots of good info, but the TL;DR is that I am re-thinking my position on btrfs now, most especially on ARM (while overall, and on x86 especially I maintain my position).  But maybe you guys know all of that already and that's why you use btrfs in the first place.  :D

 

However if I am going to shit up someone else's thread, especially with (apparently wrong) information, the very least I feel I should do is return and issue a retraction / update.

 

Cheers!

2
Tonat

Tonat

Posted
  • Author
Posted

Compared to the system of m11k, my Helios64 has two 10T Ironwolf disks in slots 1 & 2, other slots are empty.

The mount options are the same except I didn't use nodiratime (I think it is redundant because noatime is used).
Apart from the btrfs scrub I have not had any problems at all.

 

As an experiment I tried the following to see if it would crash again

  1. Do a scrub
  2. Simultaneously create two compressed tar files of the data on the disks (trying to produce significant processor and disk activity)
  3. After about 6hours I stopped the tars and did another scrub.
  4. Simultaneously un-tar the files in step 2 (which finished with expected error as the files were incomplete)
  5. Do another scrub.

All the above worked correctly.

 

This makes me think the crash originally reported was a one off event (but it would be good to know if there is an underlying cause).

I will let you know if I have any more problems.

1
Guest
This topic is now closed to further replies.
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept