lvm device not sync'd on shutdown: failure in blkdeactivate script

[douardda]

On my odroidxu4 with armbian stretch, I had hard time rebooting without having one disk of the 4-drives RAID5 lvm device being kick out of the  array on next boot.

 

The logs suggest the problem may come from the /sbin/blkdeactivate script, as seen in these logs during shutdown:

 

Dec 19 13:04:59 nas2 blkdeactivate[12440]: Deactivating block devices:
Dec 19 13:04:59 nas2 systemd[1]: Unmounted Mount shared folder data to /sharedfolders/data.
Dec 19 13:04:59 nas2 systemd[1]: watchdog.service: Succeeded.
Dec 19 13:04:59 nas2 systemd[1]: Stopped watchdog daemon.
Dec 19 13:04:59 nas2 blkdeactivate[12440]: /sbin/blkdeactivate: line 345: /bin/sort: No such file or directory
Dec 19 13:04:59 nas2 folder2ram[12443]: stop /var/log
Dec 19 13:05:00 nas2 blkdeactivate[12440]:   [MD]: deactivating part device md0p1... cat: /sys/block/md0p1/md/sync_action: No such file or directory
Dec 19 13:05:00 nas2 blkdeactivate[12440]:   [MD]: deactivating raid5 device md0... skipping

 

 

Followed by the following, during the boot:

 

Dec 19 13:05:52 nas2 kernel: [   15.408880]  sdb: sdb1
Dec 19 13:05:52 nas2 kernel: [   15.432864]  sdc: sdc1
Dec 19 13:05:52 nas2 kernel: [   15.434408]  sdd: sdd1
Dec 19 13:05:52 nas2 kernel: [   15.435467] sd 1:0:0:0: [sdb] Attached SCSI disk
Dec 19 13:05:52 nas2 kernel: [   15.436131]  sde: sde1
Dec 19 13:05:52 nas2 kernel: [   15.445062] sd 1:0:0:1: [sdc] Attached SCSI disk
Dec 19 13:05:52 nas2 kernel: [   15.445518] sd 1:0:0:2: [sdd] Attached SCSI disk
Dec 19 13:05:52 nas2 kernel: [   15.447638] sd 1:0:0:3: [sde] Attached SCSI disk
Dec 19 13:05:52 nas2 kernel: [   15.912849] md: kicking non-fresh sdb1 from array!

 

 

The problem is, on debian, the sort tool in installed in `/usr/bin/sort` and not `/bin/sort`. On the odroid xu4, there is no `/bin/sort` command. Oddly, on my laptops (bullseye and buster), there is a hardlink at /bin/sort .

 

For now, I've add a hardlink  by hand, I'm waiting for the scrub of the lvm array before attempting a reboot and see if it fixes the pb by properly syncing the devices.

 

I've also noted that blkdeactivate script attempt to access to `/sys/block/md0p1/md/sync_action`, which is not the correct path either (it's on `/sys/block/md0/md/sync_action`), not sure if the previous fix (sort) would change that, we'll see on next reboot.

 

David

 

[douardda]

Got the info that on stock debian, this /usr/bin vs /bin mess is normally handled by debootstrap, but the `usrmerge` package can be installed to "fix" an existing debian system.

[Igor]

7 minutes ago, douardda said:

but the `usrmerge` package can be installed to "fix" an existing debian system.

If we add "usrmerge" package to our package list, this problem will be solved?

Probably this problem is affected only on older builds?

[douardda]

not sure if adding the usrmerge package would be a proper fix for this. I'll try to make some more tests on another board I have around.

[douardda]

Ok it seems currently built images (debian buster ones) are fine: they have `/bin`, `/lib`, and `/sbin` symlinks to `/usr/xxx` so it should be fine. I'll try to reinstall my odroid xu4 from a fresh disk image to check mdadm volumes are properly stopped.