How to fix nfs-server.service failing to start on boot, due to slow MergerFS folder being exported

[NZJon]

 

Howdy.

 

I m running a Helios64 on Armbian 21.02.1 Buster with Linux 5.10.63-rockchip64. I realise this is somewhat old, but it works for me.

 

I have been using MergerFS to present a big "vritual" storage area on my server:

 

jon@OXYGEN:~$ more /etc/fstab
UUID=a79a14c0-3cf4-4fb9-a6c6-838571351371       /                ext4   defaults,noatime,nodiratime,commit=600,errors=remount-ro    0    1
tmpfs                                           /tmp             tmpfs  defaults,nosuid                                             0    0

# "Disk 2" = /dev/sda3 = 8d360e1c-1908-4d8d-8289-2b1c83604950
UUID=8d360e1c-1908-4d8d-8289-2b1c83604950       /srv/mnt/disk2   btrfs  defaults,noatime,nodiratime,nofail,subvol=@data             0    2

# "Disk 3" = /dev/sdb2 = eceee45f-5be5-48ba-a928-8f61686923b0
UUID=eceee45f-5be5-48ba-a928-8f61686923b0       /srv/mnt/disk3   btrfs  defaults,noatime,nodiratime,nofail,subvol=@data             0    2

# "Disk 4" = /dev/sdc3 = 8ba04a0d-92eb-44e5-a4f0-99ebe2431b22
UUID=8ba04a0d-92eb-44e5-a4f0-99ebe2431b22       /srv/mnt/disk4   btrfs  defaults,noatime,nodiratime,nofail,subvol=@data             0    2

# "Disk 5" = /dev/sdd3 = c3f0e61b-67af-4b35-acb8-e6698f8c9f64
UUID=c3f0e61b-67af-4b35-acb8-e6698f8c9f64       /srv/mnt/disk5   btrfs  defaults,noatime,nodiratime,nofail,subvol=@data             0    2


# MergerFS - aggregate the mounted disks together
/srv/mnt/disk2:/srv/mnt/disk3:/srv/mnt/disk4:/srv/mnt/disk5    /srv/mnt/virtual    fuse.mergerfs   defaults,allow_other,use_ino,fsname=/srv/mnt/virtual   0    0

 

I have recently begun exporting this virtual disk via nfs. Nice! It relly is easy to set up, and it does "just work":

 

jon@OXYGEN:~$ more /etc/exports
/srv/mnt/virtual/oxygen/media 192.168.2.0/24(rw,sync,insecure,no_root_squash,no_subtree_check,nohide,fsid=0)

 

But, when my server is rebooted, the nfs-server.service fails to start:

 

 _   _      _ _            __   _  _
| | | | ___| (_) ___  ___ / /_ | || |
| |_| |/ _ \ | |/ _ \/ __| '_ \| || |_
|  _  |  __/ | | (_) \__ \ (_) |__   _|
|_| |_|\___|_|_|\___/|___/\___/   |_|

Welcome to Armbian 21.02.1 Buster with Linux 5.10.63-rockchip64

No end-user support: work in progress

System load:   3%               Up time:       9 min
Memory usage:  17% of 3.77G     IP:            172.19.0.1 172.22.0.1 172.21.0.1 192.168.2.105
CPU temp:      43°C             Usage of /:    41% of 15G

[ General system configuration (beta): armbian-config ]

Last login: Thu May 23 16:18:16 2024 from 1975:1976:1977:e00:dead:beef:0000:1111
jon@OXYGEN:~$ sudo service nfs-kernel-server status
[sudo] password for jon:
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2024-05-25 15:36:31 NZST; 10min ago
  Process: 1210 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=1/FAILURE)
  Process: 1218 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
  Process: 1219 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)

May 25 15:36:31 OXYGEN systemd[1]: Starting NFS server and services...
May 25 15:36:31 OXYGEN exportfs[1210]: exportfs: Failed to stat /srv/mnt/virtual/oxygen/media: No such file or directory
May 25 15:36:31 OXYGEN systemd[1]: nfs-server.service: Control process exited, code=exited, status=1/FAILURE
May 25 15:36:31 OXYGEN systemd[1]: nfs-server.service: Failed with result 'exit-code'.
May 25 15:36:31 OXYGEN systemd[1]: Stopped NFS server and services.

 

Running "$ sudo service nfs-kernel-server start" manually, however, gets it going:

 

jon@OXYGEN:~$ sudo service nfs-kernel-server start
jon@OXYGEN:~$ sudo service nfs-kernel-server status
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2024-05-25 16:03:04 NZST; 1s ago
  Process: 7428 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
  Process: 7429 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
 Main PID: 7429 (code=exited, status=0/SUCCESS)

May 25 16:03:03 OXYGEN systemd[1]: Starting NFS server and services...
May 25 16:03:04 OXYGEN systemd[1]: Started NFS server and services.

 

So, to me, it seems that the MergerFS-provided "virtual" filesystem at /srv/mnt/virtual is not ready at the time that the NFS service is starting up.

 

Does anyone know:

 

1. whether I can modify the definition of the NFS Service, within systemd, so that it starts up later in the boot process? Am I able to define a dependency on some other filesystem service that has yet to be started?

2. whether I can define something else that gets kicked off late in the boot process, which simply attempts to restart the NFS Server service, in the hope that by this late stage in the boot process the MergerFS location is "ready"?

 

Many thanks,

   Jon

 

[ebin-dev]

16 hours ago, NZJon said:

whether I can modify the definition of the NFS Service, within systemd, so that it starts up later in the boot process?

 

You probably just have to change the systemd service file nfs-server.service: in its [UNIT] section add the mergerFs service file (see also here) :

 

# nfs-server.service
# replace 'mergerfs.service' by the name of your MergerFS service file

[Unit]
After=mergerfs.service
Requires=mergerfs.service

 

 

Edited May 25 by ebin-dev

[NZJon]

You absolute legend, ebin-dev, for replying to my message 🙌.

 

I don't appear to have a MergerFS "service" as such -- the mounting, I guess, is handled by FUSE...?

 

So, I don't have a solution yet, but, I've been doing a lot of reading up on SystemD, so I'm beginning to understand how this might all hang together.

 

Cheers,

   Jon

 

[ebin-dev]

MergerFS is on github - there is a section 'systemd mount' in the ReadMe with an mergerfs.service example...

 

 

[NZJon]

Kia ora ebin-dev,

 

Just dropping by to say, again, a big thank you for helping me out. 🙌

 

I ended up writing a "dummy service" that the nfs-server depended on. I had noticed that it took about 15-20 seconds for a simple `ls -al` to produce an output on the nfs-shared location, so the dummy servce I wrote simply performs a `ls -al` and waits 5 seconds, until a successful output is produced. Then, the nfs-server service is able to STAT the file location... and it starts up properly.

 

Probably not the most elegant way of getting things to work... but at least it works for me!

 

Thanks again,

 

   Jon

 

[crosser]

FWIW: I had similar problem, and it was fixed by adding a dependency with an override file that I think should have been a default dependency in the first place:

$ cat /etc/systemd/system/proc-fs-nfsd.mount.d/override.conf 
[Unit]
After=local-fs.target

That simple!