Jump to content

overlay fs/docker broken on 4.14.18-sunx


alex osadchyy

Recommended Posts

After apt-get upgrade I got 4.14.18-sunx installed on my organge pi+. docker daemon won't start.  I tried removing and reinstalling it. tried dev, next and nightly builds to no success.  docker gives an error that no overlay fs is available in kernel.

 

tried ARMBIAN 5.60 stable and 5.64 nightly 

 

>apt-get install docker-ce fails with 

Setting up docker-ce (18.06.1~ce~3-0~debian) ...
Job for docker.service failed. See 'systemctl status docker.service' and 'journa                                                                                lctl -xn' for details.
invoke-rc.d: initscript docker, action "start" failed.
dpkg: error processing package docker-ce (--configure):
 subprocess installed post-installation script returned error exit status 1

 

>dockerd gives these errors

INFO[2018-10-24T00:57:15.443736396-07:00] pickfirstBalancer: HandleSubConnStateChange: 0x1510c6f0, CONNECTING  module=grpc
INFO[2018-10-24T00:57:15.445367628-07:00] pickfirstBalancer: HandleSubConnStateChange: 0x1510c6f0, READY  module=grpc
ERRO[2018-10-24T00:57:15.455860786-07:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.  storage-driver=overlay2
ERRO[2018-10-24T00:57:15.465100115-07:00] AUFS was not found in /proc/filesystems       storage-driver=aufs
ERRO[2018-10-24T00:57:15.474183657-07:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.  storage-driver=overlay
INFO[2018-10-24T00:57:15.489862792-07:00] Graph migration to content-addressability took 0.00 seconds
WARN[2018-10-24T00:57:15.491169328-07:00] Your kernel does not support cgroup rt period
WARN[2018-10-24T00:57:15.491297657-07:00] Your kernel does not support cgroup rt runtime
INFO[2018-10-24T00:57:15.493600781-07:00] parsed scheme: "unix"                         module=grpc
INFO[2018-10-24T00:57:15.493727068-07:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2018-10-24T00:57:15.494017724-07:00] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0  <nil>}]  module=grpc
INFO[2018-10-24T00:57:15.494115054-07:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2018-10-24T00:57:15.494338795-07:00] pickfirstBalancer: HandleSubConnStateChange: 0x1510d0a0, CONNECTING  module=grpc
INFO[2018-10-24T00:57:15.495401173-07:00] pickfirstBalancer: HandleSubConnStateChange: 0x1510d0a0, READY  module=grpc
INFO[2018-10-24T00:57:15.495748369-07:00] Loading containers: start.
WARN[2018-10-24T00:57:15.506608222-07:00] Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin'
modprobe: WARNING: Module bridge not found in directory /lib/modules/4.14.18-sunxi
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin'
modprobe: WARNING: Module br_netfilter not found in directory /lib/modules/4.14.18-sunxi
, error: exit status 1
WARN[2018-10-24T00:57:15.515838384-07:00] Running modprobe nf_nat failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin'
modprobe: WARNING: Module nf_nat not found in directory /lib/modules/4.14.18-sunxi`, error: exit status 1
WARN[2018-10-24T00:57:15.524934760-07:00] Running modprobe xt_conntrack failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin'
modprobe: WARNING: Module xt_conntrack not found in directory /lib/modules/4.14.18-sunxi`, error: exit status 1
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin'
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.14.18-sunxi
iptables v1.4.21: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)
 

 

Link to comment
Share on other sites

I have exactly same problem, did an upgrade and it broke my docker... is there anything I can do to roll back or restore it minimally

The system already has quite a bit of configuration and services running.. reimaging would be highly problematic....

 

logs located here http://ix.io/1qqI

Edited by Alex Einz
added logs
Link to comment
Share on other sites

This is the output of command I ran to upgrade

Start-Date: 2018-10-29  13:24:42
Commandline: apt upgrade
Upgrade: linux-u-boot-bananapi-next:armhf (5.38, 5.60),

htop:armhf (2.0.2-1, 2.1.1-3),

armbian-firmware:armhf (5.45, 5.60),

armbian-tools-stretch:armhf (5.58, 5.63),

sunxi-tools:armhf (1.4.2-2~armbian5.38+1, 1.4.2-2~armbian5.60+1),

armbian-config:armhf (5.58, 5.64),

hostapd:armhf (2:2.6-4~armbian5.38+1, 3:2.6-99~armbian5.60+1),

linux-headers-next-sunxi:armhf (5.41, 5.60),

linux-image-next-sunxi:armhf (5.41, 5.60),

linux-dtb-next-sunxi:armhf (5.41, 5.60),

docker-ce:armhf (18.06.0~ce~3-0~debian, 18.06.1~ce~3-0~debian),

tzdata:armhf (2018e-0+deb9u1, 2018g-0+deb9u1),

linux-stretch-root-next-bananapi:armhf (5.38, 5.60)
 

 

now my docker wont start with the following error:

ERRO[2018-10-31T13:46:17.797748922+09:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded. storage-driver=overlay2

ERRO[2018-10-31T13:46:17.798372540+09:00] [graphdriver] prior storage driver overlay2 failed: driver not supported

Error starting daemon: error initializing graphdriver: driver not supported

 

 

Does it mean , I need to recompile somehow the armbian to include overlay ? The fix above was to reimage... but I would really rather avoid that because not all the system is on docker and it would cause really massive headache to restore....

 

 

Link to comment
Share on other sites

1 hour ago, Alex Einz said:

Does it mean , I need to recompile somehow the armbian to include overlay ?

 

This means nothing, in supplied logs everything looks fine. Overlayfs support is enabled as module (and all other dependencies) and you will need to debug Docker in details why it doesn't work. Is overlayfs module loaded? Make a new image from sources and see if everything works there. If not, file an issue that Docker doesn't work. I doubt we can check it in less than one month.

Link to comment
Share on other sites

On another look I also see that even though the kernel now is 4.14.18

Quote

 

AutomationServer:/lib/modules# uname -a

Linux AutomationServer 4.14.18-sunxi #24 SMP Fri Feb 9 16:24:32 CET 2018 armv7l GNU/Linux

 

 

There is no such folder under /lib/modules and I get this trying to modprobe 

user@AutomationServer:/lib/modules# modprobe overlayfs

modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin'

modprobe: FATAL: Module overlayfs not found in directory /lib/modules/4.14.18-sunxi

 

 

 

 

Link to comment
Share on other sites

17 minutes ago, Alex Einz said:

There is no such folder under /lib/modules and I get this trying to modprobe


Ahaa. There was a problem with a kernel upgrade. You must have removed /boot mount ... copy content from /boot to SDcards /boot and reboot. Keep  SDcards /boot mounted under /boot ... 

Link to comment
Share on other sites

Ah! Thank you.... yes , I use SSD for rootfs but boot indeed happens from the SDcard .......

Do you have suggestions on what I should copy to SD card for smooth boot ?

Would it work if I move just the uInitrd-4.14.78-sunxi file or do I need to redo the whole SD to SSD process?

 

Best regards

 

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines