alex osadchyy Posted October 24, 2018 Share Posted October 24, 2018 After apt-get upgrade I got 4.14.18-sunx installed on my organge pi+. docker daemon won't start. I tried removing and reinstalling it. tried dev, next and nightly builds to no success. docker gives an error that no overlay fs is available in kernel. tried ARMBIAN 5.60 stable and 5.64 nightly >apt-get install docker-ce fails with Setting up docker-ce (18.06.1~ce~3-0~debian) ... Job for docker.service failed. See 'systemctl status docker.service' and 'journa lctl -xn' for details. invoke-rc.d: initscript docker, action "start" failed. dpkg: error processing package docker-ce (--configure): subprocess installed post-installation script returned error exit status 1 >dockerd gives these errors INFO[2018-10-24T00:57:15.443736396-07:00] pickfirstBalancer: HandleSubConnStateChange: 0x1510c6f0, CONNECTING module=grpc INFO[2018-10-24T00:57:15.445367628-07:00] pickfirstBalancer: HandleSubConnStateChange: 0x1510c6f0, READY module=grpc ERRO[2018-10-24T00:57:15.455860786-07:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded. storage-driver=overlay2 ERRO[2018-10-24T00:57:15.465100115-07:00] AUFS was not found in /proc/filesystems storage-driver=aufs ERRO[2018-10-24T00:57:15.474183657-07:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded. storage-driver=overlay INFO[2018-10-24T00:57:15.489862792-07:00] Graph migration to content-addressability took 0.00 seconds WARN[2018-10-24T00:57:15.491169328-07:00] Your kernel does not support cgroup rt period WARN[2018-10-24T00:57:15.491297657-07:00] Your kernel does not support cgroup rt runtime INFO[2018-10-24T00:57:15.493600781-07:00] parsed scheme: "unix" module=grpc INFO[2018-10-24T00:57:15.493727068-07:00] scheme "unix" not registered, fallback to default scheme module=grpc INFO[2018-10-24T00:57:15.494017724-07:00] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0 <nil>}] module=grpc INFO[2018-10-24T00:57:15.494115054-07:00] ClientConn switching balancer to "pick_first" module=grpc INFO[2018-10-24T00:57:15.494338795-07:00] pickfirstBalancer: HandleSubConnStateChange: 0x1510d0a0, CONNECTING module=grpc INFO[2018-10-24T00:57:15.495401173-07:00] pickfirstBalancer: HandleSubConnStateChange: 0x1510d0a0, READY module=grpc INFO[2018-10-24T00:57:15.495748369-07:00] Loading containers: start. WARN[2018-10-24T00:57:15.506608222-07:00] Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin' modprobe: WARNING: Module bridge not found in directory /lib/modules/4.14.18-sunxi modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin' modprobe: WARNING: Module br_netfilter not found in directory /lib/modules/4.14.18-sunxi , error: exit status 1 WARN[2018-10-24T00:57:15.515838384-07:00] Running modprobe nf_nat failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin' modprobe: WARNING: Module nf_nat not found in directory /lib/modules/4.14.18-sunxi`, error: exit status 1 WARN[2018-10-24T00:57:15.524934760-07:00] Running modprobe xt_conntrack failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin' modprobe: WARNING: Module xt_conntrack not found in directory /lib/modules/4.14.18-sunxi`, error: exit status 1 Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin' modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.14.18-sunxi iptables v1.4.21: can't initialize iptables table `nat': Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded. (exit status 3) Link to comment Share on other sites More sharing options...
Igor Posted October 24, 2018 Share Posted October 24, 2018 Where are your logs? armbianmonitor -u Docker works on Armbian, download from here: https://www.armbian.com/orange-pi-pc-plus/ Link to comment Share on other sites More sharing options...
alex osadchyy Posted October 25, 2018 Author Share Posted October 25, 2018 Here are the logs http://ix.io/1pXO I will try to download stretch and re-image the system Link to comment Share on other sites More sharing options...
alex osadchyy Posted October 25, 2018 Author Share Posted October 25, 2018 An update. Re-imaging worked fine. Must be something with the upgrade path. I was upgrading from 3.4 to 4.11 then to 4.14. The last one broke docker support. Installing from the fresh image = docker works. I guess the topic can be closed. 1 Link to comment Share on other sites More sharing options...
Alex Einz Posted October 30, 2018 Share Posted October 30, 2018 (edited) I have exactly same problem, did an upgrade and it broke my docker... is there anything I can do to roll back or restore it minimally The system already has quite a bit of configuration and services running.. reimaging would be highly problematic.... logs located here http://ix.io/1qqI Edited October 30, 2018 by Alex Einz added logs Link to comment Share on other sites More sharing options...
Alex Einz Posted October 31, 2018 Share Posted October 31, 2018 This is the output of command I ran to upgrade Start-Date: 2018-10-29 13:24:42 Commandline: apt upgrade Upgrade: linux-u-boot-bananapi-next:armhf (5.38, 5.60), htop:armhf (2.0.2-1, 2.1.1-3), armbian-firmware:armhf (5.45, 5.60), armbian-tools-stretch:armhf (5.58, 5.63), sunxi-tools:armhf (1.4.2-2~armbian5.38+1, 1.4.2-2~armbian5.60+1), armbian-config:armhf (5.58, 5.64), hostapd:armhf (2:2.6-4~armbian5.38+1, 3:2.6-99~armbian5.60+1), linux-headers-next-sunxi:armhf (5.41, 5.60), linux-image-next-sunxi:armhf (5.41, 5.60), linux-dtb-next-sunxi:armhf (5.41, 5.60), docker-ce:armhf (18.06.0~ce~3-0~debian, 18.06.1~ce~3-0~debian), tzdata:armhf (2018e-0+deb9u1, 2018g-0+deb9u1), linux-stretch-root-next-bananapi:armhf (5.38, 5.60) now my docker wont start with the following error: ERRO[2018-10-31T13:46:17.797748922+09:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded. storage-driver=overlay2 ERRO[2018-10-31T13:46:17.798372540+09:00] [graphdriver] prior storage driver overlay2 failed: driver not supported Error starting daemon: error initializing graphdriver: driver not supported Does it mean , I need to recompile somehow the armbian to include overlay ? The fix above was to reimage... but I would really rather avoid that because not all the system is on docker and it would cause really massive headache to restore.... Link to comment Share on other sites More sharing options...
Igor Posted October 31, 2018 Share Posted October 31, 2018 1 hour ago, Alex Einz said: Does it mean , I need to recompile somehow the armbian to include overlay ? This means nothing, in supplied logs everything looks fine. Overlayfs support is enabled as module (and all other dependencies) and you will need to debug Docker in details why it doesn't work. Is overlayfs module loaded? Make a new image from sources and see if everything works there. If not, file an issue that Docker doesn't work. I doubt we can check it in less than one month. Link to comment Share on other sites More sharing options...
Alex Einz Posted October 31, 2018 Share Posted October 31, 2018 On another look I also see that even though the kernel now is 4.14.18 Quote AutomationServer:/lib/modules# uname -a Linux AutomationServer 4.14.18-sunxi #24 SMP Fri Feb 9 16:24:32 CET 2018 armv7l GNU/Linux There is no such folder under /lib/modules and I get this trying to modprobe user@AutomationServer:/lib/modules# modprobe overlayfs modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.18-sunxi/modules.dep.bin' modprobe: FATAL: Module overlayfs not found in directory /lib/modules/4.14.18-sunxi Link to comment Share on other sites More sharing options...
Igor Posted October 31, 2018 Share Posted October 31, 2018 17 minutes ago, Alex Einz said: There is no such folder under /lib/modules and I get this trying to modprobe Ahaa. There was a problem with a kernel upgrade. You must have removed /boot mount ... copy content from /boot to SDcards /boot and reboot. Keep SDcards /boot mounted under /boot ... Link to comment Share on other sites More sharing options...
Alex Einz Posted October 31, 2018 Share Posted October 31, 2018 Ah! Thank you.... yes , I use SSD for rootfs but boot indeed happens from the SDcard ....... Do you have suggestions on what I should copy to SD card for smooth boot ? Would it work if I move just the uInitrd-4.14.78-sunxi file or do I need to redo the whole SD to SSD process? Best regards Link to comment Share on other sites More sharing options...
Igor Posted October 31, 2018 Share Posted October 31, 2018 Read my post again. Everything from /bootWrote on mobile Link to comment Share on other sites More sharing options...
Alex Einz Posted November 1, 2018 Share Posted November 1, 2018 figured it out and sorted the boot from ssd to be auto mounted in fstab to prevent future disasters. marked kernel frozen too. docker back up and running Thank you for advice and patience. 1 Link to comment Share on other sites More sharing options...
Recommended Posts