Docker fails on Tinkerboard

[z3r0lvld]

Keep getting the error below when trying to run docker containers. I have a first gen Tinkerboard with Armbian on a SSD. Seen some post's about some error with cgroups v2, and using kernels below 5.x

 

sudo docker run -d --name hello hello-world

b22653867370a24ab900fc077f50fc6b05bb6eb986906caee88250fc9e9f5433 docker: Error response from daemon:
failed to create shim task: OCI runtime create failed: runc create failed: unable to start container
process: error during container init: error setting cgroup config for procHooks process:
bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown.

 

docker --version
Docker version 20.10.21, build baeda1f

 

uname -a
Linux tinkerboard 5.15.74-rockchip #22.08.6 SMP PREEMPT Tue Oct 18 06:40:37 UTC 2022 armv7l armv7l armv7l GNU/Linux

 

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy

 

[Igor]

Docker works on official builds:

 

 _____ _       _             _                         _ 
|_   _(_)_ __ | | _____ _ __| |__   ___   __ _ _ __ __| |
  | | | | '_ \| |/ / _ \ '__| '_ \ / _ \ / _` | '__/ _` |
  | | | | | | |   <  __/ |  | |_) | (_) | (_| | | | (_| |
  |_| |_|_| |_|_|\_\___|_|  |_.__/ \___/ \__,_|_|  \__,_|
                                                         
Welcome to Armbian 22.11.0-trunk.0121 Jammy with Linux 5.15.80-rockchip

No end-user support: untested automated build

System load:   5%           	Up time:       5 min	
Memory usage:  6% of 1.96G  	IP:	       172.17.0.1 10.0.30.119
CPU temp:      48°C           	Usage of /:    85% of 15G    	
RX today:      85.4 MiB  	

[ 0 security updates available, 2 updates total: apt upgrade ]
Last check: 2022-11-28 00:55

[ General system configuration (beta): armbian-config ]

Last login: Mon Nov 28 00:47:24 2022 from 10.0.10.12
tinkerboard:~:# sudo docker run -d --name hello hello-world                 
52aa4db25eb9c143570665ba5f3f38f9c3d9e1fc7ae8757b7664c6a9c9d4b76d
tinkerboard:~:# docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm32v7)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

tinkerboard:~:#

https://github.com/armbian/build/pull/3830
If you run old images, or some in between, it might fail.

[Walden]

$ uname -a
Linux tinkerboard 5.15.80-rockchip #22.11.1 SMP PREEMPT Wed Nov 30 11:05:19 UTC 2022 armv7l GNU/Linux

 

I followed the guide on Docker's website to install Docker and attempted to run the hello-world container.

 

$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
9b157615502d: Pull complete 
Digest: sha256:faa03e786c97f07ef34423fccceeec2398ec8a5759259f94d99078f264e9d7af
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown.

 

CONFIG_CGROUP is enabled https://github.com/armbian/build/blob/f99dedc47a1f77f487f9a140a26f4b972fe4eac8/config/kernel/linux-rockchip-current.config#L153

CONFIG_BPF is enabled https://github.com/armbian/build/blob/f99dedc47a1f77f487f9a140a26f4b972fe4eac8/config/kernel/linux-rockchip-current.config#L92

CONFIG_BPF_SYSCALL is enabled https://github.com/armbian/build/blob/f99dedc47a1f77f487f9a140a26f4b972fe4eac8/config/kernel/linux-rockchip-current.config#L98

 

I'm not sure if there's a related config that's not enabled or what's causing runc to fail the initialization.

 

Documentatoin about BPF_CGROUP_DEVICE found here https://docs.kernel.org/admin-guide/cgroup-v2.html?highlight=bpf_cgroup_device (ctrl+f to find the specific mentions).

 

Please let me if there is any more information I can provide or if there is any information you can provide to help me tackle this problem. Any help is appreciated!

Edited December 2, 2022 by Walden
Typo

[schwar3kat]

On 12/2/2022 at 8:23 PM, Walden said:

Any help is appreciated!

I don't know if this will work and I can't test because I don't have your board.  

From a different OS same error:
https://my-take-on.tech/2021/05/07/fix-docker-cgroup-errors-after-systemd-248-update/
The write-up indicates that might only affect old docker containers, so the possible fix below may only work for those containers.

To do the same thing in Armbian:
In /boot/armbianEnv.txt    add a line (or if extraargs already exists add an extra argument separated by a comma)
extraargs="systemd.unified_cgroup_hierarchy=0"

Reboot and try again. Good luck.

Let us know the outcome either way.

Edit:  great this worked for Walden.

[Walden]

On 12/2/2022 at 3:20 AM, schwar3kat said:

Let us know the outcome either way.

IT WORKED!!!

 

Thank you soooooooooooo much! I have spent so many hours determined to get this working and have given up so many times. I can finally proceed to actually use this thing the way I wanted!

 

You have no idea how happy I am right now...

[schwar3kat]

1 hour ago, Walden said:

You have no idea how happy I am right now...

  you're welcome.

[alex9534126]

Worked for me as well. Thanks!