Blade Doyle Posted January 5, 2016 Posted January 5, 2016 Hi Armbians, (If this question is already asked/answered please help me find the link.) I need the "ceph" and "rbd" kernel modules for my orange and banana pi. I am starting with the most recent images released: Armbian_4.5_xxxx_Debian_jessie_4.2.3. What are the minimum required steps to build the additional kernel modules? Where is the source tree for that kernel build (4.3.3-sunxi)? I would prefer to build directly on the PI (as opposed to x-compile) but either is ok. I tried a few things, including git clone of Igors "lib" project with build scripts. Runnig that pulled a kernel source tree: "sources/linux-vanilla/v4.3.3" which I am able to use to build my kernel modules, but they fail to load with: "modprobe: ERROR: could not insert 'rbd': Exec format error". Please enlighten me. Blade.
Igor Posted January 5, 2016 Posted January 5, 2016 I think the best / most simple way is to change to your custom kernel and install it over stock. Alter revision number to 9999 that it won't be overwritten with stock while apt-get update / upgrade. The purpose of "lib" tools are exactly that. This kernel module copy method usually doesn't work between different kernels.
Blade Doyle Posted January 5, 2016 Author Posted January 5, 2016 Thanks, that worked. Looks like the versions were not exactly the same after all, which is likely why the modules were not loading. "Unpacking linux-image-next-sunxi (4.82) over (4.81) ..." So now I have what I need! root@node4:~# lsmodModule Size Used byceph 157660 0rbd 48507 0libceph 129427 2 rbd,ceph My banana pi pro, and 2 orange pi plus are now able to join the 4 raspberry pi2 in my 'mixed fruit pi Kubernetes cluster'. Awesome. Thanks for all your hard work! Blade.
Igor Posted January 5, 2016 Posted January 5, 2016 Cool! Just one add regarding kernel name, since you haven't alter the name. Next apt-get upgrade will overwritten your kernel ... one option is altering revision/name and using a custom one, the other is to share your kernel config and i'll merge it to our default.
Blade Doyle Posted January 5, 2016 Author Posted January 5, 2016 Thanks Igor. I understand about the name, and will resolve as you suggest.
Blade Doyle Posted January 5, 2016 Author Posted January 5, 2016 For the record, the config changes to generate modules needed to for Ceph and RBD filesystems are: % diff .config.old .config1408c1408,1410< # CONFIG_CEPH_LIB is not set---> CONFIG_CEPH_LIB=m> # CONFIG_CEPH_LIB_PRETTYDEBUG is not set> # CONFIG_CEPH_LIB_USE_DNS_RESOLVER is not set1574c1576< # CONFIG_BLK_DEV_RBD is not set---> CONFIG_BLK_DEV_RBD=m4549c4551,4552< # CONFIG_CEPH_FS is not set---> CONFIG_CEPH_FS=m> # CONFIG_CEPH_FS_POSIX_ACL is not set And, to get Docker to run "group-enable=memory" needs to be added to /boot/boot.cmd
Igor Posted January 6, 2016 Posted January 6, 2016 https://github.com/igorpecovnik/lib/commit/9fcc1cbea207922aa88f0e8307e543290f8e3032 You can leave your custom naming as is since next kernel will have all this inside. Hmm, what happen if we leave this on by default? group-enable=memory
Blade Doyle Posted January 6, 2016 Author Posted January 6, 2016 My bad, it should be "cgroup-enable=memory". (I missed the initial "c" in my previous post) Also, my additional research inidicates that "swapaccount=1" should also be added to Debian boot params for Docker. For some reason, for Debian systems, even though its configued in the kernel it is still necessary to enable it using a boot option. Ref: https://github.com/docker/docker/issues/396 The docker community seems to feel its a bug in Debian, so adding those to your standard distribution is probably the right thing to do.
Igor Posted January 6, 2016 Posted January 6, 2016 https://github.com/igorpecovnik/lib/commit/7be413aca5f7727272b979a3412c1c1970d8bdb5 OK, tnx
Dmitry Posted April 21, 2017 Posted April 21, 2017 Hello All, I've tried to setup Ceph on orange, but without any success (Unable to locate package ceph-osd, ceph-mon etc during running ceph-deploy install). So I suppose I need those kernel modules too? Could you please advise me on how to build and install them?? My gratitude will have no limits!
Igor Posted April 22, 2017 Posted April 22, 2017 10 hours ago, Dmitry said: I've tried to setup Ceph on orange, but without any success (Unable to locate package ceph-osd, ceph-mon etc during running ceph-deploy install) Since install can't locate some packages means you have missed something in their install instructions. We provide stock Debian / Ubuntu and headers, which are needed for possible module installation are already in place - no need to fetch them.
rcampos Posted December 8, 2019 Posted December 8, 2019 Can we also get this applied to other boards by default, I'm using Odroid-N2 and have to recompile the kernel to get the RBD module, would be great to get it as standard. # CONFIG_BLK_DEV_RBD is not set Thanks
martinayotte Posted December 8, 2019 Posted December 8, 2019 1 hour ago, rcampos said: have to recompile the kernel to get the RBD module, would be great to get it as standard. Feel free to add CONFIG_BLK_DEV_RBD=m in every config/kernel/ files and then submit a PR.
rcampos Posted December 8, 2019 Posted December 8, 2019 2 hours ago, martinayotte said: Feel free to add CONFIG_BLK_DEV_RBD=m in every config/kernel/ files and then submit a PR. Here's the pr for Ceph + RBD as a module by default on meson, meson64, odroidn2 https://github.com/armbian/build/pull/1665 2
Recommended Posts