Jump to content

ZFS on Helios64

grek

By grek
in Rockchip

 Share

Recommended Posts

grek

grek

Posted
Posted

Hey,

I think a dedicated topic for ZFS on Helios 64 is needed. Many people want to have it :)

 

As we know, and many of us playing with dev builds for helios ,it is no easy to work with ZFS 

I wrote few scripts maybe someone of You it can help in someway. 

Thanks @jbergler and @ShadowDance , I used your idea for complete it.

The problem is with playing with OMV and ZFS plugin. then dependencies remove our latest version of ZFS. I tested. everything survived but ZFS is downgraded to latest version from repo 

for example:

 

root@helios64:~# zfs --version

zfs-0.8.4-2~bpo10+1

zfs-kmod-2.0.0-rc6

root@helios64:~# uname -a

Linux helios64 5.9.11-rockchip64 #trunk.1 SMP PREEMPT Thu Nov 26 01:32:45 CET 2020 aarch64 GNU/Linux

root@helios64:~#

 

I tested it with kernel 5.9.10 and with today clean install with 5.9.11

 

First we need to have docker installed ( armbian-config -> software -> softy -> docker )

 

Create dedicated directory with Dockerfile ( we will customize ubuntu:bionic image with required libraries and gcc 10 . in next builds we can skip this step and just customize and run build-zfs.sh script) 

mkdir zfs-builder

cd zfs-builder

vi Dockerfile

  

FROM ubuntu:bionic
RUN apt update; apt install build-essential autoconf automake bison flex libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev -y; apt install software-properties-common -y; add-apt-repository ppa:ubuntu-toolchain-r/test; apt install gcc-10 g++-10 -y; update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10; update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 10

 

Build docker image for building purposes.

  

docker build --tag zfs-build-ubuntu-bionic:0.1 .

 

Create script for ZFS build

 

vi build-zfs.sh

  

#!/bin/bash

#define zfs version
zfsver="zfs-2.0.0-rc6"
#creating building directory
mkdir /tmp/zfs-builds && cd "$_"
rm -rf /tmp/zfs-builds/inside_zfs.sh
apt-get download linux-headers-current-rockchip64
git clone -b $zfsver https://github.com/openzfs/zfs.git $zfsver-$(uname -r)

#create file to execute inside container
echo "creating file to execute it inside container"
cat > /tmp/zfs-builds/inside_zfs.sh  <<EOF
#!/bin/bash
cd scratch/
dpkg -i linux-headers-current-*.deb
zfsver="zfs-2.0.0-rc6"

cd "/scratch/$zfsver-$(uname -r)"
sh autogen.sh
./configure
make -s -j$(nproc)
make deb
mkdir "/scratch/deb-$zfsver-$(uname -r)"
cp *.deb "/scratch/deb-$zfsver-$(uname -r)"
rm -rf  "/scratch/$zfsver-$(uname -r)"
exit
EOF

chmod +x /tmp/zfs-builds/inside_zfs.sh

echo ""
echo "####################"
echo "starting container.."
echo "####################"
echo ""
docker run --rm -it -v /tmp/zfs-builds:/scratch zfs-build-ubuntu-bionic:0.1 /bin/bash /scratch/inside_zfs.sh  

# Cleanup packages (if installed).
modprobe -r zfs zunicode zzstd zlua zcommon znvpair zavl icp spl
apt remove --yes zfsutils-linux zfs-zed zfs-initramfs
apt autoremove --yes
dpkg -i "/tmp/zfs-builds/deb-$zfsver-$(uname -r)"/kmod-zfs-$(uname -r)*.deb
dpkg -i "/tmp/zfs-builds/deb-$zfsver-$(uname -r)"/{libnvpair1,libuutil1,libzfs2,libzpool2,python3-pyzfs,zfs}_*.deb 

echo ""
echo "###################"
echo "building complete"
echo "###################"
echo ""

 

chmod +x build-zfs.sh

screen -L -Logfile buildlog.txt ./build-zfs-final.sh

 

 

 

 

 

4
Gavin

Gavin

Posted
Posted

I can't get zpool to automount on reboot, i.e the zfs-import service does not want to start on boot and I have to manually start it?

 

TonyMac32

TonyMac32

Posted
Posted
  On 11/26/2020 at 2:46 PM, grek said:

First we need to have docker installed ( armbian-config -> software -> softy -> docker )

Expand  

 

I did this differently, I simply used zfs-dkms and then set up my pool.  no containers or anything needed.

1
ShadowDance

ShadowDance

Posted
Posted

Good initiative on this thread grek and nice to see a full set of instructions all the way from installing Docker :D!

 

I would also echo the zfs-dkms recommendation, when possible. But at least for Buster users we need a workaround. The zfs-dkms package can't be compiled (kernel is configured with features not supported by the old(er) gcc), also the zfs-dkms version in backports is 0.8.4, so unless patches have been backported from 0.8.5, kernel 5.8-5.9 isn't supported.

FloBaoti

FloBaoti

Posted
Posted

For information, I'm able to run ZFS compiling kmod-zfs (make deb-kmod) from 0.8.5 release with Docker method. Then I install zfsutils-linux  from buster-backports and it works well.

wurmfood

wurmfood

Posted
Posted

I've gotten ZFS working pretty well following the compile instructions on their website. I just put everything in a chroot to make it cleaner. 

# Fist, install a few things we're going to need:
sudo apt install debootstrap

# Now, create our chroot
mkdir -p chroot-zfs

# This will take a while
sudo debootstrap --variant=buildd focal chroot-zfs

# mount proc, sys, and dev
sudo mount -t proc /proc chroot-zfs/proc
sudo mount --rbind /sys chroot-zfs/sys
sudo mount --rbind /dev chroot-zfs/dev

# Copy the files we need for apt. The sources.list is missing many things we'll need.
cp /etc/apt/sources.list .
cat /etc/apt/sources.list.d/armbian >> sources.list
sudo cp sources.list chroot-zfs/etc/sources.list

# Chroot in
chroot chroot-zfs /bin/bash

# Install wget and gnupg
apt install -y wget gnupg

# Add the repository key
wget -qO - http://apt.armbian.com/armbian.key | apt-key add -

# Update
apt update
apt upgrade -y

# From here, follow the instruction from ZFS

 

The only problem I ran into is the linux-headers-current-rockchip64 were for 5.9.10 and not 5.9.11. I did a cross build in a VM for Armbian to get updated headers. Once I installed those in the chroot I was able to build kmods.

 

I'm sure there's a way to include building ZFS 2.0 with the Armbian cross-compile setup, I just haven't figured out how yet.

wurmfood

wurmfood

Posted
Posted
  On 11/26/2020 at 8:00 PM, Gavin said:

I can't get zpool to automount on reboot, i.e the zfs-import service does not want to start on boot and I have to manually start it?

 

Expand  

Yes, once you have everything installed, you need to enable zfs-import-cache, zsf-import.target, zfs-mount, zfs.target, zfs-zed.

deucalion

deucalion

Posted
Posted

Just out of curiosity:

are there any specific ZFS 2.0 features you need, or are you just building it for fun / to have the latest and greatest version?

jbergler

jbergler

Posted
Posted
  On 11/27/2020 at 3:51 AM, gprovost said:

IMHO zfs-dkms would the easiest approach for most people.

Expand  

The problem here is that it‘s not possible to compile the module on Debian because of how the kernel has been built.

I reported the issue here and while I could *fix* it, it really strikes me as something the core armbian team needs to weigh in on.

One option is to use an older GCC in the build system, the other is to disable per task stack protections in the kernel - neither seem like great choices to me.

 

2
0utc45t

0utc45t

Posted
Posted

I'm new here, joined to say that zfs support for Helios64 is very needed. Have tried to get it working, but failing miserably. Need help from more knowledgeable people, so I'll be following this thread.

deucalion

deucalion

Posted
Posted
  On 11/30/2020 at 3:04 PM, 0utc45t said:

I'm new here, joined to say that zfs support for Helios64 is very needed. Have tried to get it working, but failing miserably. Need help from more knowledgeable people, so I'll be following this thread.

Expand  

 

In that case: please be aware, that the Armbian Ubuntu zfs-dkms package works well, but you mustn't upgrade to the most recent kernel release.

(The working base image would be Armbian_20.08.21_Helios64_focal_current_5.8.17.img.xz - the zfs-dkms package from the Armbian Ubuntu repository works out of the box.)

 

There appear to be compile issues with the current upgrade/release (20.11/20.11.1 with kernel 5.9.x).

SIGSEGV

SIGSEGV

Posted
Posted

With yesterday's release of ZFSon Linux 2.0, shouldn't all of this be easier?

I'm looking forward to seeing performance and stability improvements with the new version - as well as ease of deployment.

TRS-80

TRS-80

Posted
Posted (edited)
  On 12/1/2020 at 2:56 PM, SIGSEGV said:

With yesterday's release of ZFSon Linux 2.0, shouldn't all of this be easier?

Expand  

 

That's a good point, and I suspect maybe (probably?) so.  However a quarterly release was just made, so it will probably be some time until this makes it into Armbian.  Maybe next quarterly release, maybe one after that (I have no idea, and please don't take that as any sort of promise, or ask "when" etc. because that's not how any of this works).

 

I realize that you guys have purchased a commercial product, and therefore may have certain expectations.  At the same time I sense that this may be some people's first introduction to how a true Free Software, community based project works.  So what I will say in this post applies to Armbian (the community project) itself.  You also have your support from your vendor (Kobol) who are working on things, in fact they work hand in hand with the Armbian project, and in general such partnerships are a Good Thing (for lots of reasons).  But I am part of Armbian project and so I can only speak for our part, not Kobol your vendor you bought the hardware from.

 

Now, gotten that out of the way, understand there is no big company, business model, or anything else behind Armbian.  It's just us, you and me, little guys like us all over the world.  You guys are doing great job in here helping each other out and figuring things out, together.  And this is why we have forums and everything is humming along nicely as it should so far.

 

Now the next step, for those who have an interest in this board, and in ZFS (and I detect a lot of interest in this thread), along with the requisite technical ability, would be to get involved in figuring how to get this into upstream Armbian, for the benefit of everyone.

 

For example, already in another thread a young man figured out the build steps, distilled that down into some bash script, and shared his results publicly.  That is based on current stuff, and should be a good stop-gap / workaround to point people to in the meantime.  And it's a great example of how F/LOSS development marches forward, one little step at a time, slowly but steadily, based on work of individuals.  Which is the only way any of this works.

 

Which leaves the question of new development going forward.  @SIGSEGV raises a good point about OpenZFS 2.0.  I cannot speak to specifics because I do not own this hardware and I am not a dev anyway so I cannot help out in that way.  However if any of you want to give it a try, by all means please do.  Make another forum post about getting that working, share your results with one another, test, test, and when it gets stable, I am sure it makes it into Armbian.  And probably faster than waiting for someone else to do it.

 

The "core devs" as mentioned further up thread should get around to it, but I can't say when.  There are only so many of them, remember all part time and mostly unpaid.[0]  I'm just trying to paint a realistic picture here of the resources we have to work with, which are scant (especially considering people's apparent expectations).  But also I want to make the point that you too, can become a "core dev" simply by sharing what you have learned, and contributing that back to the project in whatever form.  And every little step forward helps.  If you investigate, share your results.  Test, and share your results (good or bad!).  Many hands make light work, and thus the development (and maintenance) burden is shared and we all move forward together.

 

I hope this post was helpful in giving a broader picture of why it's against the (Armbian) rules[1] to ask "when" and similar questions.  In fact, feel free to point people towards this post when such questions come up.  Even by doing this, you are helping.  By hanging out around forums.  This is all I do, I am not a dev.  I also start contributing to docs lately, in fact I plan to re-work the relevant part of the docs touching on similar themes as this post.  I have a whole list of things I want to add or re-work in the docs.  We each do our own little part, as much as we are able (and perhaps more importantly, willing).

 

Cheers!  :beer:

 

[0] In case this comes up, yes Kobol have contributed financially to Armbian development and have been doing so for years.  They are a great company and partners and do things The Right Way.

[1] Those rules only apply to the Armbian forums overall, we take a much more hands off approach here in Kobol Club, as this is official support forum for some commercial products which are based on Armbian (i.e., rules are different here, and set by Kobol).

Edited by TRS-80
grammar
2
TRS-80

TRS-80

Posted
Posted (edited)
  On 12/1/2020 at 7:49 PM, snakekick said:

Hello,

for me with ubuntu it´s work great to add this ppa and all things are fine ;)

https://launchpad.net/~jonathonf/+archive/ubuntu/zfs?field.series_filter=focal

 

you need only install kernel headers with armbian config and than you can install zfs-dkms 

Expand  

 

Thanks for sharing workaround, glad it's working for you, and welcome to forums.

 

However fixing upstream kernel/GCC issue is a much broader topic touching many different boards and thus makes it a much more complicated question when you look at it in terms of what the project supports overall.

 

So apparently my whole rah rah about "helping out" does not exactly apply in this particular case, as it really is an architectural decision, which need to be made very carefully at high level in project.  In other words, what @jbergler already mentioned (and in fact, he was original one filing bug report at GitHub, that I linked in this post).

 

Sorry if it took me a while to come full circle, some times I can be slow on the uptake.  :D

 

Everything I wrote above is still (generally) true however, so I leave it there for now.  In fact I could cite jbergler contribution as evidence of same...

Edited by TRS-80
slow on uptake, lol
SIGSEGV

SIGSEGV

Posted
Posted

I agree with @TRS-80.

Some topics are better documented with a small How To post or even within the Wiki pages of the Kobol/Helios64 projects.

That being said - I will post how to get iSCSI targets published on Armbian - now that the new kernels has been released, hopefully it will be made sticky for others wanting to do the same.

 

Regarding the recent release of ZFSon Linux - let's try out and document our findings.

I will for sure test it out and come back here to document how it went.

1
Igor

Igor

Posted
Posted

I added zfs zfs-dkms_0.8.5-2~20.04.york0_all to our repository so at least Ubuntu version works OOB (apt install zfs-dkms). I tested installation on Odroid N2+ running 5.9.11

* packaging mirrors need some time to receive updates so you might get some file-not-found, but it will work tomorrow. Headers must be installed from armbian-config.
 

  Reveal hidden contents

 

3
0utc45t

0utc45t

Posted
Posted

I actually managed to get the zfs working, by following the post by  @grek in this very thread. I've been running some fio tests to see if everything is dandy. Seems to be. :-) 

root@helios64:/ironwolf # zpool --version
zfs-2.0.0-rc6
zfs-kmod-2.0.0-rc6
root@helios64:/ironwolf # zfs version
zfs-2.0.0-rc6
zfs-kmod-2.0.0-rc6
root@helios64:/ironwolf #

 

I've noticed that zfs 2.0.0 has been released, so that means that I'll be compiling it after the fio runs are done (they've been running for a day or two now...) . 

1
Matt Clare

Matt Clare

Posted
Posted

Adding my encouragement. I'm watching with interest.

I have a zpool in large multi-disk USB3 enclosure. I'd love to get those drives directly on SATA - but not before I know it's safe.

Best of luck all.

0utc45t

0utc45t

Posted
Posted

@SIGSEGV I did compile zfs 2.0.0 and its up and running.  Rsynced 1.4TB to Helios box after that. Seems to be working. Next, home assistant, after that's up and running I'll get rid of my ATX computer that has been doing those jobs...

2
Demodude123

Demodude123

Posted
Posted

I also have zfs-2.0-rc5 running. I tried updating to rc7 and had some trouble with the deb packages. It's probably fixed in the GA build.

 

I used this guide: https://openzfs.github.io/openzfs-docs/Developer Resources/Building ZFS.html - and then `make deb` gives you debs you can install with `dpkg -i`. I installed the shared libs, zfs-dkms, and zfsutils from here.

 

As for systemd, you have to play around with it to get it to start on boot. Usually I can `sudo systemctl unmask <zfs-thing>`. Another good double check is to check /lib/systemd/system and /etc/systemd/system on a normal debian-based system and compare what's different. Make sure zfs.target is specified under the multi-user target!

 

If you use sanoid/synoid, it's a bit of a pain. It has debian's zfsutils as a dependency. You're probably better off downloading this off of git and setting up the cron yourself. Otherwise, you have to modify your apt cache to remove zfsutils as a dependency for sanoid.

 

I'm using zstd compression and have no complaints so far.

deucalion

deucalion

Posted
Posted

"Works for me".

 

I'll write a page how I did it at the helios64 reddit wiki, if you care for that, on the weekend.

I did have to manually insert the systemd-files as the build of those somehow didn't work as expected, but maybe I'll work that out as well once I condense my notes.

  

# zfs version
zfs-2.0.0-1
zfs-kmod-2.0.0-1
# uname -a
Linux helios 5.9.11-rockchip64 #20.11.1 SMP PREEMPT Fri Nov 27 21:59:08 CET 2020 aarch64 aarch64 aarch64 GNU/Linux
# lsb_release -d
Description:    Ubuntu 20.04.1 LTS

 

SIGSEGV

SIGSEGV

Posted
Posted

@Demodude123

It might be worth a try this approach to load the module on boot 

sudo sh -c "echo zfs >/etc/modules-load.d/zfs.conf"

 

@deucalion

@Demodude123

Were you able to make a deb package with the kmod?? The dmks is nice, but a kernel module to match the Armbian release would probably be better for end users.

@Igor what would your thoughts be on this matter? I'm aware that not all armbian supported hardware might benefit from it since not all are used for storage purposes, but those that are could have a new option for the filesystem.

Demodude123

Demodude123

Posted
Posted 
zfs-2.0.0.tar.gz
zfs-kmod-2.0.0-rc5.src.rpm
kmod-zfs-5.8.14-rockchip64_2.0.0-0_arm64.deb
kmod-zfs-devel_2.0.0-0_arm64.deb
kmod-zfs-devel-5.8.14-rockchip64_2.0.0-0_arm64.deb
zfs-dkms-2.0.0-rc5.src.rpm
zfs-dkms_2.0.0-0_arm64.deb
zfs-2.0.0-rc5.src.rpm
zfs_2.0.0-0_arm64.deb
libnvpair1_2.0.0-0_arm64.deb
libuutil1_2.0.0-0_arm64.deb
libzfs2_2.0.0-0_arm64.deb
libzpool2_2.0.0-0_arm64.deb
libzfs2-devel_2.0.0-0_arm64.deb
zfs-test_2.0.0-0_arm64.deb
zfs-dracut_2.0.0-0_arm64.deb
zfs-initramfs_2.0.0-0_arm64.deb
python3-pyzfs_2.0.0-0_arm64.deb

Here's what it built for me

Guest
This topic is now closed to further replies.
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept