5 5
SvenHz

ZFS on Helios4

Recommended Posts

@gprovost Sure. I wasn't aware that you accept community contributions to the wiki.  I might take a stab at that once I have my own setup working ... If someone else, probably more experienced with ZFS, is up for that, please go ahead - I for one would certainly appreciate it.

Share this post


Link to post
Share on other sites

I finally got ZFS installed on my Helios4 using Armbian 19.11.3 Buster with Linux 4.19.84, but I had to stick with ZFS version 0.7.12, as advise by @SvenHz in this post. Loading the ZFS 0.8 module would fail with the same error.

 

@Steven Keuchel  Thanks again for your input on the kernel/ZFS memory settings. I had a more detailed look at your settings and noticed that you set `zfs_arc_max=805306368` (i.e. to 768 MB), which corresponds to the maximum `vmalloc` you seem to use. Or were those ZFS settings with `CONFIG_VMSPLIT_2G` and you use in fact an even higher value for `vmalloc`? Which ZFS version did you use, 0.6 or 0.7?

 

As @qstaq stated, the default for `zfs_arc_max` is 50% of the available memory. There is however a pull request to increase this even further to 3/4 of all memory or all but 1GB, whichever is greater. This change is enabled due to the revised memory allocation for the ARC buffers in ZFS version 0.7.  In fact, this change makes me wonder if those settings for `vmalloc` and `zfs_arc_max` still make sense for ZFS >= 0.7. 

 

The fundamental issue with the extensive use of the virtual address space in ZFS was outlined rather well by one of the developers in this ZOL issue. This was addressed however in version 0.7 with the introduction of the ARC Buffer Data (ABD), as also mentioned in the ZFS 0.7 release notes.

 

It appears that the issues with the virtual address space are limited to ZFS versions 0.6 and older. Or is my understanding of this incorrect? Any advice or comment in this regard would be greatly appreciated.

Share this post


Link to post
Share on other sites
On 12/23/2019 at 8:19 AM, SvenHz said:

If I manually use dkms to build the ZFS 0.8.2 from the sources left around by apt-get, I get a successful build, however "mobprobe zfs" gives me a Exec Format Error. dmesg shows:

zlua: section 4 reloc 44 sym 'longjmp': unsupported interworking call (Thumb -> ARM)

 

Does anyone have 0.8.2 succesfully running on Helios4?

 

@SvenHz I made some progress on tracking down this issue with zfs 0.8 on the Helios4, more precisely James Robertson (@jsrlabs) identified and fixed the missing Thumb-2 instructions used by the Armbian kernel, see the discussion on the ZFS mailinglist.

 

I went ahead and opened a bug report (zfsonlinux/zfs#9957), and one of the zfs developers proposed a fix in this pull request: zfsonlinux/zfs#9967. However, the fix needs testing, so I thought to bring this up here in the hope that someone else will give this a shot as well.

Share this post


Link to post
Share on other sites

@dkxls,

 

Thanks for your digging into using ZFS on 32-bit on ARM (I also found your recent thread on the ZoL mailing list). This revelation about changes in ZoL 0.7 appears to blow my previous assumptions about 64-bit being required (in general) for ZFS right out of the water. Which opens up a lot more hardware possibilities (in particular, ironically enough, outside of ARM).

 

Within ARM, the Helios4 is the only hardware I am aware of that has ECC memory (that is not something designed for server farms). And as far as I am concerned, there is no point really doing ZFS without ECC.

 

At this point I am really looking forward to Helios64 going forward. But this is still great news for lots of older 32-bit pre-IME x86 hardware.

 

Cheers!

Share this post


Link to post
Share on other sites
On 1/28/2020 at 4:01 AM, dkxls said:

I finally got ZFS installed on my Helios4 using Armbian 19.11.3 Buster with Linux 4.19.84, but I had to stick with ZFS version 0.7.12, as advise by @SvenHz in this post. Loading the ZFS 0.8 module would fail with the same error.

 

@Steven Keuchel  Thanks again for your input on the kernel/ZFS memory settings. I had a more detailed look at your settings and noticed that you set `zfs_arc_max=805306368` (i.e. to 768 MB), which corresponds to the maximum `vmalloc` you seem to use. Or were those ZFS settings with `CONFIG_VMSPLIT_2G` and you use in fact an even higher value for `vmalloc`? Which ZFS version did you use, 0.6 or 0.7?

 

As @qstaq stated, the default for `zfs_arc_max` is 50% of the available memory. There is however a pull request to increase this even further to 3/4 of all memory or all but 1GB, whichever is greater. This change is enabled due to the revised memory allocation for the ARC buffers in ZFS version 0.7.  In fact, this change makes me wonder if those settings for `vmalloc` and `zfs_arc_max` still make sense for ZFS >= 0.7. 

 

The fundamental issue with the extensive use of the virtual address space in ZFS was outlined rather well by one of the developers in this ZOL issue. This was addressed however in version 0.7 with the introduction of the ARC Buffer Data (ABD), as also mentioned in the ZFS 0.7 release notes.

 

It appears that the issues with the virtual address space are limited to ZFS versions 0.6 and older. Or is my understanding of this incorrect? Any advice or comment in this regard would be greatly appreciated.

The 768MB arc_max was for a vmalloc of 1024MB and I was using zfs 0.7.12 when I wrote that, but since then I have switched to 0.8 (with Thumb2 patches from github). I have looked at this again (now with 0.8), and it seems the situation has indeed changed. When looking at vmalloc of zfs (spl actually) via /proc/vmallocinfo it seems to hover around 100-150MB with about 220MB being the highest I've seen and also the total address space occupied by all allocations has always been under 300MB and usually far below. So it seems rather compact with only little fragmentation happening. So zfs appears to use less vmalloc than I assumed and more lowmem, so I will have to revise my settings, because they clearly don't make sense anymore.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
5 5