I have installed Armbian 21.02 Ubuntu 20.04 Focal with Linux kernel 5.10.16 and OpenZFS 2.0.1-1 on my Helios64. Created a three-disk encrypted RAIDZ pool with the following command:
sudo zpool create -f -o ashift=12 -O compression=lz4 -O atime=off -O acltype=posixacl -O xattr=sa -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase tank raidz1 /dev/disk/by-id/foobar1 /dev/disk/by-id/foobar2 /dev/disk/by-id/foobar3
Then I did a single sequential read benchmark with fio on /tank:
cd /tank
sudo fio --name=single-sequential-read --ioengine=posixaio --rw=read --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
This results in a speed of 62 MB/s, much lower than I would expect.
I created exactly the same pool with three disks of the exact model (same batch) on Ubuntu 20.04 with Linux kernel 5.4.0 and OpenZFS 0.8.3 on a HPE ProLiant MicroServer Gen10, and the performance of the same benchmark is 232 MB/s.
Does anyone have the same experience of this low encrypted ZFS performance on the Helios64? Is this because of the encryption? Looking at the output of top during the benchmark, the CPU seems to be taxed much, while on the HPE machine CPU usage is much less.
