Helios4 - Cryptographic Engines And Security Accelerator (CESA) Benchmarking

[gprovost 22]

It's been a while I have on my TODO list : write a guide on how to activate and use the Marvell Cryptographic Engines And Security Accelerator (CESA) on Helios4.

 

Previously I already shared some numbers related to the CESA engine while using @tkaiser sbc-bench tool. I also shared some findings on the openssl support for the kernel modules (cryptodev and af_alg) that interact with the cesa engine. My conclusion was :

 

1. performance wise : effectively cryptodev performs slightly better than af_alg.

2. openssl / libssl support : very messy and broken, it all depends which version of openssl you use.

 

Since many Debian Stretch apps depend on "old" libssl (1.0.2), I felt taking the cryptodev approach was the best way since it could expose all encryption and authentication algorithms supported by the cesa engine... even though it requires some patching in openssl. Plus cryptodev implementation in new LTS openssl version 1.1.1 has been completely reworked, so long term it should be the right way.

 

Anyhow I'm not going to describe here the step by step setup, I'm already writing a page on our wiki for that, once it's ready I will post the link here. Also I won't risk myself talking about the relevance of some of ciphers, it deserves a topic on its own.

 

I'm just going to share benchmark number on a concrete use case which is HTTPS file download :

 

So I setup on my Helios4 Apache2 to serve a 1GB file hosted on a SSD drive.

 

Then I did 3 batch of download tests, for each batch I configured Apache2 to use a specific cipher that I know is supported by the cesa engine.

• AES_128_CBC_SHA
• AES_128_CBC_SHA256
• AES_256_CBC_SHA256

 

For each batch, I do the following 3 tests :

1. download without cryptodev module loaded (100% software encryption)

2. download with cryptodev loaded and libssl (openssl) compiled with -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS

3. download with cryptodev loaded and libssl (openssl) compile only with -DHAVE_CRYPTODEV, which means hashing operation will still be done 100% by software.

 

Here are the results :

 

 

Note: CPU utilization is for both cores. Obviously each test is just a single process running on a single core therefore when you see CPU utilization around 50% (User% + Sys%) it means the core used for the test is fully loaded. Maybe i should have reported the number just for the core used, which would be more or less doing x2 of the value you see in the table.

 

For reference:

  Using AES_128_GCM_SHA256 (Default Apache 2.4 TLS cipher. GCM mode is not something that can be accelerated by CESA.)

  CPU Utilization : User %42.9, Sys %7.2
  Throughput : 30.6MB/s

  No HTTPS
  CPU Utilization : User %1.0, Sys %29.8
  Throughput : 112MB/s

 

 

CONCLUSION

1. Hashing operation are slower on the CESA engine than the CPU itself, therefore making HW encryption with hashing is less performant than 100% software encryption.

2. HW encryption without hashing provides 30 to 50% of throughput increase while decreasing the load on the CPU by 20 to 30%.

3. Still pondering if it's worth the effort to encourage people to do the move... but i think it's still cool improvement.

 

 

 

 

[gprovost Topic author 22]

I configured SSH to work with cryptodev and use cipher AES-CBC-128.

 

I did a scp download of the same 1GB file and get the following perf :

Throughput : 56.6MB/s

CPU Utilization : User %12.3, Sys %31.2

 

Pretty good :-)

 

Important note :

As concluded in previous post, in the case of Helios4, using cryptodev only for encryption and not for authentication (authentication involves hashing) is the only mode that provides some network performance and cpu load improvement. The other advantage of this mode, is that cryptodev will be completely skipped by sshd... because otherwise sshd will rise an exception during authentication because cryptodev try to do some ioctl() call that are forbidden by seccomp filter in sshd sandbox.

 

If you still want to test using cryptodev for ssh, the easy workaround is to use normal privilege separation in sshd instead of sandbox (UsePrivilegeSeparation yes). Then as for apache2, you will have to force to use a cipher that is supported by the CESA engine (e.g aes128-cbc)... and most probably you will also have to do the same on client side.

 

Disclaimer: The sshd tweaking is not recommended for security reason. Only experiment with it if you know what you are doing.

 

 

For reference with Cipher encryption algo not supported by CESA :

AES128-CTR

CPU Utilization : User %39.1, Sys %16.4

Throughput :  39.9MB/s

 

CHACHA20-POLY1305 (default cipher for ssh)

CPU Utilization : User %40.6, Sys %17.0

Throughput :  29.8MB/s

 

 

 

[gprovost Topic author 22]

Finally found the time to finish the CESA page on our Helios4 Wiki. It's not as exhaustive as it should be, but enough to help people experiment.

 

https://wiki.kobol.io/cesa/

 

For the ones who are interested, please have look... any comment welcome ;-)

 

[markbirss 2]

@gprovost thank you for effort with the guide.

 

Could you possibly include some CJDNS benchmarks ?

[gprovost Topic author 22]

On 11/20/2018 at 8:07 PM, markbirss said:

Could you possibly include some CJDNS benchmarks ?

 

I don't know much about CJDNS. Does CJDNS supports cryptodev or AF_ALG ?

[markbirss 2]

@gprovost  refer to these links for detail around how cjdns uses it's own CryptoAuth protocol with ed25519, poly1305, and salsa20.

https://github.com/cjdelisle/cjdns/blob/master/doc/Whitepaper.md

https://github.com/hyperboria/bugs/issues/112

 

To see if the encryption task the CJDNS uses benefits or not from cryptodev hw acceleration

 

[gprovost Topic author 22]

I already looked at the white paper but I don't have much time now to dig further. I'm not sure if CJDNS interfaces with the Kernel Crypto API.

 

Plus anyway the Marvell CESA engines don't support SALSA20 stream encryption... so i don't think CJDNS crypto can be accelerated.

 

 

[markbirss 2]

Ok, understood.

[Koen 0]

It could be interesting to see the test repeated while on a LUKS encrypted filesystem?

[gprovost Topic author 22]

@Koen Good point, will do a benchmark one of those days.

[djurny 1]

Hi, I would be interested in the LUKS benchmark results as well, with and without using the CESA.

 

Currently I'm trying to get my LUKS encrypted volumes to perform a bit better on the Helios4. On my previous box (AMD Athlon X2) I saw numbers above 80 MiB/sec for disk I/O when performing a 'snapraid diff/sync/scrub' on the same LUKS encrypted volumes. The drives themselves were the I/O bottleneck. On the Helios4, those numbers have dropped significantly: ~30MiB/sec for the same actions.

 

On the Helios4, with 'marvell-cesa' and 'mv_cesa' modules loaded, 'cryptsetup benchmark' shows:

/* snip */
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b   101.3 MiB/s   104.2 MiB/s
 serpent-cbc   128b    27.8 MiB/s    29.4 MiB/s
 twofish-cbc   128b    39.7 MiB/s    44.2 MiB/s
     aes-cbc   256b    91.7 MiB/s    94.1 MiB/s
 serpent-cbc   256b    28.0 MiB/s    32.0 MiB/s
 twofish-cbc   256b    39.7 MiB/s    44.3 MiB/s
     aes-xts   256b    63.4 MiB/s    55.0 MiB/s
 serpent-xts   256b    27.6 MiB/s    31.8 MiB/s
 twofish-xts   256b    43.3 MiB/s    44.0 MiB/s
     aes-xts   512b    47.9 MiB/s    41.6 MiB/s
 serpent-xts   512b    29.8 MiB/s    31.8 MiB/s
 twofish-xts   512b    43.2 MiB/s    44.0 MiB/

Without the CESA modules loaded, the aes-cbc performance drops significantly:

/* snip */
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b    25.1 MiB/s    56.2 MiB/s
 serpent-cbc   128b    28.0 MiB/s    31.9 MiB/s
 twofish-cbc   128b    39.7 MiB/s    44.3 MiB/s
     aes-cbc   256b    19.1 MiB/s    42.1 MiB/s
 serpent-cbc   256b    27.9 MiB/s    29.2 MiB/s
 twofish-cbc   256b    39.5 MiB/s    44.2 MiB/s
     aes-xts   256b    63.2 MiB/s    55.3 MiB/s
 serpent-xts   256b    29.8 MiB/s    31.8 MiB/s
 twofish-xts   256b    43.5 MiB/s    44.2 MiB/s
     aes-xts   512b    48.0 MiB/s    41.6 MiB/s
 serpent-xts   512b    27.3 MiB/s    31.7 MiB/s
 twofish-xts   512b    43.3 MiB/s    44.1 MiB/s

This already hints at the fact that 'dm-crypt' is not using the CESA.

 

After some checking, I would need to reencrypt my LUKS drives; they're using aes-xts-sha1, which is not supported by the CESA according to the Helios4 Wiki. The benchmark results shown by 'cryptsetup benchmark' show only an improvement for the aes-cbc algorithms, so first test will be to see what LUKS will do with 128bit aes-cbc-sha1 instead of 256bit aes-xts-sha1.

 

Groetjes,

 

p.s. I'm in no way a cryptography expert, so some of the terms might not be hitting the mark completely

Edited December 28, 2018 by djurny
Redundant statements, some typos.

[djurny 1]

Ok, I've made some progress already: It turns out that LUKS performance gets a boost when using cipher 'aes-cbc-essiv:sha256'. Key size did not really show any big difference during testing simple filesystem performance (~102MiB/sec vs ~103MiB/sec). The internets say that there is some concern about using cbc vs xts, but after some reading it looks like it's not necessarily a concern related to the privacy of the data, but more of an data integrity issue. Attackers are able to corrupt encrypted content in certain scenarios. For my use case, this is not a problem at all, so I'm happy to see the performance boost by using CESA!

 

Test setup & playbook:

• Make sure 'marvell-cesa' and 'mv_cesa' are modprobe'd.
• /dev/sdX1 - optimally aligned using parted.
• luksFormat /dev/sdX1
• luksOpen /dev/sdX1 decrypt
• mkfs.xfs /dev/mapper/decrypt
• mount /dev/mapper/decrypt /mnt
• Check /proc/interrupts
• dd if=/dev/zero of=/mnt/removeme bs=1048576 count=16384 conv=fsync
• Check /proc/interrupts if crypto has gotten any interrupts.

 

Averaged throughput measurement:

• Cipher: aes-xts-plain64:sha1, key size: 256bits
  • Throughput => 78.0 MB/s,
  • Interrupts for crypto => 11 + 8,
• Cipher: aes-cbc-essiv:sha256, key size: 256bits
  • Throughput => 102 MB/s,
  • Interrupts for crypto => 139421 + 144352,
• Cipher: aes-cbc-essiv:sha256, key size: 128bits
  • Throughput => 103 MB/s
  • Interrupts for crypto => 142679 + 152079.

 

Next steps;

1. Copy all content from the aes-xts LUKS volumes to the aes-cbc LUKS volumes,
2. Run snapraid check/diff/sync and check disk throughput.

Comments are welcome,

Groetjes,

Edited December 29, 2018 by djurny
Table markup flattens somehow?

[gprovost Topic author 22]

@djurny Thanks for the benchmark, very useful number. It reminds me I need to indicate on the wiki the correct cipher to use with cryptsetup (preference for aes-cbc-essiv over aes-cbc-plain*) when people create their LUKS device since on latest version of cryptsetup the default is cipher is aes-xts-plain64.

 

Time to combine both benchmark tests together as suggested by @Koen

 

BTW you don't need to load mv_cesa module. This is an old module which is replaced by marvell_cesa.

[Koen 0]

This is very useful information, as i'm planning to have boot root (SD) and data (SATA mirror) encrypted, with BTRFS on top.  Better get started the good way. 

@djurny : did you come across good links explaining the differences / risks of cbc versus xtc, or even essiv versus plain64 ?

 

Found this guide for the root fs :

 

And the data fs i should be able to do with a keyfile on the rootfs.  I think it needs to be 2x LUKS and BTRFS "mirror" on top, so i could actually benefit from the self healing functionality, in case of a scrub.

 

@gprovost : am i correct to understand the CESA will be used automatically by dm-crypt, if aes-cbc-essiv (or another supporter cypher) is used ?
Also looking forward to read updated performance numbers, to understand if it would be worth modifying the openssl libraries or not.   

[djurny 1]

@Koen, do note that on Armbian, the CESA modules are not loaded per default, so even if you choose aes-cbc-essiv but do not load the appropriate kernel module, the CESA is not utilized. (See the cryptsetup benchmark results with- and without marvell_cesa/mv_cesa.)

For the XTS vs CBC I will try to find & list the articles I found on the interwebs.

 

@gprovost, thanks for the tip, I'll refrain from loading mv_cesa.

 

[matzman666 1]

I also want to share my numbers because I think they show some interesting findings.

 

My setup: Helios4 & Raid5 with 4x Seagate Ironwolf 4TB (ST4000VN008)

To ensure comparability all numbers where obtained using the same command as used by @djurny: dd if=/dev/zero of=/mnt/removeme bs=1048576 count=16384 conv=fsync

 

 

First of all some numbers which show how important correct alignment is (no encryption yet, filesystem was btrfs):

• md-raid5 using the whole disk created via OMV or the instructions found in the wiki here:
  • Write-Throughput: ~76 MB/s
• md-raid5 using optimally aligned partitions (created via parted -a optimal /dev/sdX mkpart primary 0% 100%):
  • Write-Throughput: ~104 MB/s

That is a difference of about ~26%! Based on this numbers I would not recommend using the whole disk when creating a md-raid5. Using partitions is not supported at all by OMV, so the raid has to be created on the command-line.

 

 

Now my numbers when using encryption:

• md5-raid & luks (using aes-cbc-essiv:sha256) & xfs:
  • Write-Throughput: ~72 MB/s
• md5-raid & luks (using aes-cbc-essiv:sha256) & btrfs:
  • Write-Throughput: ~66 MB/s
• md5-raid & luks (using aes-cbc-essiv:sha256 with marvell_cesa kernel module unloaded) & btrfs:
  • Write-Throughput: ~34 MB/s
• md5-raid & luks2 (using aes-cbc-essiv:sha256) & btrfs:
  • Write-Throughput: ~ 73 MB/s

Looking at the numbers I see a performance loss of about 30% when using encryption. Hardware encryption is working and definitely speeds up encryption because when I unload the kernel module I see a performance loss of about 60% compared to the unencrypted case.

 

When creating the luks partition via the OMV encryption plugin aes-xts is used by default which is not supported by marvell_cesa, and there is no way to configure a different encryption algorithm on the web-gui. To be able to use aes-cbc the luks partition has to be created via the commandline.

 

Using luks2 instead of luks gives a bit of a performance boost. Luks2 is only supported by the Ubuntu image, the Debian image has the usual Debian problem: too old packages.

 

Based on these numbers I am ditching Debian and OMV and are moving to Ubuntu. OMV is of little use, because for best performance I have to setup everything via the commandline. Also luks2 is more future-proof and results in better performance.

 

Edit2:

 

I played a bit more with luks2, and found something very interesting: With luks2 you can also change the sector size.

 

The default sector size is 512 Byte, but when I change it to 4K, then I see massive performance improvements:

• md5-raid & luks2 (using aes-cbc-essiv:sha256 and 4K sectors) & btrfs:
  • Write-Throughput: ~ 99 MB/s

 

That's only a performance loss of ~5% compared to the unencrypted case!?! Before I had a performance loss of at least ~30%. I double-checked everything, and the numbers are real. This means luks2 is definitely the way to go.

 

To sum up. To get the best performance out of an encrypted raid5, you need to:

1. Install the Ubuntu image. This means you cannot use OMV, but that's the price to pay for best performance.
2. Create a single partition on each of your disks that is optimally aligned.
   • parted /dev/sdX mklabel gpt
   • parted -a optimal /dev/sdX mkpart 0% 100%
3. Create your raid with mdadm and pass the partitions you created in the second step.
   • mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
4. Create a luks2 partition using aes-cbc-essiv:sha256 and a sector size of 4K.
   • cryptsetup -s 256 -c aes-cbc-essiv:sha256 --sector-size 4096 --type luks2 luksFormat /dev/md0
5. Create your file-system on top of your encrypted raid device.

 

 

 

On 12/29/2018 at 7:15 AM, djurny said:

Averaged throughput measurement:

• Cipher: aes-xts-plain64:sha1, key size: 256bits
  • Throughput => 78.0 MB/s,
  • Interrupts for crypto => 11 + 8,
• Cipher: aes-cbc-essiv:sha256, key size: 256bits
  • Throughput => 102 MB/s,
  • Interrupts for crypto => 139421 + 144352,
• Cipher: aes-cbc-essiv:sha256, key size: 128bits
  • Throughput => 103 MB/s
  • Interrupts for crypto => 142679 + 152079.

 

Do you have any numbers for the unencrypted case? I am curious because I want to know if you also see a performance loss of about 30%.

 

Edit:

Should be parted -a not parted -o.

[gprovost Topic author 22]

@matzman666 Very interesting findings and well broke down tests

 

1. I wasn't aware that with LUKS2 you can define sector size bigger than 512b, therefore decreasing the number of IO operations.

 

2. I never investigate how RAID perf can be impacted with nowadays HDD that support block size of 4K (Advanced Format). Non-aligned 4K disk can effectively show a significant performance penalty. But I'm still a bit surprise that you see that much perf difference (26%) with and without disk alignment on your tests without encryption.

https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm#Blocks.2C_chunks_and_stripes

 

Would you be interested to help us improve our wiki, on the disk alignment part and then on the disk encryption part ?

https://wiki.kobol.io/mdadm/

https://wiki.kobol.io/cesa/

[djurny 1]

@matzman666 Sorry, no measurements. From memory the numbers for raw read performance were way above 100MB/sec according to 'hdparm -t'. Currently my box is live, so no more testing with empty FSes on unencrypted devices for now. Perhaps someone else can help out?

 

-[ edit ]-

So my 2nd box is alive. The setup is slightly different and not yet complete.

 

I quickly built cryptsetup 2.x from sources on Armbian, was not as tough as I expected - pretty straightforward: configure, correct & configure, correct & ...

 

cryptsetup 2.x requires the following packages to be installed:

• uuid-dev
• libdevmapper-dev
• libpopt-dev
• pkg-config
• libgcrypt-dev
• libblkid-dev

 

Not sure about these ones, but I installed them anyway:

• libdmraid-dev
• libjson-c-dev

 

Build & install:

• Download cryptsetup 2.x via https://gitlab.com/cryptsetup/cryptsetup/tree/master. 
• Unxz the tarball.
• ./configure --prefix=/usr/local
• make
• sudo make install
• sudo ldconfig

 

Configuration:

• 2x WD Blue 2TB HDDs.
• 4KiB sector aligned GPT partitions.
• mdadm RAID6 (degraded).

 

Test:

• Write 10GiB 4GiB worth of zeroes; dd if=/dev/zero of=[dut] bs=4096 count=1048576 conv=fsync.
  • directly to mdadm device.
  • to a file on an XFS filesystem on top of an mdadm device.
  • directly to LUKS2 device on top of an mdadm device (512B/4096KiB sector sizes for LUKS2).
  • to a file on an XFS filesystem on top of a LUKS2 device on top of an mdadm device (512B/4096KiB sector sizes for LUKS2).

 

Results: See attached table.

 

Caveat:

• CPU load is high: >75% due to mdadm using CPU for parity calculations. If using the box as just a fileserver for a handful of clients, this should be no problem. But if more processing is done besides serving up files, e.g. transcoding, (desktop) applications, this might become problematic.
• RAID6 under test was in degraded mode. I don't have enough disks to have a fully functional RAID6 array yet. No time to tear down my old server yet. Having a full RAID6 array might impact parity calculations and add 2x more disk I/O to the mix.

 

I might consider re-encrypting the disks on my first box, to see if LUKS2 w/4KiB sectors will increase the SnapRAID performance over the LUKS(1) w/512B sectors. Currently it takes over 13 hours to scrub 50% on a 2-parity SnapRAID configuration holding less than 4TB of data.

 

Groetjes,