WARNING: if you do this wrong or don’t understand the concepts, you risk losing your data. Be sure you know your way around linux and what you’re getting into before attempting!
To me, encryption of data at rest is just as important as encryption of data in transit. You never know if someone is going to break into your house and steal your computer. With so much personal information like financial data and pictures stored on the computer, it could be a major mess to recover from theft of your computer. (Of course, always keep an off-site backup for the really important stuff!)
I chose to migrate from the Solaris based OpenIndiana to Ubuntu. I had grown to love ZFS on OpenIndiana and didn’t want to lose its features. Luckily ZFS on Linux is now ready for prime-time! Unfortunately, ZFS on Linux is a few versions behind the official Oracle ZFS just like all other third part implementations of ZFS and does not support native encryption through the filesystem.
My solution was to use the Linux Unified Key Setup (LUKS) to encrypt the raw devices underneath ZFS. LUKS is relatively new to the disk encryption space but is considered mature. The setup goes like this:
ZFS Filesystem <-top
Raw disks <- bottom
Modified from ServerFault.
With this setup, the mapper devices that are created via the LUKS unlocking/opening process are simply presented as block devices that can then be leveraged by ZFS. The only gotcha is you can’t have your zpools mount on boot – which makes sense because if you’re using encryption you want to be able to unlock them first before they are usable. So on boot, you unlock the disks first, then import your zpools. I’ve provided a script below to make it easier.
Start with a base, fully up to date Ubuntu 14.04 LTS release, then install a few extra packages:
sudo apt-get install cryptsetup nfs-common nfs-kernel-server samba iscsitarget-dkms iscsitarget
To get ZFS on Linux installed, you need to add and install from a PPA:
sudo add-apt-repository ppa:zfs-native/stable sudo apt-get update sudo apt-get install ubuntu-zfs
Next, select the algorithm and key size to use in your setup. The larger the key size, the longer you will be safe from attack before needing to re-encrypt with a larger key, so go as big as you can afford to with performance. If you have a more modern CPU that supports AES-NI, there should be next to no CPU hit using a 512b key. Also, you should use a cipher using the XFS block mode, as it has been reported there may be vulnerabilities in CBC. From the cryptsetup man page: “The available combinations of ciphers, modes, hashes and key sizes depend on kernel support. See /proc/crypto for a list of available options. You might need to load additional kernel crypto modules in order to get more options.”
To get a rough idea of performance for your CPU, you can run the `cryptsetup benchmark` command:
dan@storage:~$ cryptsetup benchmark # Tests are approximate using memory only (no storage IO). PBKDF2-sha1 197993 iterations per second PBKDF2-sha256 140786 iterations per second PBKDF2-sha512 90145 iterations per second PBKDF2-ripemd160 215578 iterations per second PBKDF2-whirlpool 115992 iterations per second # Algorithm | Key | Encryption | Decryption aes-cbc 128b 141.2 MiB/s 155.1 MiB/s serpent-cbc 128b 73.1 MiB/s 181.5 MiB/s twofish-cbc 128b 157.0 MiB/s 199.1 MiB/s aes-cbc 256b 111.5 MiB/s 121.4 MiB/s serpent-cbc 256b 74.4 MiB/s 182.4 MiB/s twofish-cbc 256b 158.7 MiB/s 198.6 MiB/s aes-xts 256b 151.4 MiB/s 153.5 MiB/s serpent-xts 256b 167.0 MiB/s 169.0 MiB/s twofish-xts 256b 181.9 MiB/s 181.2 MiB/s aes-xts 512b 115.7 MiB/s 118.1 MiB/s serpent-xts 512b 167.0 MiB/s 170.2 MiB/s twofish-xts 512b 182.2 MiB/s 180.4 MiB/s
My hardware is a bit slow, being about 3 years old and without AES-NI, but when you think about it, I don’t need more than 100MB/s because that will saturate my gigabit ethernet connection and my primary use for this system is network attached storage.
Next, encrypt your disks. Note that you don’t need to create partitions on the raw disks, as you’re just going to create ‘partitions’ via ZFS anyways. The example below formats the raw devices. Make sure you have the right disks selected and that there is no data you need on them, as all data will be rendered useless!
EDIT: I’ve modified the commands to include –iter-time and –use-random for greater security at the suggestion of Valentin in the comments below. A value of 10000 means PBKDF2 will spend 10 seconds processing the passphrase each time you unlock the disk, slowing your mount but also slowing a brute force attempt against your key. I’ve also changed the cipher to aes-xts-plain64 per his suggestion. Thanks Valentin!
cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 512 --iter-time 10000 --use-random -y /dev/sdb cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 512 --iter-time 10000 --use-random -y /dev/sdc
When running this command you’ll be prompted for a passphrase. Choose a very long and strong passphrase. The strength of this passphrase also determines how vulnerable your disks will be. Also, it is a little counter intuitive for security, but I suggest for ease of use and to use the script provided below to use the same passphrase for all the disks.
Then, unlock them:
cryptsetup luksOpen /dev/sdb sdb-enc cryptsetup luksOpen /dev/sdc sdc-enc
The final argument can be whatever you want to use to reference the encrypted volume. It will be placed in /dev/mapper and is what you will reference when creating your zpool.
Take note that if you have one of the newer 4K format drives, you will likely gain better performance from using an ashift value of 12 in order to have ZFS align with the sectors on the disk. You won’t hurt the drives by not aligning them – just your performance might not be as great, especially if you’re using raidz1 or raidz2. See this post for methods to determine if you have 4K drives: https://wiki.archlinux.org/index.php/Advanced_Format#How_to_determine_if_HDD_employ_a_4k_sector
This command will create a mirror using our encrypted volumes with ashift set appropriately for 4K drives. If you still have 512 byte drives, just remove the -o ashift=12 option.
sudo zpool create -o ashift=12 tank mirror sdb-enc sdc-enc
And that’s it! ZFS is now riding on LUKS encrypted disks and runs just like Oracle ZFS. Then you can do things like enable deduplication, compression, SMB or NFS via ZFS like you normally would.
Gotchas – solved with a script
In my testing, I crashed my system in every way and state I could think of to make sure my pools came back after reboot and had no problems. I’ve confirmed that if the power goes out, the pools will come back. ZFS isn’t fragile and a scrub after an unclean shutdown does wonders for peace of mind, but it is a little disconcerting to forcefully close the LUKS containers on shutdown without having unmounted the zpools first. There is always the possibility of data loss if the pool is in the middle of writing data and dies. So I created a script to try to make it more easy and safe to reboot the system, as well as help unlock LUKS and remount ZFS on boot via an included rc script.
Note that the script is generic. You need to modify the first few lines to include references to your zpools and LUKS disks and mapper names. If you use other services that leverage the pools such as iSCSI or ZFS native SMB/NFS, you might need to modify the script to add commands to stop those services before unmounting the zpools, otherwise they will be busy.
- Get all zpool statuses and exports any active pools, which handles the case of an unclean shutdown without first exporting the zpools
- Close all open LUKS containers
- Opens all LUKS containers specified at the top of the script
- Import all zpools specified at the top of the script
- Shares out any ZFS shares
- Unshare any ZFS shares
- Export active zpools
- Close all LUKS containers.
It can be run with the parameters of status, mount, unmount, reboot, and shutdown.
To cleanly shutdown automatically on reboot or shutdown, copy the ‘storage’ script from the repository into /etc/init.d, then run this command to set the script up to automatically stop for each runlevel, while not running on startup:
update-rc.d storage stop 99 0 1 2 3 4 5 6 .
Then when you startup, simply run `service storage start` and enter your passphrase to unlock the disks.
The script is available on my BitBucket here. If you have suggestions, submit a pull request!