Motivation

Instead of encrypting only the sensitive parts of hosted applications (e.g. mail storage, database partitions, etc.) it was an appealing idea to try full disk encryption (FDE).

Hardware

Setup

Most of this is also covered by muellis blog but instead of btrfs I preferred ext4 and utilized a modified version of the Hetzner installation script.

Preparation

Boot into linux rescue mode and make a local copy of the installation script cp -a /root/.oldroot/nfs/install /root/.oldroot/nfs/images/Ubuntu-1604-xenial-64-minimal.tar.gz /root/.
Then modify the copied script(s) so they setup also LUKS (Linux Unified Key Setup):

install.sh: Add before # LVM

#
# LUKS
#
  
inc_step
status_busy "Creating LUKS container"
make_luks 
status_donefailed $?

wait_for_udev

functions.sh: Add before make_lvm() {

make_luks() {
  echo -n "CHANGEME" | cryptsetup luksFormat /dev/md1 -d -
  echo -n "CHANGEME" | cryptsetup luksOpen /dev/md1 md1 -d -
  return 0
}

functions.sh: Edit pv variable definitions in make_lvm() { to match

    # create PVs
    for i in $(seq 1 $LVM_VG_COUNT) ; do
      pv="/dev/mapper/md${i}"
      debug "# Creating PV $pv"
      wipefs -af $pv |& debugoutput
      pvcreate -ff $pv 2>&1 | debugoutput
    done

    # create VGs
    for i in $(seq 1 $LVM_VG_COUNT) ; do
      vg=${LVM_VG_NAME[$i]}
      pv="/dev/mapper/md${i}"

This changes just add another step which encrypts md1 with LUKS and password CHANGEME and adapts the physical volume paths which are used for LVM.
Of course this is just are very hacky “implementation” because it uses a hard coded password (CHANGEME) and hard coded mapper names/paths. But for my singular use case everything else would be impractical.

Installation

Now we utilize the installation magic from Hetzner:
/root/install/installimage -n my.hostname.tld -r yes -l 1 -p /boot:ext4:512M,lvm:vg0:all -v vg0:swap:swap:swap:4G,vg0:root:/:ext4:20G -i /root/Ubuntu-1604-xenial-64-minimal.tar.gz -K /root/.ssh/robot_user_keys
As you can see I’d only used 20G for logical volume root. That’s just because this is a XEN server where each DomU has it’s own logical volume and has nothing to do with the FDE.

To be able to decrypt md1 after the reboot (and without KVM) it’s necessary to finalize the installation in chroot.

change root:

mount /dev/vg0/root /mnt
rm -r /mnt/run/lock/
mount /dev/md0 /mnt/boot/
chroot-prepare /mnt
chroot /mnt

Configure crypttab and avoid bug 1256730:

echo "# https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1256730" > /usr/share/initramfs-tools/conf-hooks.d/forcecryptsetup
echo "export CRYPTSETUP=y" >> /usr/share/initramfs-tools/conf-hooks.d/forcecryptsetup
echo "md1 /dev/md1 none luks" > /etc/crypttab

Install necessary packages:

apt-get update
apt-get install -y cryptsetup busybox dropbear

Configure initramfs and dropbear:
Replace {IP}, {GATEWAY}, {NETMASK}, {HOSTNAME} with appropriate values

mkdir -p /etc/initramfs-tools/root/.ssh/
chmod ug=rwX,o= /etc/initramfs-tools/root/.ssh/
cp /root/.ssh/authorized_keys /etc/initramfs-tools/root/.ssh/authorized_keys
echo "ifconfig eth0 0.0.0.0" >> /usr/share/initramfs-tools/scripts/init-bottom/dropbear
echo "DEVICE={IP}::{GATEWAY}:{NETMASK}:{HOSTNAME}:eth0" >> /etc/initramfs-tools/initramfs.conf

(Optional) Add a short unlock script (borrowed by stinkyparkia.wordpress.com) for convenience:

cat <<"OUTEREOF" > /etc/initramfs-tools/hooks/crypt_unlock.sh
#!/bin/bash

if [ "$1" == "prereqs" ]; then
  echo dropbear
  exit 0
fi

. "${CONFDIR}/initramfs.conf"
. /usr/share/initramfs-tools/hook-functions

if [ "${DROPBEAR}" == "n" ] || [ ! -r "/etc/crypttab" ]; then
  exit 0
fi

cat > "${DESTDIR}/bin/unlock" <<"EOF"
#!/bin/sh
if PATH=/lib/unlock:/bin:/sbin /scripts/local-top/cryptroot; then
  kill `ps | grep '[c]ryptroot' | awk '{ print $1 }'`
  # following line kill the remote shell right after the passphrase has
  # been entered.
  kill -9 `ps | grep "\-[s]h" | awk '{ print $1 }'`
  exit 0
fi
exit 1
EOF

chmod 755 "${DESTDIR}/bin/unlock"
mkdir -p "${DESTDIR}/lib/unlock"
cat > "${DESTDIR}/lib/unlock/plymouth" <<"EOF"
#!/bin/sh
[ "$1" == "--ping" ] && exit 1
/bin/plymouth "$@"
EOF

chmod 755 "${DESTDIR}/lib/unlock/plymouth"
echo "To unlock root-partition run 'unlock'" >> ${DESTDIR}/etc/motd
OUTEREOF

And finaly update initramfs and grub

update-initramfs -u -k all
update-grub2

Now everything should be fine and you can reboot into your fresh installed Ubuntu with full disk encryption.

Client side

Because SSH key errors are annoying and normally you don’t login as root to your server, just add the following to your local SSH client configuration (~/.ssh/config) and you can SSH into busybox with ssh unlock:

Host unlock
  User root
  HostName {HOSTNAME}
  UserKnownHostsFile ~/.ssh/dropbear.known

Troubleshooting

If there are any problems (e.g. you are not able to SSH into busybox) just boot into the rescue system and chroot into the installed Ubuntu:

echo -n "CHANGEME" | cryptsetup luksOpen /dev/md1 md1 -d -
pvscan
vgscan
vgchange -a y
mount /dev/vg0/root /mnt
rm -r /mnt/run/lock/
mount /dev/md0 /mnt/boot/
chroot-prepare /mnt
chroot /mnt

After fixing any errors (e.g. DEVICE config in /etc/initramfs-tools/initramfs.conf) exit from chroot and try again:

exit
umount /mnt/boot
umount /mnt/proc
umount /mnt/dev
umount /mnt/sys
umount /mnt
# ^ all optional
reboot

Hetzner KVM can be very handy while debuging any boot issues. I’d much fun with a wrong DEVICE string.

Change passphrase

LUKS provides up to 8 key slots (0 to 7) to store different passwords. To change your current password add a new one cryptsetup luksAddKey /dev/md/1 (asks for a valid password), verify it with cryptsetup luksOpen --test-passphrase --key-slot 1 /dev/md/1 && echo OK and then remove the old password with cryptsetup luksRemoveKey /dev/md/1 (asks for a valid password).

Before removing the old passphrase I wait until next reboot to verify that everything works just fine with the new password.

About

Blog

Privacy

Ubuntu Xenial FDE on Hetzner