blk-crypto: add basic hardware-wrapped key support

============

We introduce ``struct blk_crypto_key`` to represent an inline encryption key and

how it will be used.  This includes the actual bytes of the key; the size of the

key; the algorithm and data unit size the key will be used with; and the number

of bytes needed to represent the maximum data unit number the key will be used

with.

how it will be used.  This includes the type of the key (raw or

hardware-wrapped); the actual bytes of the key; the size of the key; the

algorithm and data unit size the key will be used with; and the number of bytes

needed to represent the maximum data unit number the key will be used with.

We introduce ``struct bio_crypt_ctx`` to represent an encryption context.  It

contains a data unit number and a pointer to a blk_crypto_key.  We add pointers

When the crypto API fallback is enabled, this means that all bios with and

encryption context will use the fallback, and IO will complete as usual.  When

the fallback is disabled, a bio with an encryption context will be failed.

.. _hardware_wrapped_keys:

Hardware-wrapped keys

=====================

Motivation and threat model

---------------------------

Linux storage encryption (dm-crypt, fscrypt, eCryptfs, etc.) traditionally

relies on the raw encryption key(s) being present in kernel memory so that the

encryption can be performed.  This traditionally isn't seen as a problem because

the key(s) won't be present during an offline attack, which is the main type of

attack that storage encryption is intended to protect from.

However, there is an increasing desire to also protect users' data from other

types of attacks (to the extent possible), including:

- Cold boot attacks, where an attacker with physical access to a system suddenly

  powers it off, then immediately dumps the system memory to extract recently

  in-use encryption keys, then uses these keys to decrypt user data on-disk.

- Online attacks where the attacker is able to read kernel memory without fully

  compromising the system, followed by an offline attack where any extracted

  keys can be used to decrypt user data on-disk.  An example of such an online

  attack would be if the attacker is able to run some code on the system that

  exploits a Meltdown-like vulnerability but is unable to escalate privileges.

- Online attacks where the attacker fully compromises the system, but their data

  exfiltration is significantly time-limited and/or bandwidth-limited, so in

  order to completely exfiltrate the data they need to extract the encryption

  keys to use in a later offline attack.

Hardware-wrapped keys are a feature of inline encryption hardware that is

designed to protect users' data from the above attacks (to the extent possible),

without introducing limitations such as a maximum number of keys.

Note that it is impossible to **fully** protect users' data from these attacks.

Even in the attacks where the attacker "just" gets read access to kernel memory,

they can still extract any user data that is present in memory, including

plaintext pagecache pages of encrypted files.  The focus here is just on

protecting the encryption keys, as those instantly give access to **all** user

data in any following offline attack, rather than just some of it (where which

data is included in that "some" might not be controlled by the attacker).

Solution overview

-----------------

Inline encryption hardware typically has "keyslots" into which software can

program keys for the hardware to use; the contents of keyslots typically can't

be read back by software.  As such, the above security goals could be achieved

if the kernel simply erased its copy of the key(s) after programming them into

keyslot(s) and thereafter only referred to them via keyslot number.

However, that naive approach runs into a couple problems:

- It limits the number of unlocked keys to the number of keyslots, which

  typically is a small number.  In cases where there is only one encryption key

  system-wide (e.g., a full-disk encryption key), that can be tolerable.

  However, in general there can be many logged-in users with many different

  keys, and/or many running applications with application-specific encrypted

  storage areas.  This is especially true if file-based encryption (e.g.

  fscrypt) is being used.

- Inline crypto engines typically lose the contents of their keyslots if the

  storage controller (usually UFS or eMMC) is reset.  Resetting the storage

  controller is a standard error recovery procedure that is executed if certain

  types of storage errors occur, and such errors can occur at any time.

  Therefore, when inline crypto is being used, the operating system must always

  be ready to reprogram the keyslots without user intervention.

Thus, it is important for the kernel to still have a way to "remind" the

hardware about a key, without actually having the raw key itself.

Somewhat less importantly, it is also desirable that the raw keys are never

visible to software at all, even while being initially unlocked.  This would

ensure that a read-only compromise of system memory will never allow a key to be

extracted to be used off-system, even if it occurs when a key is being unlocked.

To solve all these problems, some vendors of inline encryption hardware have

made their hardware support *hardware-wrapped keys*.  Hardware-wrapped keys

are encrypted keys that can only be unwrapped (decrypted) and used by hardware

-- either by the inline encryption hardware itself, or by a dedicated hardware

block that can directly provision keys to the inline encryption hardware.

(We refer to them as "hardware-wrapped keys" rather than simply "wrapped keys"

to add some clarity in cases where there could be other types of wrapped keys,

such as in file-based encryption.  Key wrapping is a commonly used technique.)

The key which wraps (encrypts) hardware-wrapped keys is a hardware-internal key

that is never exposed to software; it is either a persistent key (a "long-term

wrapping key") or a per-boot key (an "ephemeral wrapping key").  The long-term

wrapped form of the key is what is initially unlocked, but it is erased from

memory as soon as it is converted into an ephemerally-wrapped key.  In-use

hardware-wrapped keys are always ephemerally-wrapped, not long-term wrapped.

As inline encryption hardware can only be used to encrypt/decrypt data on-disk,

the hardware also includes a level of indirection; it doesn't use the unwrapped

key directly for inline encryption, but rather derives both an inline encryption

key and a "software secret" from it.  Software can use the "software secret" for

tasks that can't use the inline encryption hardware, such as filenames

encryption.  The software secret is not protected from memory compromise.

Key hierarchy

-------------

Here is the key hierarchy for a hardware-wrapped key::

                       Hardware-wrapped key

                                |

                                |

                          <Hardware KDF>

                                |

                  -----------------------------

                  |                           |

        Inline encryption key           Software secret

The components are:

- *Hardware-wrapped key*: a key for the hardware's KDF (Key Derivation

  Function), in ephemerally-wrapped form.  The key wrapping algorithm is a

  hardware implementation detail that doesn't impact kernel operation, but a

  strong authenticated encryption algorithm such as AES-256-GCM is recommended.

- *Hardware KDF*: a KDF (Key Derivation Function) which the hardware uses to

  derive subkeys after unwrapping the wrapped key.  The hardware's choice of KDF

  doesn't impact kernel operation, but it does need to be known for testing

  purposes, and it's also assumed to have at least a 256-bit security strength.

  All known hardware uses the SP800-108 KDF in Counter Mode with AES-256-CMAC,

  with a particular choice of labels and contexts; new hardware should use this

  already-vetted KDF.

- *Inline encryption key*: a derived key which the hardware directly provisions

  to a keyslot of the inline encryption hardware, without exposing it to

  software.  In all known hardware, this will always be an AES-256-XTS key.

  However, in principle other encryption algorithms could be supported too.

  Hardware must derive distinct subkeys for each supported encryption algorithm.

- *Software secret*: a derived key which the hardware returns to software so

  that software can use it for cryptographic tasks that can't use inline

  encryption.  This value is cryptographically isolated from the inline

  encryption key, i.e. knowing one doesn't reveal the other.  (The KDF ensures

  this.)  Currently, the software secret is always 32 bytes and thus is suitable

  for cryptographic applications that require up to a 256-bit security strength.

  Some use cases (e.g. full-disk encryption) won't require the software secret.

Example: in the case of fscrypt, the fscrypt master key (the key that protects a

particular set of encrypted directories) is made hardware-wrapped.  The inline

encryption key is used as the file contents encryption key, while the software

secret (rather than the master key directly) is used to key fscrypt's KDF

(HKDF-SHA512) to derive other subkeys such as filenames encryption keys.

Note that currently this design assumes a single inline encryption key per

hardware-wrapped key, without any further key derivation.  Thus, in the case of

fscrypt, currently hardware-wrapped keys are only compatible with the "inline

encryption optimized" settings, which use one file contents encryption key per

encryption policy rather than one per file.  This design could be extended to

make the hardware derive per-file keys using per-file nonces passed down the

storage stack, and in fact some hardware already supports this; future work is

planned to remove this limitation by adding the corresponding kernel support.

Kernel support

--------------

The inline encryption support of the kernel's block layer ("blk-crypto") has

been extended to support hardware-wrapped keys as an alternative to raw keys,

when hardware support is available.  This works in the following way:

- A ``key_types_supported`` field is added to the crypto capabilities in

  ``struct blk_crypto_profile``.  This allows device drivers to declare that

  they support raw keys, hardware-wrapped keys, or both.

- ``struct blk_crypto_key`` can now contain a hardware-wrapped key as an

  alternative to a raw key; a ``key_type`` field is added to

  ``struct blk_crypto_config`` to distinguish between the different key types.

  This allows users of blk-crypto to en/decrypt data using a hardware-wrapped

  key in a way very similar to using a raw key.

- A new method ``blk_crypto_ll_ops::derive_sw_secret`` is added.  Device drivers

  that support hardware-wrapped keys must implement this method.  Users of

  blk-crypto can call ``blk_crypto_derive_sw_secret()`` to access this method.

- The programming and eviction of hardware-wrapped keys happens via

  ``blk_crypto_ll_ops::keyslot_program`` and

  ``blk_crypto_ll_ops::keyslot_evict``, just like it does for raw keys.  If a

  driver supports hardware-wrapped keys, then it must handle hardware-wrapped

  keys being passed to these methods.

blk-crypto-fallback doesn't support hardware-wrapped keys.  Therefore,

hardware-wrapped keys can only be used with actual inline encryption hardware.

Testability

-----------

Both the hardware KDF and the inline encryption itself are well-defined

algorithms that don't depend on any secrets other than the unwrapped key.

Therefore, if the unwrapped key is known to software, these algorithms can be

reproduced in software in order to verify the ciphertext that is written to disk

by the inline encryption hardware.

However, the unwrapped key will only be known to software for testing if the

"import" functionality is used.  Proper testing is not possible in the

"generate" case where the hardware generates the key itself.  The correct

operation of the "generate" mode thus relies on the security and correctness of

the hardware RNG and its use to generate the key, as well as the testing of the

"import" mode as that should cover all parts other than the key generation.

For an example of a test that verifies the ciphertext written to disk in the

"import" mode, see the fscrypt hardware-wrapped key tests in xfstests, or

`Android's vts_kernel_encryption_test

<https://android.googlesource.com/platform/test/vts-testcase/kernel/+/refs/heads/main/encryption/>`_.

 * This is the key we set when evicting a keyslot. This *should* be the all 0's

 * key, but AES-XTS rejects that key, so we use some random bytes instead.

 */

static u8 blank_key[BLK_CRYPTO_MAX_KEY_SIZE];

static u8 blank_key[BLK_CRYPTO_MAX_RAW_KEY_SIZE];

static void blk_crypto_fallback_evict_keyslot(unsigned int slot)

{

		blk_crypto_fallback_evict_keyslot(slot);

	slotp->crypto_mode = crypto_mode;

	err = crypto_skcipher_setkey(slotp->tfms[crypto_mode], key->raw,

	err = crypto_skcipher_setkey(slotp->tfms[crypto_mode], key->bytes,

				     key->size);

	if (err) {

		blk_crypto_fallback_evict_keyslot(slot);

	if (blk_crypto_fallback_inited)

		return 0;

	get_random_bytes(blank_key, BLK_CRYPTO_MAX_KEY_SIZE);

	get_random_bytes(blank_key, sizeof(blank_key));

	err = bioset_init(&crypto_bio_split, 64, 0, 0);

	if (err)

	blk_crypto_fallback_profile->ll_ops = blk_crypto_fallback_ll_ops;

	blk_crypto_fallback_profile->max_dun_bytes_supported = BLK_CRYPTO_MAX_IV_SIZE;

	blk_crypto_fallback_profile->key_types_supported = BLK_CRYPTO_KEY_TYPE_RAW;

	/* All blk-crypto modes have a crypto API fallback. */

	for (i = 0; i < BLK_ENCRYPTION_MODE_MAX; i++)

	const char *name; /* name of this mode, shown in sysfs */

	const char *cipher_str; /* crypto API name (for fallback case) */

	unsigned int keysize; /* key size in bytes */

	unsigned int security_strength; /* security strength in bytes */

	unsigned int ivsize; /* iv size in bytes */

};

		return false;

	if (profile->max_dun_bytes_supported < cfg->dun_bytes)

		return false;

	if (!(profile->key_types_supported & cfg->key_type))

		return false;

	return true;

}

}

EXPORT_SYMBOL_GPL(blk_crypto_register);

/**

 * blk_crypto_derive_sw_secret() - Derive software secret from wrapped key

 * @bdev: a block device that supports hardware-wrapped keys

 * @eph_key: a hardware-wrapped key in ephemerally-wrapped form

 * @eph_key_size: size of @eph_key in bytes

 * @sw_secret: (output) the software secret

 *

 * Given a hardware-wrapped key in ephemerally-wrapped form (the same form that

 * it is used for I/O), ask the hardware to derive the secret which software can

 * use for cryptographic tasks other than inline encryption.  This secret is

 * guaranteed to be cryptographically isolated from the inline encryption key,

 * i.e. derived with a different KDF context.

 *

 * Return: 0 on success, -EOPNOTSUPP if the block device doesn't support

 *	   hardware-wrapped keys, -EBADMSG if the key isn't a valid

 *	   ephemerally-wrapped key, or another -errno code.

 */

int blk_crypto_derive_sw_secret(struct block_device *bdev,

				const u8 *eph_key, size_t eph_key_size,

				u8 sw_secret[BLK_CRYPTO_SW_SECRET_SIZE])

{

	struct blk_crypto_profile *profile =

		bdev_get_queue(bdev)->crypto_profile;

	int err;

	if (!profile)

		return -EOPNOTSUPP;

	if (!(profile->key_types_supported & BLK_CRYPTO_KEY_TYPE_HW_WRAPPED))

		return -EOPNOTSUPP;

	if (!profile->ll_ops.derive_sw_secret)

		return -EOPNOTSUPP;

	blk_crypto_hw_enter(profile);

	err = profile->ll_ops.derive_sw_secret(profile, eph_key, eph_key_size,

					       sw_secret);

	blk_crypto_hw_exit(profile);

	return err;

}

/**

 * blk_crypto_intersect_capabilities() - restrict supported crypto capabilities

 *					 by child device

			    child->max_dun_bytes_supported);

		for (i = 0; i < ARRAY_SIZE(child->modes_supported); i++)

			parent->modes_supported[i] &= child->modes_supported[i];

		parent->key_types_supported &= child->key_types_supported;

	} else {

		parent->max_dun_bytes_supported = 0;

		memset(parent->modes_supported, 0,

		       sizeof(parent->modes_supported));

		parent->key_types_supported = 0;

	}

}

EXPORT_SYMBOL_GPL(blk_crypto_intersect_capabilities);

	    target->max_dun_bytes_supported)

		return false;

	if (reference->key_types_supported & ~target->key_types_supported)

		return false;

	return true;

}

EXPORT_SYMBOL_GPL(blk_crypto_has_capabilities);

	       sizeof(dst->modes_supported));

	dst->max_dun_bytes_supported = src->max_dun_bytes_supported;

	dst->key_types_supported = src->key_types_supported;

}

EXPORT_SYMBOL_GPL(blk_crypto_update_capabilities);

		.name = "AES-256-XTS",

		.cipher_str = "xts(aes)",

		.keysize = 64,

		.security_strength = 32,

		.ivsize = 16,

	},

	[BLK_ENCRYPTION_MODE_AES_128_CBC_ESSIV] = {

		.name = "AES-128-CBC-ESSIV",

		.cipher_str = "essiv(cbc(aes),sha256)",

		.keysize = 16,

		.security_strength = 16,

		.ivsize = 16,

	},

	[BLK_ENCRYPTION_MODE_ADIANTUM] = {

		.name = "Adiantum",

		.cipher_str = "adiantum(xchacha12,aes)",

		.keysize = 32,

		.security_strength = 32,

		.ivsize = 32,

	},

	[BLK_ENCRYPTION_MODE_SM4_XTS] = {

		.name = "SM4-XTS",

		.cipher_str = "xts(sm4)",

		.keysize = 32,

		.security_strength = 16,

		.ivsize = 16,

	},

};

	/* This is assumed in various places. */

	BUILD_BUG_ON(BLK_ENCRYPTION_MODE_INVALID != 0);

	/* Sanity check that no algorithm exceeds the defined limits. */

	/*

	 * Validate the crypto mode properties.  This ideally would be done with

	 * static assertions, but boot-time checks are the next best thing.

	 */

	for (i = 0; i < BLK_ENCRYPTION_MODE_MAX; i++) {

		BUG_ON(blk_crypto_modes[i].keysize > BLK_CRYPTO_MAX_KEY_SIZE);

		BUG_ON(blk_crypto_modes[i].keysize >

		       BLK_CRYPTO_MAX_RAW_KEY_SIZE);

		BUG_ON(blk_crypto_modes[i].security_strength >

		       blk_crypto_modes[i].keysize);

		BUG_ON(blk_crypto_modes[i].ivsize > BLK_CRYPTO_MAX_IV_SIZE);

	}

/**

 * blk_crypto_init_key() - Prepare a key for use with blk-crypto

 * @blk_key: Pointer to the blk_crypto_key to initialize.

 * @raw_key: Pointer to the raw key. Must be the correct length for the chosen

 *	     @crypto_mode; see blk_crypto_modes[].

 * @key_bytes: the bytes of the key

 * @key_size: size of the key in bytes

 * @key_type: type of the key -- either raw or hardware-wrapped

 * @crypto_mode: identifier for the encryption algorithm to use

 * @dun_bytes: number of bytes that will be used to specify the DUN when this

 *	       key is used

 * @data_unit_size: the data unit size to use for en/decryption

 *

 * Return: 0 on success, -errno on failure.  The caller is responsible for

 *	   zeroizing both blk_key and raw_key when done with them.

 *	   zeroizing both blk_key and key_bytes when done with them.

 */

int blk_crypto_init_key(struct blk_crypto_key *blk_key, const u8 *raw_key,

int blk_crypto_init_key(struct blk_crypto_key *blk_key,

			const u8 *key_bytes, size_t key_size,

			enum blk_crypto_key_type key_type,

			enum blk_crypto_mode_num crypto_mode,

			unsigned int dun_bytes,

			unsigned int data_unit_size)

		return -EINVAL;

	mode = &blk_crypto_modes[crypto_mode];

	if (mode->keysize == 0)

	switch (key_type) {

	case BLK_CRYPTO_KEY_TYPE_RAW:

		if (key_size != mode->keysize)

			return -EINVAL;

		break;

	case BLK_CRYPTO_KEY_TYPE_HW_WRAPPED:

		if (key_size < mode->security_strength ||

		    key_size > BLK_CRYPTO_MAX_HW_WRAPPED_KEY_SIZE)

			return -EINVAL;

		break;

	default:

		return -EINVAL;

	}

	if (dun_bytes == 0 || dun_bytes > mode->ivsize)

		return -EINVAL;

	blk_key->crypto_cfg.crypto_mode = crypto_mode;

	blk_key->crypto_cfg.dun_bytes = dun_bytes;

	blk_key->crypto_cfg.data_unit_size = data_unit_size;

	blk_key->crypto_cfg.key_type = key_type;

	blk_key->data_unit_size_bits = ilog2(data_unit_size);

	blk_key->size = mode->keysize;

	memcpy(blk_key->raw, raw_key, mode->keysize);

	blk_key->size = key_size;

	memcpy(blk_key->bytes, key_bytes, key_size);

	return 0;

}

bool blk_crypto_config_supported(struct block_device *bdev,

				 const struct blk_crypto_config *cfg)

{

	return IS_ENABLED(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK) ||

	       blk_crypto_config_supported_natively(bdev, cfg);

	if (IS_ENABLED(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK) &&

	    cfg->key_type == BLK_CRYPTO_KEY_TYPE_RAW)

		return true;

	return blk_crypto_config_supported_natively(bdev, cfg);

}

/**

 * an skcipher, and *should not* be called from the data path, since that might

 * cause a deadlock

 *

 * Return: 0 on success; -ENOPKG if the hardware doesn't support the key and

 *	   blk-crypto-fallback is either disabled or the needed algorithm

 *	   is disabled in the crypto API; or another -errno code.

 * Return: 0 on success; -EOPNOTSUPP if the key is wrapped but the hardware does

 *	   not support wrapped keys; -ENOPKG if the key is a raw key but the

 *	   hardware does not support raw keys and blk-crypto-fallback is either

 *	   disabled or the needed algorithm is disabled in the crypto API; or

 *	   another -errno code if something else went wrong.

 */

int blk_crypto_start_using_key(struct block_device *bdev,

			       const struct blk_crypto_key *key)

{

	if (blk_crypto_config_supported_natively(bdev, &key->crypto_cfg))

		return 0;

	if (key->crypto_cfg.key_type != BLK_CRYPTO_KEY_TYPE_RAW) {

		pr_warn_ratelimited("%pg: no support for wrapped keys\n", bdev);

		return -EOPNOTSUPP;

	}

	return blk_crypto_fallback_start_using_mode(key->crypto_cfg.crypto_mode);

}