Encrypting Confidential Data at Rest
All of the APIs in Kubernetes that let you write persistent API resource data support at-rest encryption. For example, you can enable at-rest encryption for Secrets. This at-rest encryption is additional to any system-level encryption for the etcd cluster or for the filesystem(s) on hosts where you are running the kube-apiserver.
This page shows how to enable and configure encryption of API data at rest.
This task covers encryption for resource data stored using the Kubernetes API. For example, you can encrypt Secret objects, including the key-value data they contain.
If you want to encrypt data in filesystems that are mounted into containers, you instead need to either:
- use a storage integration that provides encrypted volumes
- encrypt the data within your own application
Before you begin
-
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:
-
This task assumes that you are running the Kubernetes API server as a static pod on each control plane node.
-
Your cluster's control plane must use etcd v3.x (major version 3, any minor version).
-
To encrypt a custom resource, your cluster must be running Kubernetes v1.26 or newer.
-
To use a wildcard to match resources, your cluster must be running Kubernetes v1.27 or newer.
kubectl version
.
Configuration and determining whether encryption at rest is already enabled
The kube-apiserver
process accepts an argument --encryption-provider-config
that controls how API data is encrypted in etcd.
The configuration is provided as an API named
EncryptionConfiguration
. An example configuration is provided below.
kube-apiserver
component cannot
decrypt data stored in the etcd.
Understanding the encryption at rest configuration
---
#
# CAUTION: this is an example configuration.
# Do not use this for your own cluster!
#
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps
- pandas.awesome.bears.example # a custom resource API
providers:
# This configuration does not provide data confidentiality. The first
# configured provider is specifying the "identity" mechanism, which
# stores resources as plain text.
#
- identity: {} # plain text, in other words NO encryption
- aesgcm:
keys:
- name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2
secret: dGhpcyBpcyBwYXNzd29yZA==
- aescbc:
keys:
- name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2
secret: dGhpcyBpcyBwYXNzd29yZA==
- secretbox:
keys:
- name: key1
secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
- resources:
- events
providers:
- identity: {} # do not encrypt Events even though *.* is specified below
- resources:
- '*.apps' # wildcard match requires Kubernetes 1.27 or later
providers:
- aescbc:
keys:
- name: key2
secret: c2VjcmV0IGlzIHNlY3VyZSwgb3IgaXMgaXQ/Cg==
- resources:
- '*.*' # wildcard match requires Kubernetes 1.27 or later
providers:
- aescbc:
keys:
- name: key3
secret: c2VjcmV0IGlzIHNlY3VyZSwgSSB0aGluaw==
Each resources
array item is a separate config and contains a complete configuration. The
resources.resources
field is an array of Kubernetes resource names (resource
or resource.group
)
that should be encrypted like Secrets, ConfigMaps, or other resources.
If custom resources are added to EncryptionConfiguration
and the cluster version is 1.26 or newer,
any newly created custom resources mentioned in the EncryptionConfiguration
will be encrypted.
Any custom resources that existed in etcd prior to that version and configuration will be unencrypted
until they are next written to storage. This is the same behavior as built-in resources.
See the Ensure all secrets are encrypted section.
The providers
array is an ordered list of the possible encryption providers to use for the APIs that you listed.
Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider
is the first provider, the first key is used for encryption.
Only one provider type may be specified per entry (identity
or aescbc
may be provided,
but not both in the same item).
The first provider in the list is used to encrypt resources written into the storage. When reading
resources from storage, each provider that matches the stored data attempts in order to decrypt the
data. If no provider can read the stored data due to a mismatch in format or secret key, an error
is returned which prevents clients from accessing that resource.
EncryptionConfiguration
supports the use of wildcards to specify the resources that should be encrypted.
Use '*.<group>
' to encrypt all resources within a group (for eg '*.apps
' in above example) or '*.*
'
to encrypt all resources. '*.
' can be used to encrypt all resource in the core group. '*.*
' will
encrypt all resources, even custom resources that are added after API server start.
resources
list's processing order and precedence
are determined by the order it's listed in the configuration.
Opting out of encryption for specific resources while wildcard is enabled can be achieved by adding a new
resources
array item with the resource name, followed by the providers
array item with the identity
provider.
For example, if '*.*
' is enabled and you want to opt-out encryption for the events
resource, add a new item
to the resources
array with events
as the resource name, followed by the providers array item with identity
.
The new item should look like this:
- resources:
- events
providers:
- identity: {}
Ensure that the new item is listed before the wildcard '*.*
' item in the resources array to give it precedence.
For more detailed information about the EncryptionConfiguration
struct, please refer to the
encryption configuration API.
Available providers
Before you configure encryption-at-rest for data in your cluster's Kubernetes API, you need to select which provider(s) you will use.
The following table describes each available provider.
Name | Encryption | Strength | Speed | Key length |
---|---|---|---|---|
identity | None | N/A | N/A | N/A |
Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written. Existing encrypted resources are not automatically overwritten with the plaintext data. The identity provider is the default if you do not specify otherwise. | ||||
aescbc | AES-CBC with PKCS#7 padding | Weak | Fast | 32-byte |
Not recommended due to CBC's vulnerability to padding oracle attacks. Key material accessible from control plane host. | ||||
aesgcm | AES-GCM with random nonce | Must be rotated every 200,000 writes | Fastest | 16, 24, or 32-byte |
Not recommended for use except when an automated key rotation scheme is implemented. Key material accessible from control plane host. | ||||
kms v1 (deprecated since Kubernetes v1.28) | Uses envelope encryption scheme with DEK per resource. | Strongest | Slow (compared to kms version 2) | 32-bytes |
Data is encrypted by data encryption keys (DEKs) using AES-GCM;
DEKs are encrypted by key encryption keys (KEKs) according to
configuration in Key Management Service (KMS).
Simple key rotation, with a new DEK generated for each encryption, and
KEK rotation controlled by the user.
Read how to configure the KMS V1 provider. |
||||
kms v2 (beta) | Uses envelope encryption scheme with DEK per API server. | Strongest | Fast | 32-bytes |
Data is encrypted by data encryption keys (DEKs) using AES-GCM; DEKs
are encrypted by key encryption keys (KEKs) according to configuration
in Key Management Service (KMS).
Kubernetes defaults to generating a new DEK at API server startup, which is then
reused for object encryption.
If you enable the KMSv2KDF
feature gate,
Kubernetes instead generates a new DEK per encryption from a secret seed.
Whichever approach you configure, the DEK or seed is also rotated whenever the KEK is rotated. A good choice if using a third party tool for key management. Available in beta from Kubernetes v1.27. Read how to configure the KMS V2 provider. |
||||
secretbox | XSalsa20 and Poly1305 | Strong | Faster | 32-byte |
Uses relatively new encryption technologies that may not be considered acceptable in environments that require high levels of review. Key material accessible from control plane host. |
The identity
provider is the default if you do not specify otherwise. The identity
provider does not
encrypt stored data and provides no additional confidentiality protection.
Key storage
Local key storage
Encrypting secret data with a locally managed key protects against an etcd compromise, but it fails to protect against a host compromise. Since the encryption keys are stored on the host in the EncryptionConfiguration YAML file, a skilled attacker can access that file and extract the encryption keys.
Managed (KMS) key storage
The KMS provider uses envelope encryption: Kubernetes encrypts resources using a data key, and then encrypts that data key using the managed encryption service. Kubernetes generates a unique data key for each resource. The API server stores an encrypted version of the data key in etcd alongside the ciphertext; when reading the resource, the API server calls the managed encryption service and provides both the ciphertext and the (encrypted) data key. Within the managed encryption service, the provider use a key encryption key to decipher the data key, deciphers the data key, and finally recovers the plain text. Communication between the control plane and the KMS requires in-transit protection, such as TLS.
Using envelope encryption creates dependence on the key encryption key, which is not stored in Kubernetes. In the KMS case, an attacker who intends to get unauthorised access to the plaintext values would need to compromise etcd and the third-party KMS provider.
Write an encryption configuration file
Create a new encryption configuration file. The contents should be similar to:
---
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps
- pandas.awesome.bears.example
providers:
- aescbc:
keys:
- name: key1
# See the following text for more details about the secret value
secret: <BASE 64 ENCODED SECRET>
- identity: {} # this fallback allows reading unencrypted secrets;
# for example, during initial migration
To create a new Secret, perform the following steps:
-
Generate a 32-byte random key and base64 encode it. If you're on Linux or macOS, run the following command:
head -c 32 /dev/urandom | base64
-
Place that value in the
secret
field of theEncryptionConfiguration
struct. -
Set the
--encryption-provider-config
flag on thekube-apiserver
to point to the location of the config file.You will need to mount the new encryption config file to the
kube-apiserver
static pod. Here is an example on how to do that:- Save the new encryption config file to
/etc/kubernetes/enc/enc.yaml
on the control-plane node. - Edit the manifest for the
kube-apiserver
static pod:/etc/kubernetes/manifests/kube-apiserver.yaml
similarly to this:
--- # # This is a fragment of a manifest for a static Pod. # Check whether this is correct for your cluster and for your API server. # apiVersion: v1 kind: Pod metadata: annotations: kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.20.30.40:443 creationTimestamp: null labels: app.kubernetes.io/component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver ... - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml # add this line volumeMounts: ... - name: enc # add this line mountPath: /etc/kubernetes/enc # add this line readOnly: true # add this line ... volumes: ... - name: enc # add this line hostPath: # add this line path: /etc/kubernetes/enc # add this line type: DirectoryOrCreate # add this line ...
- Save the new encryption config file to
-
Restart your API server.
kube-apiserver
can read it.
Reconfigure other control plane hosts
If you have multiple API servers in your cluster, you should deploy the changes in turn to each API server.
Make sure that you use the same encryption configuration on each control plane host.
Verify that newly written data is encrypted
Data is encrypted when written to etcd. After restarting your kube-apiserver
, any newly
created or updated Secret (or other resource kinds configured in EncryptionConfiguration
)
should be encrypted when stored.
To check this, you can use the etcdctl
command line
program to retrieve the contents of your secret data.
This example shows how to check this for encrypting the Secret API.
-
Create a new Secret called
secret1
in thedefault
namespace:kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
-
Using the
etcdctl
command line tool, read that Secret out of etcd:ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C
where
[...]
must be the additional arguments for connecting to the etcd server.For example:
ETCDCTL_API=3 etcdctl \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ get /registry/secrets/default/secret1 | hexdump -C
The output is similar to this (abbreviated):
00000000 2f 72 65 67 69 73 74 72 79 2f 73 65 63 72 65 74 |/registry/secret| 00000010 73 2f 64 65 66 61 75 6c 74 2f 73 65 63 72 65 74 |s/default/secret| 00000020 31 0a 6b 38 73 3a 65 6e 63 3a 61 65 73 63 62 63 |1.k8s:enc:aescbc| 00000030 3a 76 31 3a 6b 65 79 31 3a c7 6c e7 d3 09 bc 06 |:v1:key1:.l.....| 00000040 25 51 91 e4 e0 6c e5 b1 4d 7a 8b 3d b9 c2 7c 6e |%Q...l..Mz.=..|n| 00000050 b4 79 df 05 28 ae 0d 8e 5f 35 13 2c c0 18 99 3e |.y..(..._5.,...>| [...] 00000110 23 3a 0d fc 28 ca 48 2d 6b 2d 46 cc 72 0b 70 4c |#:..(.H-k-F.r.pL| 00000120 a5 fc 35 43 12 4e 60 ef bf 6f fe cf df 0b ad 1f |..5C.N`..o......| 00000130 82 c4 88 53 02 da 3e 66 ff 0a |...S..>f..| 0000013a
-
Verify the stored Secret is prefixed with
k8s:enc:aescbc:v1:
which indicates theaescbc
provider has encrypted the resulting data. Confirm that the key name shown inetcd
matches the key name specified in theEncryptionConfiguration
mentioned above. In this example, you can see that the encryption key namedkey1
is used inetcd
and inEncryptionConfiguration
. -
Verify the Secret is correctly decrypted when retrieved via the API:
kubectl get secret secret1 -n default -o yaml
The output should contain
mykey: bXlkYXRh
, with contents ofmydata
encoded using base64; read decoding a Secret to learn how to completely decode the Secret.
Ensure all relevant data are encrypted
It's often not enough to make sure that new objects get encrypted: you also want that encryption to apply to the objects that are already stored.
For this example, you have configured your cluster so that Secrets are encrypted on write. Performing a replace operation for each Secret will encrypt that content at rest, where the objects are unchanged.
You can make this change across all Secrets in your cluster:
# Run this as an administrator that can read and write all Secrets
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
The command above reads all Secrets and then updates them with the same data, in order to apply server side encryption.
If an error occurs due to a conflicting write, retry the command. It is safe to run that command more than once.
For larger clusters, you may wish to subdivide the Secrets by namespace, or script an update.
Rotating a decryption key
Changing a Secret without incurring downtime requires a multi-step operation, especially in
the presence of a highly-available deployment where multiple kube-apiserver
processes are running.
- Generate a new key and add it as the second key entry for the current provider on all servers
- Restart all
kube-apiserver
processes to ensure each server can decrypt using the new key - Make the new key the first entry in the
keys
array so that it is used for encryption in the config - Restart all
kube-apiserver
processes to ensure each server now encrypts using the new key - Run
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
to encrypt all existing Secrets with the new key - Remove the old decryption key from the config after you have backed up etcd with the new key in use and updated all Secrets
When running a single kube-apiserver
instance, step 2 may be skipped.
Configure automatic reloading
You can configure automatic reloading of encryption provider configuration.
That setting determines whether the
API server should
load the file you specify for --encryption-provider-config
only once at
startup, or automatically whenever you change that file. Enabling this option
allows you to change the keys for encryption at rest without restarting the
API server.
To allow automatic reloading, configure the API server to run with:
--encryption-provider-config-automatic-reload=true
What's next
- Read about decrypting data that are already stored at rest
- Learn more about the EncryptionConfiguration configuration API (v1).