Edit

Validate and troubleshoot secret encryption on an AKS Edge Essentials cluster

Following Kubernetes security best practices, your secrets stored in the Kubernetes secret store on AKS Edge Essentials clusters are encrypted at rest by default using the Key Management Service (KMS) provider. Each secret within the secret store is first encrypted with a unique Data Encryption Key (DEK) generated by the Kubernetes API server. This DEK is temporarily held in memory and never written to disk. The API server then requests that the KMS provider encrypt the DEK using a Key Encryption Key (KEK). The KEK is automatically rotated every 30 days and protected with administrator credentials and is accessible only to administrators. The KMS provider can be enabled using the aksedge-config for product versions 1.10.868.0 and 1.11.247.0. Starting with AKS Edge Essentials version 1.12.269.0, the KMS provider is enabled by default for all newly created clusters.

For more detailed information about using KMS, see the official KMS provider documentation.

Prerequisites

The KMS provider is supported for all AKS Edge Essentials clusters, version 1.10.868.0 and later.

For more information about deploying a new AKS Edge Essentials cluster, see the official documentation.

Note

The KMS provider can only be used for single node clusters. The provider can't be used with experimental features, such as multi-node.

Verify that the KMS provider is healthy

This command can be used for AKS Edge Essentials version 1.10.868.0 and later.

To verify that the KMS provider is healthy, run the following command and ensure that the health status of kms-providers is OK:

kubectl get --raw='/readyz?verbose'

[+]ping ok
[+]Log ok
[+]etcd ok
[+]kms-providers ok
[+]poststarthook/start-encryption-provider-config-automatic-reload ok

To create secrets in AKS Edge Essentials clusters, see Managing Secrets using kubectl in the Kubernetes documentation.

If you encounter errors, see the Troubleshooting section.

How to update your secrets after KEK is rotated (optional)

The KEK is automatically updated every 30 days. At that point, the DEK encrypting each secret is securely encrypted with the KEK that was in use when you created the secret. When you next update the secret, the associated DEK is re-encrypted with the current KEK. If your workloads update secrets regularly, no additional action is required. If your secrets change infrequently, you may optionally choose to re‑write the same secret values every 30 days so the associated DEKs re‑encrypted with the latest KEKs. This is recommended as per Kubernetes best practices, not a requirement. Even without this step your secrets remain encrypted at rest. Use the command below if you wish to re-encrypt your secrets with the latest KEK:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Enable the KMS provider for versions 1.10.868.0 and 1.11.247.0

In your aksedge-config.json file, in the Init section, set Init.KmsPlugin.Enable to true:

"Init": {
 "KmsPlugin": {
     "Enable": true
  }
}

The following output is displayed during deployment, showing that the KMS provider is enabled:

Preparing to install kms-plugin as encryption provider...

Note

You can only enable or disable the KMS provider when you create a new deployment. Once you set the flag, it can't be changed.

Troubleshooting

If there are errors with the KMS provider, follow this procedure:

  1. Check that the AKS version is 1.10.868.0 or later. Use the following command to check the current version of AKS Edge Essentials:

    Get-Command -Module AKSEdge | Format-Table Name, Version

    If the version is older, upgrade to the latest version. For more information, see Upgrade an AKS cluster.

  2. View the readyz API. If the problem persists, verify that the KMS provider is healthy. For more information, see the Verify that the KMS provider is healthy section.

    If you receive "[-]" before the kms-providers field, collect diagnostic logs for debugging. For more information, see Get kubelet logs from cluster nodes.

  3. If there are still errors, the machine running the AKS Edge Essentials cluster might be paused or turned off for an extended period of time (over 30 days). To get KMS back into a healthy state, you can use the Repair-Kms command to restore any necessary tokens:

    Repair-AksEdgeKms

  4. If you still encounter errors, contact Microsoft Customer Support and collect logs.

Next steps

  • Last updated on