CSI Driver vs. ESO

Before we dive into the tools, let’s briefly review why standard Kubernetes Secret objects are problematic for sensitive data: 

The Secrets Store CSI Driver is a specific implementation of the Container Storage Interface (CSI) standard for secrets. You install the driver on your cluster and configure a SecretProviderClass custom resource per application or namespace. This resource defines which secrets to retrieve from which external provider. In our case: Azure Key Vault.  

The CSI Driver offers two ways to make secrets available to your pod: 

Secrets are mounted as files in a (temporary, in-memory) volume, directly within the pod. The big advantage here is that no Kubernetes Secret object is ever created, so the sensitive data never enters the Kubernetes API or etcd. This is the most secure option, especially if you’re unsure about etcd encryption. The downside is that your application must be adapted to be able to read secrets from files. 

You could also configure it to create or update a standard Kubernetes Secret object with the values from Key Vault. You do this by adding the secretObjects property to the SecretProviderClass custom resource. This is useful for apps that expect environment variables (which you can mount from this secret), but it does mean the (Base64 encoded) secret will exist in the API and (encrypted in AKS) in the etcd. 

In both cases, secrets only become available after the pod has started and the CSI Driver has completed its work. Applications that need their secrets immediately at startup may therefore run into issues. A common workaround is to use init containers that wait until the secrets are available before launching the main application. However, this does add some extra complexity. 

The driver can also periodically check for updates in Key Vault using enableSecretRotation. If you’re using volume mounts, the files will be updated, but your application must be able to reload them. Syncing to a Kubernetes secret, that object will be updated accordingly.

The External Secrets Operator (ESO) takes a different approach. It’s a Kubernetes Operator that continuously monitors specific custom resources: 

ESO always syncs secrets to a standard Kubernetes Secret object; either by creating a new one or by updating an existing one. Unlike the CSI Driver, there’s no second option. That’s why it’s so important to have etcd encryption at rest enabled (although this is standard in AKS). 

Because the operator manages and syncs the Secret object independently of your pods, the secret is already available when the pod starts. Your application can immediately use environment variables mounted from this ESO-managed Kubernetes secret. That is also why we tend to use ESO more often in our AKS environments. 

Another tool you might come across is Sealed Secrets. It works differently: you encrypt your Kubernetes Secret manifest locally, using a public key. The result (a SealedSecret object) can be safely committed to Git. A controller in the cluster with the private key can then decrypt it back into a standard Secret. 

Sealed Secrets solves the issue of storing secrets in Git but doesn’t integrate with external managers like Key Vault for centralized management and rotation. It's a viable option if you don’t use an external vault, but it’s less powerful than CSI/ESO in that scenario. In other words: we don’t generally recommend it. 

Positively essential for both CSI Driver and ESO: to communicate safely with Azure Key Vault, we strongly recommend using Workload ID. It makes sure the sync tool itself doesn’t need any long-lived credentials to access Key Vault. 

It links the Service Account of the CSI Driver pod or the ESO Operator pod to an Azure Managed Identity with the appropriate permissions on Key Vault. This way, the entire chain – from external store to pod – works without secrets. You can find all the details in our two-part blog on Workload ID. 

For secrets that you can’t avoid using Workload ID, standard Kubernetes Secret objects are too risky and unsuitable for GitOps workflows. The best approach is to manage everything centrally in Azure Key Vault and sync it to your cluster, using the right tool. 

Both the Secrets Store CSI Driver and the External Secrets Operator (ESO) provide strong solutions, ideally combined with Workload ID for authentication to Key Vault. 

The CSI Driver offers maximum security with its volume mount option (no Kubernetes Secrets involved) but may be more complex due to application compatibility and startup timing issues. ESO is often easier to use and resolves the startup problem but always stores the secret as a(n encrypted in AKS) Kubernetes Secret. 

So, your choice depends on your balance between security, application requirements, and ease of use. At Lume, we typically use ESO, especially in managed AKS environments.