I recently had to test the new ExternalSecrets operator and its capabilities when using HashiCorp Vault as a backend. I spent some time figuring out how to install them on a local K3D cluster and wanted to share it so you won't have to.
IMPORTANT: this is for test purposes only, it is not suitable for production!
About Vault and ExternalSecrets
Secrets resources are a way to store sensitive information. Those Secret resources may be created directly (YAML files), from a Helm Chart, from Kustomize, etc. If you follow a gitops approach (you should!) those YAML files, Helm Charts, etc. will live in a git repo somewhere. But you don't want to commit sensitive information in git repos, so how to proceed?
A good approach instead is to use a secret management system (there are plenty to choose from: AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, etc.), and to have a way to retrieve those secrets dynamically from your Kubernetes cluster. This is where ExternalSecrets shines.
ExternalSecrets is a cluster-wide operator that you install once. Then, instead of creating a
Secret directly, you create an
ExternalSecret (a custom resource) that defines what secrets to retrieve, and from which backend. The operator then creates the
Secret for you.
Backends are configured by creating
ClusterSecretStore resources, which hold the connection information to a given secret management system. Each
ExternalSecret must reference one of those secret stores, so the operator knows from which backend it should retrieve secrets.
The documentation at https://external-secrets.io/main/ is quite good, so I will stop here.
Installing Vault and ExternalSecrets
To simplify the installation, I use helmfile, which uses helm under the hood.
The helm diff plugin (
helm plugin install https://github.com/databus23/helm-diff). This is necessary if you plan to use
k3d to be able to spawn a local Kubernetes cluster on Docker (
brew install k3d)
First, clone the following repo: https://github.com/derlin/externalsecrets-with-hashicorp-vault-kubernetes-easy-install.
Start a k3d cluster:
k3d cluster create test --api-port 6550 -p "80:80@loadbalancer"
Install Vault, the ExternalSecret operator, and a
ClusterSecretStore by running the following at the root of the repository:
The above command will:
Launch a Vault instance in the
vaultnamespace, and configure it with a token
Install the ExternalSecrets operator in the
defaultnamespace, which connects to the Vault. You can reference it in an ExternalSecret resource using:
apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret # ... spec: secretStoreRef: name: vault-backend kind: ClusterSecretStore # ...
Create a secret in the vault under the path
secret/foowith one property,
Done! Now, you can test the operator by creating an
ExternalSecret resource and wait for the
test to be created:
kubectl apply -f extsecret-example.yaml
Accessing the vault
The setting above automatically creates the
secret/foo for you. To access the vault interface and add more secrets, create a port forward to access the vault:
kubectl port-forward -n vault vault-0 8200
You can now go to localhost:8200 and log in with the default token
To access the vault using the command line (and assuming the port-forwarding is still on):
export VAULT_ADDR=http://127.0.0.1:8200 export VAULT_TOKEN=root vault kv get secret/foo
You can also set secrets programmatically using
kubectl exec vault-0 -n vault -- vault kv put secret/foo app-secret-key=123