helmfile: understand (and visualize !) the order in which releases are deployed
4 min read
This article looks at helmfile, and how it determines the order in which releases are deployed. More importantly, it gives you a bash script you can use to discover (i.e. print to the console) this order, without having to do a full
Skip the first sections if you are already familiar with helmfile and only want the script !
Please, like this feature request if, like me, you would like a command to show the order !
🐙 ⇨ Recap' on helmfile ⇦ 🐙
If you work with Kubernetes and Helm charts, chances are you are already familiar with helmfile. It is
a declarative spec that brings additional functionality to Helm by allowing you to compose several charts together to create a comprehensive deployment artifact. In other words, it allows you to bundle charts together.
In short, you declare releases, that is instances of specific Helm charts that need to be deployed. By default, all releases are deployed in parallel. To change this default, you declare needs (
releases.needs): when a release A needs another release B, A is guaranteed to be deployed before B. This is very similar to the
depends_on in docker-compose.
The final order will be computed (and applied) when you run
helmfile sync or
helmfile apply, which installs/update those releases in a given kubernetes namespace.
Mastering the ordering of releases in big helmfiles is difficult, and it is easy to miss some
needs. Unfortunately, with helmfile the only way to get the final deployment order it to install everything (
apply) and see how it goes... Or is it ? (See the hack below)
How helmfile plans the deployment of releases
Internally, helmfile generates a DAG (Direct Acyclic Graph) based on the
needs. From there, it groups releases so that each release in a group can be deployed in parallel, and each subsequent group depends only on the groups coming before. In other words, releases in group 3 can only need releases in group 1-2, etc.
When those groups are formed, helmfile knows it can go one group at a time, executing in parallel
helm install for all the releases it contains.
If you run
helmfile sync without any constraint on parallelism, it is exactly what it does. However, it is possible to provide a threshold to the concurrency, using the
--concurrency N flag. When
N is smaller than the number of releases in a group, helmfile picks randomly N releases at a time, group by group.
This randomness comes from the way
range over maps is implemented in go:
When iterating over a map with a range loop, the iteration order is not specified and is not guaranteed to be the same from one iteration to the next. Since Go 1 the runtime randomizes map iteration order, as programmers relied on the stable iteration order of the previous implementation. src
How to get the deployment order without installing
helmfile sync or
helmfile apply with a log level
DEBUG, the groups are printed at the beginning. For example:
helmfile -log-level debug sync ... processing 3 groups of releases in this order: GROUP RELEASES 1 orwell, freud, bradburry, wells, weber 2 foo, bar 3 buzz ...
The trick is thus to trigger a
apply in debug mode, and stop (
ctrl+c) the command right after the groups are printed, but before any release gets deployed.
As it is a pain to do manually, here is a script to do the job. Note that you will need to be connected to a kubernetes cluster for it to succeed, even though nothing will be deployed (rest easy). If you are in doubt, run
kubectl login before running the script:
trap "exit 0" INT # ctrl+c is somewhat a regular exit trap "exit" TERM ERR trap "rm $file 2>/dev/null || true" EXIT # ensure a helmfile is present helmfile $@ list > $file # read helmfile output and stop when the info is displayed start=0 helmfile --log-level debug $@ sync 2>&1 | tee $file | while IFS='' read -r line do [[ $line =~ ^"processing "[0-9+]" groups of releases in this order:"$ ]] && start=1 if [[ "$line" == "processing releases in group 1"* ]]; then start=0 rm $file # all went fine kill -INT -$$ # send ctrl+c to the process, the only way to kill everything right away fi [[ "$start" == 1 ]] && echo "$line" done if [ -f $file ]; then # if the file is still present, it means the process # didn't complete as expected. Show the logs echo -e "An error occurred. Here is the full log:" cat $file fifile=$(mktemp -t hf)
Assuming the script is saved as
helmfile-dag.sh in the user directory and is executable, the following:
cd folder-with-helmfile ~/helmfile-dag.sh # will use helmfile.yaml in CWD
processing 3 groups of releases in this order: GROUP RELEASES 1 orwell, freud, bradburry, wells, weber 2 foo, bar 3 buzz
Note that it is possible to pass extra global options to helmfile, for example:
# process a helmfile not in CWD ~/helmfile-dag.sh --file folder-with-helmfile/helmfile.yaml
Written with ❤ by derlin, thank you for reading !