I’ve recently begun dipping my toes into the world of Kubernetes security,
which is a giant can of worms that can feel slightly overwhelming at first. One
tool which is particularly useful, at least as an introduction, is the
kube-bench tool by Aqua Security.
The tool can run against Kubernetes master and worker nodes, checking
how closely they match the CIS Kubernetes benchmark. This has been one starting
point for me as I try and understand the various security settings you can apply
to Kubernetes clusters and how they fit together.
I did run into a minor issue when setting up the benchmark on a test cluster and thought it might be worth a note for anyone who is similarly getting to grips with Kubernetes. When running in a Kubernetes cluster, the README for the software suggests that it be run against master nodes like so:
This didn’t work for me. The command hung, and going to the Kubernetes dashboard
displayed the error
No nodes are available that match all of the predicates:
MatchNodeSelector (58), PodToleratesNodeTaints (4).
The reason for this error (in my case) was the part of the
which runs the benchmarking software only on master nodes. The
command above looks for master nodes using the
nodeSelector syntax, which
in the above case has been told to look for the label
master. Except in the Kubernetes cluster I was working on, we
don’t have any nodes with that label - I’m not sure if this is a quirk of
the cluster I was working with or because the default settings have changed
since the README was written.
How did I identify and fix this? I ran
kubectl describe nodes (it’s worth
mentioning I have admin access to this cluster). This can output a huge amount
of information depending on your cluster size, so it might be worth storing
the output in a file.
I then grepped the output for any mentions of
master, and discovered that
all the nodes in the file which had the
tolerations key of
set also had the label key
type set, with a value of
controller. As an
optimistic punt, I changed the command above to:
In the general case, it doesn’t seem as if you can rely on nodes having
particular default labels. There’s more documentation about the
option here - it
does say that
The value of these [default] labels is cloud provider specific
and is not guaranteed to be reliable. So if you’re having trouble with a similar
error I would suggest following the same steps as me and figuring out what labels
are common to your master nodes (or creating a label that is common to them). I would
be interested to know if there is a more elegant general solution to running a container
across only master nodes, especially as I only have a basic working knowledge of Kubernetes
at the moment.