This is a rather niche post, but over the last week I ran into an issue where
I had two
StatefulSets in Kubernetes that I was writing readiness checks for.
This would allow the use of a
an update of a
StatefulSet would update one pod, wait for it to pass its readiness
check, and then move onto to the next pod, repeating until all the pods were
One of the caveats of readiness checks, however, is that
Services won’t serve
traffic to pods until they are ready. This didn’t work for me, because a particular
quirk of the application I was working on is that it requires the
be able to ‘discover’ each other before they are happy. This is a straightforward deadlock:
to be ready, the pods need to use DNS, but to use DNS, they need to be ready.
The workaround for this,
which is simple but lightly documented, is to set
Services which point to the
StatefulSet in question. This resolves the chicken-and-egg
situation: the DNS records are created regardless of pod readiness, discovery can occur and
StatefulSets can set themselves up. None of this is rocket science, but hopefully I’ve
stuffed enough keywords in here to make the fix easily discoverable for the next person to
stumble upon this issue!