DNS discovery for readiness in Kubernetes

This is a rather niche post, but over the last week I ran into an issue where I had two StatefulSets in Kubernetes that I was writing readiness checks for. This would allow the use of a RollingUpdate updateStrategy: an update of a StatefulSet would update one pod, wait for it to pass its readiness check, and then move onto to the next pod, repeating until all the pods were updated.

One of the caveats of readiness checks, however, is that Services won’t serve traffic to pods until they are ready. This didn’t work for me, because a particular quirk of the application I was working on is that it requires the StatefulSets to be able to ‘discover’ each other before they are happy. This is a straightforward deadlock: to be ready, the pods need to use DNS, but to use DNS, they need to be ready.

The workaround for this, which is simple but lightly documented, is to set publishNotReadyAddresses=True on the Services which point to the StatefulSet in question. This resolves the chicken-and-egg situation: the DNS records are created regardless of pod readiness, discovery can occur and the StatefulSets can set themselves up. None of this is rocket science, but hopefully I’ve stuffed enough keywords in here to make the fix easily discoverable for the next person to stumble upon this issue!

Written by Feroz Salam on 26 October 2019