This is a rather niche post, but over the last week I ran into an issue where
I had two StatefulSets in Kubernetes that I was writing readiness checks for.
This would allow the use of a RollingUpdate updateStrategy:
an update of a StatefulSet would update one pod, wait for it to pass its readiness
check, and then move onto to the next pod, repeating until all the pods were
updated.
One of the caveats of readiness checks, however, is that Services won’t serve
traffic to pods until they are ready. This didn’t work for me, because a particular
quirk of the application I was working on is that it requires the StatefulSets to
be able to ‘discover’ each other before they are happy. This is a straightforward deadlock:
to be ready, the pods need to use DNS, but to use DNS, they need to be ready.
The workaround for this,
which is simple but lightly documented, is to set publishNotReadyAddresses=True
on the Services which point to the StatefulSet in question. This resolves the chicken-and-egg
situation: the DNS records are created regardless of pod readiness, discovery can occur and
the StatefulSets can set themselves up. None of this is rocket science, but hopefully I’ve
stuffed enough keywords in here to make the fix easily discoverable for the next person to
stumble upon this issue!