Thursday, July 25, 2019

Error: cni config uninitialized when creating Kubernetes Cluster

Symptom:

When we create kubernetes cluster, we see below error in kubelet logs (journalctl -r -u kubelet)
docker can't pull any images from registry thus creation failed
Jul 18 06:13:17 oke-cytsnjqmizt-nsdomrwmnrt-sjr43hcwtea-0 kubelet[17065]: W0718 06:13:17.513278 17065 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 18 06:13:17 oke-cytsnjqmizt-nsdomrwmnrt-sjr43hcwtea-0 kubelet[17065]: E0718 06:13:17.515774 17065 kubelet.go:2167] Container runtime network
not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 18 06:13:22 oke-cytsnjqmizt-nsdomrwmnrt-sjr43hcwtea-0 kubelet[17065]: W0718 06:13:22.518341 17065 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 18 06:13:22 oke-cytsnjqmizt-nsdomrwmnrt-sjr43hcwtea-0 kubelet[17065]: E0718 06:13:22.519319 17065 kubelet.go:2167] Container runtime network
not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Solution:

It turns out that DNS servers have issues at that time, 2 of 3 DNS servers are not working well.
The last one is working well. We have to comment the 2 failed DNS servers in /etc/resolv.conf  of all worker nodes and leave good one in the resolv.conf.  After that, issue is gone

Another possible reason for this issue is:  If Pod security Policy is enabled for your kubernetes cluster, you need a policy to let system pods ie kube-dns or flannel...etc to be created in kube-system
Some quotes from  https://kubernetes.io/docs/concepts/policy/pod-security-policy/
Pod security policy control is implemented as an optional (but recommended) admission controller. PodSecurityPolicies are enforced by enabling the admission controller, but doing so without authorizing any policies will prevent any pods from being created in the cluster.

No comments: