Error: cni config uninitialized when creating Kubernetes Cluster

Symptom:

When we create kubernetes cluster, we see below error in kubelet logs (journalctl -r -u kubelet)
docker can't pull any images from registry thus creation failed

Jul 18 06:13:17 oke-cytsnjqmizt-nsdomrwmnrt-sjr43hcwtea-0 kubelet[17065]: W0718 06:13:17.513278 17065 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 18 06:13:17 oke-cytsnjqmizt-nsdomrwmnrt-sjr43hcwtea-0 kubelet[17065]: E0718 06:13:17.515774 17065 kubelet.go:2167] Container runtime network
not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 18 06:13:22 oke-cytsnjqmizt-nsdomrwmnrt-sjr43hcwtea-0 kubelet[17065]: W0718 06:13:22.518341 17065 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 18 06:13:22 oke-cytsnjqmizt-nsdomrwmnrt-sjr43hcwtea-0 kubelet[17065]: E0718 06:13:22.519319 17065 kubelet.go:2167] Container runtime network
not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Solution:

It turns out that DNS servers have issues at that time, 2 of 3 DNS servers are not working well.
The last one is working well. We have to comment the 2 failed DNS servers in /etc/resolv.conf  of all worker nodes and leave good one in the resolv.conf.  After that, issue is gone

Another possible reason for this issue is:  If Pod security Policy is enabled for your kubernetes cluster, you need a policy to let system pods ie kube-dns or flannel...etc to be created in kube-system
Some quotes from  https://kubernetes.io/docs/concepts/policy/pod-security-policy/

Pod security policy control is implemented as an optional (but recommended) admission controller. PodSecurityPolicies are enforced by enabling the admission controller, but doing so without authorizing any policies will prevent any pods from being created in the cluster.

BPF Hello World Examples

What is BPF:

Refer from this doc
BPF is a highly flexible and efficient virtual machine-like construct in the Linux kernel allowing to execute bytecode at various hook points in a safe manner. It is used in a number of Linux kernel subsystems, most prominently networking, tracing and security (e.g. sandboxing).

BPF in the Linux kernel is allowing to execute bytecode at various hook points in a safe manner. It is used in a number of Linux kernel subsystems, most prominently networking, tracing and security (e.g. sandboxing).

Github BPF Hello World examples

Tip to Rolling Restart Kubernetes Deployment Statefulset Daemonset

From kubectl 1.15.0 , kubectl supports rolling restart Kubernetes Deployment Statefulset Daemonset.

kubectl rollout restart  deployment  <name>
kubectl rollout restart  statefulset    <name>
kubectl rollout restart  daemonset   <name>

Error:cannot list resource "deployments" in API group "apps" at the cluster scope

Symptom:

    We have operator running in the cluster, it error out when creating deployment. The error is like

cannot list resource "deployments" in API group "apps" at the cluster scope

Solution:

It is due to the clusterrole granted to the operator lack of permssion to create deployment.... We need to add such permission in the role as well as statefulsets, secrects ....... The sample of clusterrole is below

- apiGroups:
  - ""
  resources:
  - pods
  - secrets
  - services
  - configmaps
  verbs:
  - '*'
- apiGroups:
  - apps
  resources:
  - deployments
  - statefulsets
  verbs:
  - '*'

How To Run Docker Without Sudo

• sudo groupadd docker
• sudo usermod -aG docker <username>
• logout all sessions , not only terminals but also desktop
• login again
• to test:   docker run hello-world