Tip: Pods keep crashloopbackoff

 Symptom:

 Pods always crashloopbackoff 

"kubectl describe pod..."  does not give meaningful info, as well as "kubectl get events"

Reason:

One of the likely reason is related to pod security policy. My situation is the existing pod security policy does not allow Nginx or Apache to run. It does not have

 allowedCapabilities:

  - NET_BIND_SERVICE

  # apache or nginx need escalation to root to function well

  allowPrivilegeEscalation: true

So the pods keep crashloopbackoff. To fix it is to add the above into the pod security policy.

Tip: Istio TLS secrets, Gateway, VirtualService namespace scope

There is some confusion about where we should put istio objects. Is it in the istio-system or users namespace?

Here are some tips:

For TLS,mTLS CA, certs, key management in istio, the Kubernetes secrets should be created in the istio-system. Not in users' namespace

Gateway and VirtualService need to be created on the users' namespace 

Tip: Slow Wi-FI Speed due to the second Monitor

 Symptom:

  The wifi speed dropped to half after we connected a second monitor for extended display.

Tip:

  It is likely the 2 monitors have different refresh rate,60 Hertz vs 59 Hertz. Make them the same could help.

How to find which type of VMs pods are running via promQL

Requirement:

     Users need to know which type of VMs their pods are running. i.e. users wanna verify pods are running on GPU VMs

Solution:

In Prometheus, we have 2 metrics:  kube_pod_info{} and kube_node_lables{}

kube_node_labels often has a label to tell which type of VM it is. 

We can use "node" to join these 2 metrics to provide a report to users

sum( kube_pod_info{}) by(pod,node) *on(node) group_left(label_beta_kubernetes_io_instance_type) sum(kube_node_labels{}) by (node,label_beta_kubernetes_io_instance_type)

Please refer official promQL doc 

Tip: create grafana API for it:

curl -g -k -H "Authorization: Bearer ******" https://grafana.testtest.com/api/datasources/proxy/1/api/v1/query?query=sum\(kube_pod_info{}\)by\(pod,node\)*on\(node\)group_left\(label_beta_kubernetes_io_instance_type\)sum\(kube_node_labels{}\)by\(node,label_beta_kubernetes_io_instance_type\)

Als refer my blog how to convert promQL into grafana API call

How to convert PromQL into Grafana API call

Requirement:

     We use promQL to fetch some metadata of a Kubernetes cluster. i.e existing namespaces

sum(kube_pod_info) by (namespace)

We would like to convert it to a grafana API call, so other apps can consume this metadata

Solution:

• First, we need to generate an API token. Refer grafana doc 
• Second, below is a curl example to consume it:

curl -k -H "Authorization: Bearer e*****dfwefwef0=" https://grafana-test.testtest.com/api/datasources/proxy/1/api/v1/query?query=sum\(kube_pod_info\)by\(namespace\)