How to convert PromQL into Grafana API call

Requirement:

     We use promQL to fetch some metadata of a Kubernetes cluster. i.e existing namespaces

sum(kube_pod_info) by (namespace)

We would like to convert it to a grafana API call, so other apps can consume this metadata

Solution:

• First, we need to generate an API token. Refer grafana doc 
• Second, below is a curl example to consume it:

curl -k -H "Authorization: Bearer e*****dfwefwef0=" https://grafana-test.testtest.com/api/datasources/proxy/1/api/v1/query?query=sum\(kube_pod_info\)by\(namespace\)

Tip:Node and Namespace drop down menu missing node names in Grafana

 Symptom:

      We have Prometheus and Grafana setup running well. Suddenly the node and namespace drop-down list disappeared.  No config changes were made. 

Solution:

   It is very likely the kube-state-metrics service have problems. That's the place grafana get the info from.

   Bounce the pod or recreate the deployment to fix it

Sample Nginx Ingress Controller Integrate With Prometheus Grafana in OKE (Oracle Kubernetes Engine)

Requirement:

Ingress is proved to be very useful and efficient in Kubernetes world. The concept of ingress can be found official Kubernetes doc.  Ingress is similar to bigip on-premise. It provides rich functions for how to route and control ingress traffic in OKE. It is also adopted by the OKE team.

This note is based on https://kubernetes.github.io/ingress-nginx/   version 0.26

Kubernetes version needs to be at least v1.14.0

You would need cluster-admin role to proceed

Installation Steps:

• git clone https://github.com/HenryXie1/Prometheus-Granafa-Ingress-OKE.git
• cd  Prometheus-Granafa-Ingress-OKE/ingress-controllers/nginx
• kubectl create -f ingress-controller.yaml
• It will create internal Loadbalancer in OKE
• typical output is

  • kubectl get po -n ingress-nginx
    NAME                                       READY   STATUS    RESTARTS   AGE
    nginx-ingress-controller-d7976cdbd-d2zr6   1/1     Running   0          71m

    kubectl get svc -n ingress-nginx
    NAME            TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)                      AGE
    ingress-nginx   LoadBalancer   10.96.197.52   123.123.123.123   80:32155/TCP,443:31641/TCP   70m

Access Prometheus via Ingress Controller:

• About how to install Prometheus, please refer Install Prometheus and Grafana with High Availability in OKE (Oracle Kubernetes Engine)
• Steps of accessing Prometheus via ingress controller
  • Spoof IP address for DNS names via below entry and take off www proxy of the browser
    • 123.123.123.123         prometheus.bar.com  grafana.bar.com  alertmanager.bar.com
  • prometheus.bar.com
    • 
  • grafana.bar.com  
    • 
  • alertmanager.bar.com
    • 
• Ingress for Grafana

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: grafana-ingress
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      rules:
      - host: grafana.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: grafana
              servicePort: 3000

• Ingress for Alert manager

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: alertmanager-ingress
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      rules:
      - host: alertmanager.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: alertmanager-main
              servicePort: 9093

• Ingress for Prometheus

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: prometheus-ingress
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      rules:
      - host: prometheus.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: prometheus-k8s
              servicePort: 9090

  • 


Install Prometheus and Grafana with High Availability in OKE (Oracle Kubernetes Engine)

Requirement:

To monitor and get metrics of containerized environment managed by K8S, we are going to use Prometheus and Grafana. They can provide a visualized dashboard for K8S systems with useful charts.

This note is based on https://github.com/coreos/prometheus-operator  and  https://github.com/coreos/kube-prometheus

We also update the Prometheus kind to use storageclass "oci" where TSDB data of Prometheus would be stored
Prometheus is a statefulset with replicas = 2 by default which provides high availability

Kubernetes version needs to be at least v1.14.0

You would need cluster-admin role to proceed

Installation Steps:

• git clone https://github.com/HenryXie1/Prometheus-Granafa-Ingress-OKE.git
• cd  Prometheus-Granafa-Ingress-OKE
• kubectl create -f manifests/setup
• kubectl create -f manifests/
• Storage section of yaml to ask Prometheus to use block storage of OCI.   In the future, we need to adopt CSI for storageclass of OKE "oci-bv". 

  •     volumeClaimTemplate:
          spec:
            storageClassName: "oci"
            selector:
              matchLabels:
                app: prometheus
            resources:
              requests:
                storage: 100Gi

• Typical output is

  • $ kubectl get po -n monitoring
    NAME                                   READY   STATUS    RESTARTS   AGE
    alertmanager-main-0                    1/2     Running   9          35m
    alertmanager-main-1                    2/2     Running   0          35m
    alertmanager-main-2                    2/2     Running   0          23m
    grafana-65b66797b7-zdntc               1/1     Running   0          34m
    kube-state-metrics-6cf548479-w9dtq     3/3     Running   0          34m
    node-exporter-2kw4v                    2/2     Running   0          34m
    node-exporter-9wv7j                    2/2     Running   0          34m
    node-exporter-lphfg                    2/2     Running   0          34m
    node-exporter-s2f2f                    2/2     Running   0          34m
    prometheus-adapter-8bbfdc6db-6pnsk     1/1     Running   0          34m
    prometheus-k8s-0                       3/3     Running   0          34m
    prometheus-k8s-1                       3/3     Running   1          23m
    prometheus-operator-65fbfd78b8-7dq5r   1/1     Running   0          35m

Test Access the Dashboards

• Prometheus

$ kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

Then access via http://localhost:9090

• Grafana

$ kubectl --namespace monitoring port-forward svc/grafana 3000

Then access via http://localhost:3000 and use the default grafana user:password of admin:admin.

• Alert Manager

$ kubectl --namespace monitoring port-forward svc/alertmanager-main 9093

Then access via http://localhost:9093

Integrate Ingress with Prometheus :

Sample Nginx Ingress Controller Integrate With Prometheus Grafana in OKE (Oracle Kubernetes Engine)

Uninstallation Steps:

• kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup