Tuesday, February 04, 2020

Install Prometheus and Grafana with High Availability in OKE (Oracle Kubernetes Engine)


To monitor and get metrics of containerized environment managed by K8S, we are going to use Prometheus and Grafana. They can provide a visualized dashboard for K8S systems with useful charts.
We also update the Prometheus kind to use storageclass "oci" where TSDB data of Prometheus would be stored
Prometheus is a statefulset with replicas = 2 by default which provides high availability
Kubernetes version needs to be at least v1.14.0
You would need cluster-admin role to proceed

Installation Steps:

  • git clone https://github.com/HenryXie1/Prometheus-Granafa-Ingress-OKE.git
  • cd  Prometheus-Granafa-Ingress-OKE
  • kubectl create -f manifests/setup
  • kubectl create -f manifests/
  • Storage section of yaml to ask Prometheus to use block storage of OCI.   In the future, we need to adopt CSI for storageclass of OKE "oci-bv". 
    •     volumeClaimTemplate:
              storageClassName: "oci"
                  app: prometheus
                  storage: 100Gi
  • Typical output is
    • $ kubectl get po -n monitoring
      NAME                                   READY   STATUS    RESTARTS   AGE
      alertmanager-main-0                    1/2     Running   9          35m
      alertmanager-main-1                    2/2     Running   0          35m
      alertmanager-main-2                    2/2     Running   0          23m
      grafana-65b66797b7-zdntc               1/1     Running   0          34m
      kube-state-metrics-6cf548479-w9dtq     3/3     Running   0          34m
      node-exporter-2kw4v                    2/2     Running   0          34m
      node-exporter-9wv7j                    2/2     Running   0          34m
      node-exporter-lphfg                    2/2     Running   0          34m
      node-exporter-s2f2f                    2/2     Running   0          34m
      prometheus-adapter-8bbfdc6db-6pnsk     1/1     Running   0          34m
      prometheus-k8s-0                       3/3     Running   0          34m
      prometheus-k8s-1                       3/3     Running   1          23m
      prometheus-operator-65fbfd78b8-7dq5r   1/1     Running   0          35m

Test Access the Dashboards

  • Prometheus

$ kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
Then access via http://localhost:9090

  • Grafana

$ kubectl --namespace monitoring port-forward svc/grafana 3000
Then access via http://localhost:3000 and use the default grafana user:password of admin:admin.

  • Alert Manager

$ kubectl --namespace monitoring port-forward svc/alertmanager-main 9093
Then access via http://localhost:9093

Integrate Ingress with Prometheus :

Uninstallation Steps:

  • kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup

No comments: