Tuesday, February 04, 2020

Install Prometheus and Grafana with High Availability in OKE (Oracle Kubernetes Engine)

Requirement:

To monitor and get metrics of containerized environment managed by K8S, we are going to use Prometheus and Grafana. They can provide a visualized dashboard for K8S systems with useful charts.
We also update the Prometheus kind to use storageclass "oci" where TSDB data of Prometheus would be stored
Prometheus is a statefulset with replicas = 2 by default which provides high availability
Kubernetes version needs to be at least v1.14.0
You would need cluster-admin role to proceed

Installation Steps:

  • git clone https://github.com/HenryXie1/Prometheus-Granafa-Ingress-OKE.git
  • cd  Prometheus-Granafa-Ingress-OKE
  • kubectl create -f manifests/setup
  • kubectl create -f manifests/
  • Storage section of yaml to ask Prometheus to use block storage of OCI.   In the future, we need to adopt CSI for storageclass of OKE "oci-bv". 
    •     volumeClaimTemplate:
            spec:
              storageClassName: "oci"
              selector:
                matchLabels:
                  app: prometheus
              resources:
                requests:
                  storage: 100Gi
  • Typical output is
    • $ kubectl get po -n monitoring
      NAME                                   READY   STATUS    RESTARTS   AGE
      alertmanager-main-0                    1/2     Running   9          35m
      alertmanager-main-1                    2/2     Running   0          35m
      alertmanager-main-2                    2/2     Running   0          23m
      grafana-65b66797b7-zdntc               1/1     Running   0          34m
      kube-state-metrics-6cf548479-w9dtq     3/3     Running   0          34m
      node-exporter-2kw4v                    2/2     Running   0          34m
      node-exporter-9wv7j                    2/2     Running   0          34m
      node-exporter-lphfg                    2/2     Running   0          34m
      node-exporter-s2f2f                    2/2     Running   0          34m
      prometheus-adapter-8bbfdc6db-6pnsk     1/1     Running   0          34m
      prometheus-k8s-0                       3/3     Running   0          34m
      prometheus-k8s-1                       3/3     Running   1          23m
      prometheus-operator-65fbfd78b8-7dq5r   1/1     Running   0          35m

Test Access the Dashboards

  • Prometheus

$ kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
Then access via http://localhost:9090


  • Grafana

$ kubectl --namespace monitoring port-forward svc/grafana 3000
Then access via http://localhost:3000 and use the default grafana user:password of admin:admin.

  • Alert Manager

$ kubectl --namespace monitoring port-forward svc/alertmanager-main 9093
Then access via http://localhost:9093

Integrate Ingress with Prometheus :

Uninstallation Steps:

  • kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup

No comments: