Install Prometheus and Grafana with High Availability in OKE (Oracle Kubernetes Engine)

Requirement:

To monitor and get metrics of containerized environment managed by K8S, we are going to use Prometheus and Grafana. They can provide a visualized dashboard for K8S systems with useful charts.

This note is based on https://github.com/coreos/prometheus-operator  and  https://github.com/coreos/kube-prometheus

We also update the Prometheus kind to use storageclass "oci" where TSDB data of Prometheus would be stored
Prometheus is a statefulset with replicas = 2 by default which provides high availability

Kubernetes version needs to be at least v1.14.0

You would need cluster-admin role to proceed

Installation Steps:

• git clone https://github.com/HenryXie1/Prometheus-Granafa-Ingress-OKE.git
• cd  Prometheus-Granafa-Ingress-OKE
• kubectl create -f manifests/setup
• kubectl create -f manifests/
• Storage section of yaml to ask Prometheus to use block storage of OCI.   In the future, we need to adopt CSI for storageclass of OKE "oci-bv". 

  •     volumeClaimTemplate:
          spec:
            storageClassName: "oci"
            selector:
              matchLabels:
                app: prometheus
            resources:
              requests:
                storage: 100Gi

• Typical output is

  • $ kubectl get po -n monitoring
    NAME                                   READY   STATUS    RESTARTS   AGE
    alertmanager-main-0                    1/2     Running   9          35m
    alertmanager-main-1                    2/2     Running   0          35m
    alertmanager-main-2                    2/2     Running   0          23m
    grafana-65b66797b7-zdntc               1/1     Running   0          34m
    kube-state-metrics-6cf548479-w9dtq     3/3     Running   0          34m
    node-exporter-2kw4v                    2/2     Running   0          34m
    node-exporter-9wv7j                    2/2     Running   0          34m
    node-exporter-lphfg                    2/2     Running   0          34m
    node-exporter-s2f2f                    2/2     Running   0          34m
    prometheus-adapter-8bbfdc6db-6pnsk     1/1     Running   0          34m
    prometheus-k8s-0                       3/3     Running   0          34m
    prometheus-k8s-1                       3/3     Running   1          23m
    prometheus-operator-65fbfd78b8-7dq5r   1/1     Running   0          35m

Test Access the Dashboards

• Prometheus

$ kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

Then access via http://localhost:9090

• Grafana

$ kubectl --namespace monitoring port-forward svc/grafana 3000

Then access via http://localhost:3000 and use the default grafana user:password of admin:admin.

• Alert Manager

$ kubectl --namespace monitoring port-forward svc/alertmanager-main 9093

Then access via http://localhost:9093

Integrate Ingress with Prometheus :

Sample Nginx Ingress Controller Integrate With Prometheus Grafana in OKE (Oracle Kubernetes Engine)

Uninstallation Steps:

• kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup