Tip: Ngnix ingress controller can't startup

Symptom:

     We try to restart a pod of nginx ingress controller. After the restart, the pods can't startup 

Error like

status.go:274] updating Ingress ingress-nginx-internal/prometheus-ingress status from [] to [{100.114.90.8 }]

I1226 02:11:14.106423       6 event.go:255] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-nginx-internal", Name:"prometheus-ingress", UID:"e26f55f2-d87d-4efe-a4dd-5ae02768814a", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"46816813", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress-nginx-internal/prometheus-ingress

I1226 02:11:49.153889       6 main.go:153] Received SIGTERM, shutting down

I1226 02:11:49.153931       6 nginx.go:390] Shutting down controller queues

Workaround:

   Somehow the existing ingress rule "prometheus-ingress" is the cause. Remove the rule then the pod can startup well. We can add the rule back after that.

Tip: Error Http 504 gateway timeout on ingress controller

 Symptom:

    We have micro-services behind our ingress controller in our Kubernetes cluster. We are hitting HTTP 504 error in our ingress controller logs intermittently.

100.112.95.12 - - [01/Oct/2020:20:32:13 +0000] "GET /mos/products?limit=50&offset=0&orderBy=Name%3Aasc HTTP/2.0" 504 173 "https://ep******" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" 1578 180.004 [ingress-nginx-external2-mag-oke-products-svc-8080] [] 10.96.63.211:8080, 10.96.63.211:8080, 10.96.63.211:8080 0, 0, 0 60.001, 60.001, 60.002 504, 504, 504 c5b8cb67927d3997b4019e9830762694

  Bounce ingress controller would fix the issues temporarily.

Solution:

  We find the issues are caused parameters of nginx which stated

https://github.com/kubernetes/ingress-nginx/issues/4567

Add below annotations into ingress rules to fix it

nginx.ingress.kubernetes.io/proxy-connect-timeout: "5"

nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "10"

How To Add TLS Certificate Into Ingress in OKE

Requirement:

         In order to secure the traffic, we need to deploy  TLS certificates into our ingress running in OKE.  We are going to use self-signed certificates to demonstrate it. 

Solution:

• Generate self-signed certificates via openssl
  • openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout tls.key -out tls.crt  -config req.conf -extensions 'v3_req'

  • req.conf:
    
    [req]
    distinguished_name = ingress_tls_prometheus_test
    x509_extensions = v3_req
    prompt = no
    [ingress_tls_prometheus_test]
    C = US
    ST = VA
    L = NY
    O = BAR
    OU = BAR
    CN = www.bar.com
    [v3_req]
    keyUsage = keyEncipherment, dataEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = prometheus.bar.com
    DNS.2 = grafana.bar.com
    DNS.3 = alertmanager.bar.com

    
    
  • To verify it:    openssl x509 -in tls.crt -noout -text
• Create Kubernetes TLS secret for that
  • kubectl create secret tls tls-prometheus-test --key tls.key --cert tls.crt -n monitoring
• Add TLS section into the ingress yaml file. Example:

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: prometheus-ingress
      namespace: monitoring
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      tls:
      - hosts:
        - prometheus.bar.com
        secretName: tls-prometheus-test
      - hosts:
        - grafana.bar.com
        secretName: tls-prometheus-test
      - hosts:
        - alertmanager.bar.com
        secretName: tls-prometheus-test
      rules:
      - host: prometheus.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: prometheus-k8s
              servicePort: 9090
      - host: grafana.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: grafana
              servicePort: 3000
      - host: alertmanager.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: alertmanager-main
              servicePort: 9093

• Ingress controller would redirect http traffic to https traffic automatically for these 3 domains 
• Spoof IP address for DNS names via the below entry and take off www proxy of the browser if necessary.
  • 123.123.123.123         prometheus.bar.com  grafana.bar.com  alertmanager.bar.com

Cross Namespace Ingress Usage Example in OKE

Requirement:

      The normal use case to create ingress is to create one in the application namespace where application services and TLS certificates/keys are sitting.

      In the enterprise world, the security team is not comfortable to store TLS private keys in the application namespace. TLS private keys need to be stored securely in the namespace of the ingress controller.   In this case, we need to create ingress in "ingress controller" namespace instead of the application namespace.   We need to find a way to let ingress in "ingress controller" namespace to point to services in the application namespace (cross namespace service ).  Below is the solution of how we can achieve that in OKE ( Oracle Kubernetes Engine).

Solution:

• Create TLS secrets in ingress controller namespace. Refer doc

  • $ openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout tls.key -out tls.crt  -config req.conf -extensions 'v3_req'
    
    req.conf:
    [req]
    distinguished_name = ingress_tls_prometheus_test
    x509_extensions = v3_req
    prompt = no
    [ingress_tls_prometheus_test]
    C = US
    ST = VA
    L = NY
    O = BAR
    OU = BAR
    CN = www.bar.com
    [v3_req]
    keyUsage = keyEncipherment, dataEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = prometheus.bar.com
    DNS.2 = grafana.bar.com
    DNS.3 = alertmanager.bar.com

  • kubectl create secret tls tls-prometheus-test  --key tls.key --cert tls.crt -n ingress-nginx

• The key to using services in different namespaces is ExternalName.  It is working in OKE, but may not be working other  Cloud providers. One of the externalname examples is:

  • apiVersion: v1
    kind: Service
    metadata:
      annotations:
      name: prometheus-k8s-svc
      namespace: ingress-nginx
    spec:
      externalName: prometheus-k8s.monitoring.svc.cluster.local
      ports:
      - port: 9090
        protocol: TCP
        targetPort: 9090
      type: ExternalName

• Create ingress in ingress controller namespace.

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: prometheus-ingress
      namespace: ingress-nginx
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      tls:
      - hosts:
        - prometheus.bar.com
        - grafana.bar.com
        - alertmanager.bar.com
        secretName: tls-prometheus-test
      rules:
      - host: prometheus.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: prometheus-k8s-svc
              servicePort: 9090
      - host: grafana.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: grafana-svc
              servicePort: 3000
      - host: alertmanager.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: alertmanager-main-svc
              servicePort: 9093

Sample Nginx Ingress Controller Integrate With Prometheus Grafana in OKE (Oracle Kubernetes Engine)

Requirement:

Ingress is proved to be very useful and efficient in Kubernetes world. The concept of ingress can be found official Kubernetes doc.  Ingress is similar to bigip on-premise. It provides rich functions for how to route and control ingress traffic in OKE. It is also adopted by the OKE team.

This note is based on https://kubernetes.github.io/ingress-nginx/   version 0.26

Kubernetes version needs to be at least v1.14.0

You would need cluster-admin role to proceed

Installation Steps:

• git clone https://github.com/HenryXie1/Prometheus-Granafa-Ingress-OKE.git
• cd  Prometheus-Granafa-Ingress-OKE/ingress-controllers/nginx
• kubectl create -f ingress-controller.yaml
• It will create internal Loadbalancer in OKE
• typical output is

  • kubectl get po -n ingress-nginx
    NAME                                       READY   STATUS    RESTARTS   AGE
    nginx-ingress-controller-d7976cdbd-d2zr6   1/1     Running   0          71m

    kubectl get svc -n ingress-nginx
    NAME            TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)                      AGE
    ingress-nginx   LoadBalancer   10.96.197.52   123.123.123.123   80:32155/TCP,443:31641/TCP   70m

Access Prometheus via Ingress Controller:

• About how to install Prometheus, please refer Install Prometheus and Grafana with High Availability in OKE (Oracle Kubernetes Engine)
• Steps of accessing Prometheus via ingress controller
  • Spoof IP address for DNS names via below entry and take off www proxy of the browser
    • 123.123.123.123         prometheus.bar.com  grafana.bar.com  alertmanager.bar.com
  • prometheus.bar.com
    • 
  • grafana.bar.com  
    • 
  • alertmanager.bar.com
    • 
• Ingress for Grafana

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: grafana-ingress
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      rules:
      - host: grafana.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: grafana
              servicePort: 3000

• Ingress for Alert manager

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: alertmanager-ingress
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      rules:
      - host: alertmanager.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: alertmanager-main
              servicePort: 9093

• Ingress for Prometheus

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: prometheus-ingress
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      rules:
      - host: prometheus.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: prometheus-k8s
              servicePort: 9090

  • 


Install Prometheus and Grafana with High Availability in OKE (Oracle Kubernetes Engine)

Requirement:

To monitor and get metrics of containerized environment managed by K8S, we are going to use Prometheus and Grafana. They can provide a visualized dashboard for K8S systems with useful charts.

This note is based on https://github.com/coreos/prometheus-operator  and  https://github.com/coreos/kube-prometheus

We also update the Prometheus kind to use storageclass "oci" where TSDB data of Prometheus would be stored
Prometheus is a statefulset with replicas = 2 by default which provides high availability

Kubernetes version needs to be at least v1.14.0

You would need cluster-admin role to proceed

Installation Steps:

• git clone https://github.com/HenryXie1/Prometheus-Granafa-Ingress-OKE.git
• cd  Prometheus-Granafa-Ingress-OKE
• kubectl create -f manifests/setup
• kubectl create -f manifests/
• Storage section of yaml to ask Prometheus to use block storage of OCI.   In the future, we need to adopt CSI for storageclass of OKE "oci-bv". 

  •     volumeClaimTemplate:
          spec:
            storageClassName: "oci"
            selector:
              matchLabels:
                app: prometheus
            resources:
              requests:
                storage: 100Gi

• Typical output is

  • $ kubectl get po -n monitoring
    NAME                                   READY   STATUS    RESTARTS   AGE
    alertmanager-main-0                    1/2     Running   9          35m
    alertmanager-main-1                    2/2     Running   0          35m
    alertmanager-main-2                    2/2     Running   0          23m
    grafana-65b66797b7-zdntc               1/1     Running   0          34m
    kube-state-metrics-6cf548479-w9dtq     3/3     Running   0          34m
    node-exporter-2kw4v                    2/2     Running   0          34m
    node-exporter-9wv7j                    2/2     Running   0          34m
    node-exporter-lphfg                    2/2     Running   0          34m
    node-exporter-s2f2f                    2/2     Running   0          34m
    prometheus-adapter-8bbfdc6db-6pnsk     1/1     Running   0          34m
    prometheus-k8s-0                       3/3     Running   0          34m
    prometheus-k8s-1                       3/3     Running   1          23m
    prometheus-operator-65fbfd78b8-7dq5r   1/1     Running   0          35m

Test Access the Dashboards

• Prometheus

$ kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

Then access via http://localhost:9090

• Grafana

$ kubectl --namespace monitoring port-forward svc/grafana 3000

Then access via http://localhost:3000 and use the default grafana user:password of admin:admin.

• Alert Manager

$ kubectl --namespace monitoring port-forward svc/alertmanager-main 9093

Then access via http://localhost:9093

Integrate Ingress with Prometheus :

Sample Nginx Ingress Controller Integrate With Prometheus Grafana in OKE (Oracle Kubernetes Engine)

Uninstallation Steps:

• kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup