Cross Namespace Ingress Usage Example in OKE

Requirement:

      The normal use case to create ingress is to create one in the application namespace where application services and TLS certificates/keys are sitting.

      In the enterprise world, the security team is not comfortable to store TLS private keys in the application namespace. TLS private keys need to be stored securely in the namespace of the ingress controller.   In this case, we need to create ingress in "ingress controller" namespace instead of the application namespace.   We need to find a way to let ingress in "ingress controller" namespace to point to services in the application namespace (cross namespace service ).  Below is the solution of how we can achieve that in OKE ( Oracle Kubernetes Engine).

Solution:

• Create TLS secrets in ingress controller namespace. Refer doc

  • $ openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout tls.key -out tls.crt  -config req.conf -extensions 'v3_req'
    
    req.conf:
    [req]
    distinguished_name = ingress_tls_prometheus_test
    x509_extensions = v3_req
    prompt = no
    [ingress_tls_prometheus_test]
    C = US
    ST = VA
    L = NY
    O = BAR
    OU = BAR
    CN = www.bar.com
    [v3_req]
    keyUsage = keyEncipherment, dataEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = prometheus.bar.com
    DNS.2 = grafana.bar.com
    DNS.3 = alertmanager.bar.com

  • kubectl create secret tls tls-prometheus-test  --key tls.key --cert tls.crt -n ingress-nginx

• The key to using services in different namespaces is ExternalName.  It is working in OKE, but may not be working other  Cloud providers. One of the externalname examples is:

  • apiVersion: v1
    kind: Service
    metadata:
      annotations:
      name: prometheus-k8s-svc
      namespace: ingress-nginx
    spec:
      externalName: prometheus-k8s.monitoring.svc.cluster.local
      ports:
      - port: 9090
        protocol: TCP
        targetPort: 9090
      type: ExternalName

• Create ingress in ingress controller namespace.

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: prometheus-ingress
      namespace: ingress-nginx
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      tls:
      - hosts:
        - prometheus.bar.com
        - grafana.bar.com
        - alertmanager.bar.com
        secretName: tls-prometheus-test
      rules:
      - host: prometheus.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: prometheus-k8s-svc
              servicePort: 9090
      - host: grafana.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: grafana-svc
              servicePort: 3000
      - host: alertmanager.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: alertmanager-main-svc
              servicePort: 9093