Apex Ords Operator for Kubernetes

Requirement:

We often need to provision Apex and Ords for Dev, Stage, Prod. 

This is the operator to automate Apex Oracle Application Express 19.1 and Ords oracle rest data service via Kubernetes CRD, it creates a brand new Oracle 19c database statefulset, apex, ords deployment plus load balancer in the Kubernetes cluster

Solution:

Full details and source codes are on GitHub repository

Demo:

OKE Admission Control Webhook Sample

Requirement:

We need to implement a policy requested by the security team that Kubernetes service should have an annotation : service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: None Thus no security list will be updated by Kubernetes. This is an example that how we build our own admission controller which implements various policies from security or other teams. ie we can add only internal load balancer is allowed for internal service.....etc

Solution:

• Please refer github repo
• git clone https://github.com/HenryXie1/oke-admission-webhook
• go build -o oke-admission-webhook
• docker build --no-cache -t repo-url/oke-admission-webhook:v1 .
• rm -rf oke-admission-webhook
• docker push repo-url/oke-admission-webhook:v1
• ./deployment/webhook-create-signed-cert.sh --service oke-admission-webhook-svc --namespace kube-system --secret oke-admission-webhook-secret
• kubectl replace --force -f deployment/validatingwebhook.yaml
• kubectl replace --force -f deployment/deployment.yaml
• kubectl replace --force -f deployment/service.yaml

Demo:

Tip: SSL handshake error when setting upsteam TLS in nginx

 Symptom:

     We hit various SSL handshake, connection, verification errors when we config upstream SSL in nginx following the guide

*1 upstream SSL certificate does not match "loginapp" while SSL handshaking to upstream, client: 110.126.174.93, server: test.com, request: "GET /auth/ HTTP/1.1", upstream: "https://198.19.163.33:5555/", host: "platform.nw-dev.kubernetes.cba", referrer: "https://platform.nw-dev.kubernetes.cba/"

2021/12/10 01:06:27 [error] 7#7: *1 upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream, client: 110.126.174.93, server: test.com, request: "GET /auth/ HTTP/1.1", upstream: "https://198.19.163.33:5555/", host: "test.com", ref

Diagnose:

It turns out the upstream in nginx config has the same DOMAIN requirement of HTTPS. For example:

upstream myapp{
    server myapp.svc.test.com:556;
}

location /myapp{
    proxy_pass https://myapp;
    proxy_ssl_trusted_certificate /etc/nginx/rootca/ClusterCA.pem;
    proxy_ssl_verify on;
    proxy_ssl_session_reuse on;
    }

The TLS certificate from the myapp.svc.test.com would need to have COMMON or DNS alias myapp in it, otherwise, the SSL handshake will fail. In other words, the upstream "myapp" needs to match the the TLS certificate CN in the myapp.svc.test.com. The tricky part is that the CN in the TLS certficate needs to match upstream, not the "myapp.svc.test.com"  ie below will work as long as TLS in random.example.com has "myapp1.svc.test.com" in it.

upstream myapp1.svc.test.com {
    server random.example.com:556;
}

To add another layer of security, to prevent man-in-the middle attack which host fake HTTPS for hacking purpose, we can set certain CA we trust for the upsteam server

Set  proxy_ssl_verify is on, the upsteam TLS certificate needs to be signed by the certain CA nginx can trust which is configured at proxy_ssl_trusted_certificate, otherwise, SSL verifies will fail.

To further harden security, we can add mTLS,example like below refer what is mTLS here 

location /myapp{
        proxy_pass https://backend.example.com;
        proxy_ssl_certificate /etc/nginx/client.pem;
        proxy_ssl_certificate_key /etc/nginx/client.key;
        proxy_ssl_trusted_certificate /etc/nginx/trusted_ca_cert.crt;
        proxy_ssl_verify on;
        proxy_ssl_session_reuse on;
    }

Refer more detailed info via nginx bug doc

How to expose kube api server via nginx proxy

Requirement:

    Kubernetes API (Control Plane) are often sitting behind the firewall. To provide more security and load balancing, we need to set up an nginx proxy in front of them. There are 2 solutions.

Solution1: Use L4 TCP proxy pass of nginx

nginx stream core module provides L4 TCP UDP proxy pass functionalities. link
To proxy pass K8S API on port 6443 via nginx listening port 8888, we can implement the below code in nginx.conf:

stream {
    upstream api {
    server kubernetes.default.svc.cluster.local:6443;
    }
server {
    listen 8888;
    proxy_timeout 20s;
    proxy_pass API;
    }
}

kubeconfig has below elements:

• the server is pointing nginx proxy ie https://myapi.myk8s.com:8888
• certificate-authority is the CA of K8S API CA( not the CA of myapi.myk8s.com )
• client-certificate: path/to/my/client/cert
• client-key: path/to/my/client/key

Solution2: Use L7 Https proxy pass of nginx

nginx HTTP core module provides L7 SSL proxy pass functionalities. link
To proxy pass K8S API on https://myapi.myk8s.com/api/  via nginx listening 443 SSL, we can implement the below code in nginx.conf

http {

    upstream api {

        kubernetes.default.svc.cluster.local:6443;

    }

    server {

      listen              443 ssl;

      server_name         myapi.myk8s.com;

      ssl_certificate     /etc/nginx/ssl/tls.crt;

      ssl_certificate_key /etc/nginx/ssl/tls.key;

      location / {

        root /usr/local/nginx/html;

        index index.htm index.html;

      }

      location /api/ {

        rewrite ^/api(/.*)$ $1 break;

        proxy_pass https://api;

       

      }

    }

}

kubeconfig has below elements:

• the server is pointing nginx proxy ie https://myapi.myk8s.com/api/
• certificate-authority is the CA of myapi.myk8s.com (not K8S API CA)
• can't use client-certificate and client-key like we do on L4 TCP proxy pass
• Because TLS traffic to kube API server 6443 is regular anonymous TLS from nginx proxy, API server won't allow it. To solve it:
• Option 1: use JWT token via OpenID connect

users:

- name: testuser

  user:

    auth-provider:

      config:

        idp-issuer-url: https://openid.myk8s.com/dex

        client-id: oidc-loginapp

        id-token: eyJhbGciOiJSUzI1NiIs....****

      name: oidc

•  Option 2: Use mTLS and add client-certificate and client-key in the nginx proxy pass settings.

location /api/ {

        rewrite ^/api(/.*)$ $1 break;

        proxy_pass https://api;

        proxy_ssl_certificate         /etc/nginx/k8s-client-certificate.pem;

        proxy_ssl_certificate_key     /etc/nginx/k8s-client-key.key;

        proxy_ssl_session_reuse on;

      }

Tip: kube-apiserver can't start after adding a parameter

 Symptom:

We add a new oidc parameter for kube-apiserver to integrate with openID Dex.

The parameter is  --oidc-groups-prefix=oidc:

After that, kubelet can't start kube-apiserver static pod, and no obvious error reported

Solution:

The issue is to ":"  which is special character. It prevents kubelet to parse the parameter.

The right way is to quote it.  "--oidc-groups-prefix=oidc:" See more details in this github thread