How to expose kube api server via nginx proxy

Requirement:

    Kubernetes API (Control Plane) are often sitting behind the firewall. To provide more security and load balancing, we need to set up an nginx proxy in front of them. There are 2 solutions.

Solution1: Use L4 TCP proxy pass of nginx

nginx stream core module provides L4 TCP UDP proxy pass functionalities. link
To proxy pass K8S API on port 6443 via nginx listening port 8888, we can implement the below code in nginx.conf:

stream {
    upstream api {
    server kubernetes.default.svc.cluster.local:6443;
    }
server {
    listen 8888;
    proxy_timeout 20s;
    proxy_pass API;
    }
}

kubeconfig has below elements:

• the server is pointing nginx proxy ie https://myapi.myk8s.com:8888
• certificate-authority is the CA of K8S API CA( not the CA of myapi.myk8s.com )
• client-certificate: path/to/my/client/cert
• client-key: path/to/my/client/key

Solution2: Use L7 Https proxy pass of nginx

nginx HTTP core module provides L7 SSL proxy pass functionalities. link
To proxy pass K8S API on https://myapi.myk8s.com/api/  via nginx listening 443 SSL, we can implement the below code in nginx.conf

http {

    upstream api {

        kubernetes.default.svc.cluster.local:6443;

    }

    server {

      listen              443 ssl;

      server_name         myapi.myk8s.com;

      ssl_certificate     /etc/nginx/ssl/tls.crt;

      ssl_certificate_key /etc/nginx/ssl/tls.key;

      location / {

        root /usr/local/nginx/html;

        index index.htm index.html;

      }

      location /api/ {

        rewrite ^/api(/.*)$ $1 break;

        proxy_pass https://api;

       

      }

    }

}

kubeconfig has below elements:

• the server is pointing nginx proxy ie https://myapi.myk8s.com/api/
• certificate-authority is the CA of myapi.myk8s.com (not K8S API CA)
• can't use client-certificate and client-key like we do on L4 TCP proxy pass
• Because TLS traffic to kube API server 6443 is regular anonymous TLS from nginx proxy, API server won't allow it. To solve it:
• Option 1: use JWT token via OpenID connect

users:

- name: testuser

  user:

    auth-provider:

      config:

        idp-issuer-url: https://openid.myk8s.com/dex

        client-id: oidc-loginapp

        id-token: eyJhbGciOiJSUzI1NiIs....****

      name: oidc

•  Option 2: Use mTLS and add client-certificate and client-key in the nginx proxy pass settings.

location /api/ {

        rewrite ^/api(/.*)$ $1 break;

        proxy_pass https://api;

        proxy_ssl_certificate         /etc/nginx/k8s-client-certificate.pem;

        proxy_ssl_certificate_key     /etc/nginx/k8s-client-key.key;

        proxy_ssl_session_reuse on;

      }

Tip: kube-apiserver can't start after adding a parameter

 Symptom:

We add a new oidc parameter for kube-apiserver to integrate with openID Dex.

The parameter is  --oidc-groups-prefix=oidc:

After that, kubelet can't start kube-apiserver static pod, and no obvious error reported

Solution:

The issue is to ":"  which is special character. It prevents kubelet to parse the parameter.

The right way is to quote it.  "--oidc-groups-prefix=oidc:" See more details in this github thread

Tip: Error: vault kv get invalid character - in numeric literal

Symptom:

When we use vault kv get to fetch CA from pki endpoint, we hit error

 $ vault kv get pki/ca/pem
Error reading pki/ca/pem: invalid character '-' in numeric literal

Workaround:

The proper way to fetch it is:

 $ vault kv get -field=certificate pki/ca/pem

or

$ vault read -field=certificate pki/ca/pem

Error: invalid bearer token, oidc: verify token: oidc: expected audience

Symptom:

After we implemented dex + github via link. With example-app,we are able to get ID-token via http://127.0.0.1:5555/
With ID-token, we construct kubeconfig, but when we access k8s cluster we hit error:

error: You must be logged in to the server (Unauthorized)

In kube api server logs, we see error:

invalid bearer token, oidc: verify token: oidc: expected audience \"123456\" got [\"example-app\"]]"

 Triage:

Check payload and verify JWT ID-token on https://jwt.io

Check dex container logs 

Find similar issues in github link1 link2

Solution:

It turns out the client-id is not matched. 

The client-id set on K8S API server (--oidc-client-id) link   needs to match the client-id in example-app.

In above example, “123456” is the one I set on K8S API server, while client-id is “example-app”  in the example-app which caused the problem

Kubectl Plugin for Oracle Database

 Requirement:

We often need to provision new oracle databases for developers

This is the kubectl plugin to automate the creation of oracle database statefulset in the Kubernetes cluster

Solution:

Full details and source codes are on the GitHub repository

Demo: