Requirement:
Kubernetes API (Control Plane) are often sitting behind the firewall. To provide more security and load balancing, we need to set up an nginx proxy in front of them. There are 2 solutions.Solution1: Use L4 TCP proxy pass of nginx
nginx stream core module provides L4 TCP UDP proxy pass functionalities. linkTo proxy pass K8S API on port 6443 via nginx listening port 8888, we can implement the below code in nginx.conf:
stream {kubeconfig has below elements:
upstream api {
server kubernetes.default.svc.cluster.local:6443;
}
server {
listen 8888;
proxy_timeout 20s;
proxy_pass API;
}
}
- the server is pointing nginx proxy ie https://myapi.myk8s.com:8888
- certificate-authority is the CA of K8S API CA( not the CA of myapi.myk8s.com )
- client-certificate: path/to/my/client/cert
- client-key: path/to/my/client/key
Solution2: Use L7 Https proxy pass of nginx
nginx HTTP core module provides L7 SSL proxy pass functionalities. link
To proxy pass K8S API on https://myapi.myk8s.com/api/ via nginx listening 443 SSL, we can implement the below code in nginx.conf
To proxy pass K8S API on https://myapi.myk8s.com/api/ via nginx listening 443 SSL, we can implement the below code in nginx.conf
http {upstream api {kubernetes.default.svc.cluster.local:6443;}server {listen 443 ssl;server_name myapi.myk8s.com;ssl_certificate /etc/nginx/ssl/tls.crt;ssl_certificate_key /etc/nginx/ssl/tls.key;location / {root /usr/local/nginx/html;index index.htm index.html;}location /api/ {rewrite ^/api(/.*)$ $1 break;proxy_pass https://api;}}}
kubeconfig has below elements:
- the server is pointing nginx proxy ie https://myapi.myk8s.com/api/
- certificate-authority is the CA of myapi.myk8s.com (not K8S API CA)
- can't use client-certificate and client-key like we do on L4 TCP proxy pass
- Because TLS traffic to kube API server 6443 is regular anonymous TLS from nginx proxy, API server won't allow it. To solve it:
- Option 1: use JWT token via OpenID connect
users:- name: testuseruser:auth-provider:config:idp-issuer-url: https://openid.myk8s.com/dexclient-id: oidc-loginappid-token: eyJhbGciOiJSUzI1NiIs....****name: oidc
- Option 2: Use mTLS and add client-certificate and client-key in the nginx proxy pass settings.
location /api/ {rewrite ^/api(/.*)$ $1 break;proxy_pass https://api;proxy_ssl_certificate /etc/nginx/k8s-client-certificate.pem;proxy_ssl_certificate_key /etc/nginx/k8s-client-key.key;proxy_ssl_session_reuse on;}