How to expose kube api server via nginx proxy

Requirement:

    Kubernetes API (Control Plane) are often sitting behind the firewall. To provide more security and load balancing, we need to set up an nginx proxy in front of them. There are 2 solutions.

Solution1: Use L4 TCP proxy pass of nginx

nginx stream core module provides L4 TCP UDP proxy pass functionalities. link
To proxy pass K8S API on port 6443 via nginx listening port 8888, we can implement the below code in nginx.conf:

stream {
    upstream api {
    server kubernetes.default.svc.cluster.local:6443;
    }
server {
    listen 8888;
    proxy_timeout 20s;
    proxy_pass API;
    }
}

kubeconfig has below elements:

• the server is pointing nginx proxy ie https://myapi.myk8s.com:8888
• certificate-authority is the CA of K8S API CA( not the CA of myapi.myk8s.com )
• client-certificate: path/to/my/client/cert
• client-key: path/to/my/client/key

Solution2: Use L7 Https proxy pass of nginx

nginx HTTP core module provides L7 SSL proxy pass functionalities. link
To proxy pass K8S API on https://myapi.myk8s.com/api/  via nginx listening 443 SSL, we can implement the below code in nginx.conf

http {

    upstream api {

        kubernetes.default.svc.cluster.local:6443;

    }

    server {

      listen              443 ssl;

      server_name         myapi.myk8s.com;

      ssl_certificate     /etc/nginx/ssl/tls.crt;

      ssl_certificate_key /etc/nginx/ssl/tls.key;

      location / {

        root /usr/local/nginx/html;

        index index.htm index.html;

      }

      location /api/ {

        rewrite ^/api(/.*)$ $1 break;

        proxy_pass https://api;

       

      }

    }

}

kubeconfig has below elements:

• the server is pointing nginx proxy ie https://myapi.myk8s.com/api/
• certificate-authority is the CA of myapi.myk8s.com (not K8S API CA)
• can't use client-certificate and client-key like we do on L4 TCP proxy pass
• Because TLS traffic to kube API server 6443 is regular anonymous TLS from nginx proxy, API server won't allow it. To solve it:
• Option 1: use JWT token via OpenID connect

users:

- name: testuser

  user:

    auth-provider:

      config:

        idp-issuer-url: https://openid.myk8s.com/dex

        client-id: oidc-loginapp

        id-token: eyJhbGciOiJSUzI1NiIs....****

      name: oidc

•  Option 2: Use mTLS and add client-certificate and client-key in the nginx proxy pass settings.

location /api/ {

        rewrite ^/api(/.*)$ $1 break;

        proxy_pass https://api;

        proxy_ssl_certificate         /etc/nginx/k8s-client-certificate.pem;

        proxy_ssl_certificate_key     /etc/nginx/k8s-client-key.key;

        proxy_ssl_session_reuse on;

      }

How to Segregate Applications in Kubernetes Cluster without Compromise Cluster-Admin Role

Requirement:

   In enterprise world, we often have a few applications running on same Kubernete cluster. Each application owners would like to operate actions on his own applications without interfering other applications.  We would not like to grant cluster-admin to application owners for security reasons. Meanwhile application owner would have fully privilege in their own application scope.
This is for Oracle DBA to better understand how Kubernetes RBAC works. They both have similar RBAC concepts

Oracle Database	Kubernetes
dba role	cluster-admin role
grant dba role	grant cluster-admin role
create apps-user role to access tablespace example only	create apps-user role to access namespace example only
create apps-user	create apps-user or service account
grant apps-user role to apps-user	role-binding apps-user role to apps-user
apps-users work happily in tablespace example	apps-users work happily in namespace example

Solution:

• Create namespace for each application

kubectl create namespace  test-apps-ns

• Cluster admin create  role, serviceaccount, rolebinding for each application . Below is an example yaml file

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace:  test-apps-ns
  name: test-role
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: oke-test-user
  namespace:  test-apps-ns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace:  test-apps-ns
  name: test-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: test-role
subjects:
- kind: ServiceAccount
  name: oke-test-user
  namespace:  test-apps-ns