Thursday, November 10, 2022

Apex Ords Operator for Kubernetes


We often need to provision Apex and Ords for Dev, Stage, Prod. 
This is the operator to automate Apex Oracle Application Express 19.1 and Ords oracle rest data service via Kubernetes CRD, it creates a brand new Oracle 19c database statefulset, apex, ords deployment plus load balancer in the Kubernetes cluster


Full details and source codes are on GitHub repository


Tuesday, November 08, 2022

OKE Admission Control Webhook Sample


We need to implement a policy requested by the security team that Kubernetes service should have an annotation : None Thus no security list will be updated by Kubernetes. This is an example that how we build our own admission controller which implements various policies from security or other teams. ie we can add only internal load balancer is allowed for internal service.....etc


  • Please refer github repo
  • git clone
  • go build -o oke-admission-webhook
  • docker build --no-cache -t repo-url/oke-admission-webhook:v1 .
  • rm -rf oke-admission-webhook
  • docker push repo-url/oke-admission-webhook:v1
  • ./deployment/ --service oke-admission-webhook-svc --namespace kube-system --secret oke-admission-webhook-secret
  • kubectl replace --force -f deployment/validatingwebhook.yaml
  • kubectl replace --force -f deployment/deployment.yaml
  • kubectl replace --force -f deployment/service.yaml


Wednesday, February 09, 2022

Tip: kubectl apply --dry-run=client server RBAC role

 kubectl apply --dry-run is very useful to test manifests.

There are differences of RBAC requirement with kubectl apply --dry-run=client and server.

Both need a role in fetching CRD to go through the validation admission chain and the mutating admission chain.

We need READ ONLY  role for kubectl apply --dry-run=client and READ WRITE role for kubectl apply --dry-run=server

Wednesday, December 01, 2021

How to expose kube api server via nginx proxy


    Kubernetes API (Control Plane) are often sitting behind the firewall. To provide more security and load balancing, we need to set up an nginx proxy in front of them. There are 2 solutions.

Solution1: Use L4 TCP proxy pass of nginx

nginx stream core module provides L4 TCP UDP proxy pass functionalities. link
To proxy pass K8S API on port 6443 via nginx listening port 8888, we can implement the below code in nginx.conf:
stream {
    upstream api {
    server kubernetes.default.svc.cluster.local:6443;
server {
    listen 8888;
    proxy_timeout 20s;
    proxy_pass API;
kubeconfig has below elements:
  • the server is pointing nginx proxy ie
  • certificate-authority is the CA of K8S API CA( not the CA of )
  • client-certificate: path/to/my/client/cert
  • client-key: path/to/my/client/key

Solution2: Use L7 Https proxy pass of nginx

nginx HTTP core module provides L7 SSL proxy pass functionalities. link
To proxy pass K8S API on  via nginx listening 443 SSL, we can implement the below code in nginx.conf
http {
    upstream api {
    server {
      listen              443 ssl;
      ssl_certificate     /etc/nginx/ssl/tls.crt;
      ssl_certificate_key /etc/nginx/ssl/tls.key;
      location / {
        root /usr/local/nginx/html;
        index index.htm index.html;
      location /api/ {
        rewrite ^/api(/.*)$ $1 break;
        proxy_pass https://api;
kubeconfig has below elements:
  • the server is pointing nginx proxy ie
  • certificate-authority is the CA of (not K8S API CA)
  • can't use client-certificate and client-key like we do on L4 TCP proxy pass
  • Because TLS traffic to kube API server 6443 is regular anonymous TLS from nginx proxy, API server won't allow it. To solve it:
    • Option 1: use JWT token via OpenID connect
- name: testuser
        client-id: oidc-loginapp
        id-token: eyJhbGciOiJSUzI1NiIs....****
      name: oidc

    •  Option 2: Use mTLS and add client-certificate and client-key in the nginx proxy pass settings.

location /api/ {
        rewrite ^/api(/.*)$ $1 break;
        proxy_pass https://api;
        proxy_ssl_certificate         /etc/nginx/k8s-client-certificate.pem;
        proxy_ssl_certificate_key     /etc/nginx/k8s-client-key.key;
        proxy_ssl_session_reuse on;

Wednesday, November 17, 2021

Tip: kube-apiserver can't start after adding a parameter


We add a new oidc parameter for kube-apiserver to integrate with openID Dex.

The parameter is  --oidc-groups-prefix=oidc:

After that, kubelet can't start kube-apiserver static pod, and no obvious error reported


The issue is to ":"  which is special character. It prevents kubelet to parse the parameter.

The right way is to quote it.  "--oidc-groups-prefix=oidc:" See more details in this github thread