Thursday, November 10, 2022

Apex Ords Operator for Kubernetes


We often need to provision Apex and Ords for Dev, Stage, Prod. 
This is the operator to automate Apex Oracle Application Express 19.1 and Ords oracle rest data service via Kubernetes CRD, it creates a brand new Oracle 19c database statefulset, apex, ords deployment plus load balancer in the Kubernetes cluster


Full details and source codes are on GitHub repository


Tuesday, November 08, 2022

OKE Admission Control Webhook Sample


We need to implement a policy requested by the security team that Kubernetes service should have an annotation : None Thus no security list will be updated by Kubernetes. This is an example that how we build our own admission controller which implements various policies from security or other teams. ie we can add only internal load balancer is allowed for internal service.....etc


  • Please refer github repo
  • git clone
  • go build -o oke-admission-webhook
  • docker build --no-cache -t repo-url/oke-admission-webhook:v1 .
  • rm -rf oke-admission-webhook
  • docker push repo-url/oke-admission-webhook:v1
  • ./deployment/ --service oke-admission-webhook-svc --namespace kube-system --secret oke-admission-webhook-secret
  • kubectl replace --force -f deployment/validatingwebhook.yaml
  • kubectl replace --force -f deployment/deployment.yaml
  • kubectl replace --force -f deployment/service.yaml


Wednesday, November 17, 2021

Tip: kube-apiserver can't start after adding a parameter


We add a new oidc parameter for kube-apiserver to integrate with openID Dex.

The parameter is  --oidc-groups-prefix=oidc:

After that, kubelet can't start kube-apiserver static pod, and no obvious error reported


The issue is to ":"  which is special character. It prevents kubelet to parse the parameter.

The right way is to quote it.  "--oidc-groups-prefix=oidc:" See more details in this github thread

Sunday, October 24, 2021

Tip: Error: vault kv get invalid character - in numeric literal


When we use vault kv get to fetch CA from pki endpoint, we hit error

 $ vault kv get pki/ca/pem
Error reading pki/ca/pem: invalid character '-' in numeric literal


The proper way to fetch it is:

 $ vault kv get -field=certificate pki/ca/pem


$ vault read -field=certificate pki/ca/pem

Wednesday, September 22, 2021

Error: invalid bearer token, oidc: verify token: oidc: expected audience


After we implemented dex + github via link. With example-app,we are able to get ID-token via
With ID-token, we construct kubeconfig, but when we access k8s cluster we hit error:

error: You must be logged in to the server (Unauthorized)

In kube api server logs, we see error:

invalid bearer token, oidc: verify token: oidc: expected audience \"123456\" got [\"example-app\"]]"


Check payload and verify JWT ID-token on

Check dex container logs 

Find similar issues in github link1 link2


It turns out the client-id is not matched. 

The client-id set on K8S API server (--oidc-client-id) link   needs to match the client-id in example-app.

In above example, “123456” is the one I set on K8S API server, while client-id is “example-app”  in the example-app which caused the problem