Apex Ords Operator for Kubernetes

Requirement:

We often need to provision Apex and Ords for Dev, Stage, Prod. 

This is the operator to automate Apex Oracle Application Express 19.1 and Ords oracle rest data service via Kubernetes CRD, it creates a brand new Oracle 19c database statefulset, apex, ords deployment plus load balancer in the Kubernetes cluster

Solution:

Full details and source codes are on GitHub repository

Demo:

OKE Admission Control Webhook Sample

Requirement:

We need to implement a policy requested by the security team that Kubernetes service should have an annotation : service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: None Thus no security list will be updated by Kubernetes. This is an example that how we build our own admission controller which implements various policies from security or other teams. ie we can add only internal load balancer is allowed for internal service.....etc

Solution:

• Please refer github repo
• git clone https://github.com/HenryXie1/oke-admission-webhook
• go build -o oke-admission-webhook
• docker build --no-cache -t repo-url/oke-admission-webhook:v1 .
• rm -rf oke-admission-webhook
• docker push repo-url/oke-admission-webhook:v1
• ./deployment/webhook-create-signed-cert.sh --service oke-admission-webhook-svc --namespace kube-system --secret oke-admission-webhook-secret
• kubectl replace --force -f deployment/validatingwebhook.yaml
• kubectl replace --force -f deployment/deployment.yaml
• kubectl replace --force -f deployment/service.yaml

Demo:

Istio install against different Docker Repos

Requirement:

       With istioctl, it has built-in manifests. However, these manifests or docker images may not be accessible in the corporate network, or users use other docker repo other than docker.io.  How to install it?

Solution:

• istioctl manifest generate --set profile=demo > istio_generate_manifests_demo.yaml
• find docker images path in the yaml ,download and upload them to your internal docker repo.
• edit the file with right docker image path of internal docker repo
• kubectl apply -f istio_generate_manifests_demo.yaml
• istioctl verify-install -f istio_generate_manifests_iad_demo.yaml
• to purge the deployment:
• istioctl x uninstall --purge

Tip: Pod FQDN in Kubernetes

Pods from deployment, statefulset. daemonset exposed by service

FQDN is  pod-ip-address.svc-name.my-namespace.svc.cluster.local

i.e  172-12-32-12.test-svc.test-namespace.svc.cluster.local

not 172.12.32.12.test-svc.test-namespace.svc.cluster.local

Isolated Pods:

FQDN is  pod-ip-address.my-namespace.pod.cluster.local

i.e  172-12-32-12.test-namespace.pod.cluster.local

Tip: Kubernetes intermittent DNS issues of pods

 Symptom:

     The pods get "unknown name" or "no such host" for the external domain name. i.e. test.testcorp.com

The issues are intermittent.

Actions:

• Follow k8s guide and check all  DNS pods are running well. 
• One possible reason is some of the nodes have network issues which DNS pods are not functioning well.  use below commands to test DNS pods. 

kubectl -n kube-system get po -owide|grep coredns |awk '{print $6 }' > /tmp/1.txt

cat /tmp/1.txt  | while read -r line; do echo $line | awk '{print "curl -v --connect-timeout 10 telnet://"$1":53", "\n"}'; done

• Enable debug log of DNS pods per  k8s guide
• test the DNS and kubectl tail all DNS pods to get debug info

kubectl -n kube-system logs -f deployment/coredns --all-containers=true --since=1m |grep testcorp

• You may get log like

INFO] 10.244.2.151:43653 - 48702 "AAAA IN test.testcorp.com.default.svc.cluster.local. udp 78 false 512" NXDOMAIN qr,aa,rd 171 0.000300408s

[INFO] 10.244.2.151:43653 - 64047 "A IN test.testcorp.com.default.svc.cluster.local. udp 78 false 512" NXDOMAIN qr,aa,rd 171 0.000392158s 

• The /etc/resolv.conf has  "options ndots:5"  which may impact the external domain DNS resolution. To use full qualified name can mitigate the issue. test.testcorp.com --> test.testcorp.com.  (there is a .  at the end)
• Disable coredns AAAA (IPv6) queries. it will reduce NXDOMAIN (not found), thus reduce the fail rate back to the dns client
• Add below into coredns config file. refer coredns rewrite
• rewrite stop type AAAA A
• Install node local DNS to speed DNS queries. Refer kubernetes doc
• Add below parameter in /etc/resolv.conf to improve DNS query performance
• options single-request-reopen   refer manual
• options single-request   refer manual

• Another solution is to use an external name:

  // code placeholder
  apiVersion: v1
  kind: Service
  metadata:
    annotations:
    name: test-stage
    namespace: default
  spec:
    externalName: test-stage.testcorp.com
    ports:
    - port: 636
      protocol: TCP
      targetPort: 636
    type: ExternalName