Apex Ords Operator for Kubernetes

Requirement:

We often need to provision Apex and Ords for Dev , Stage, Prod. 

This is operator to automate Apex Oracle Application Express 19.1 and Ords oracle rest data service via kubernetes CRD ,it create a brand new Oracle 19c database statefulset,apex, ords deployment plus load balancer in the kubernetes cluster

Solution:

Full details and source codes are on github repository

OKE Admission Control Webhook Sample

Requirement:

We need implement a policy requested by security team that kubernetes service should have an annotation : service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: None Thus no security list will be updated by kuerbnetes. This is an example that how we build our own admission controller which implements various policies from security or other teams. ie we can add only internal loadbalancer is allowed for internal service.....etc

Solution:

• Please refer github repo
• git clone https://github.com/HenryXie1/oke-admission-webhook
• go build -o oke-admission-webhook
• docker build --no-cache -t repo-url/oke-admission-webhook:v1 .
• rm -rf oke-admission-webhook
• docker push repo-url/oke-admission-webhook:v1
• ./deployment/webhook-create-signed-cert.sh --service oke-admission-webhook-svc --namespace kube-system --secret oke-admission-webhook-secret
• kubectl replace --force -f deployment/validatingwebhook.yaml
• kubectl replace --force -f deployment/deployment.yaml
• kubectl replace --force -f deployment/service.yaml

Tip: Pods are not created while deployment is created

Symptom:

  We have a normal deployment which was working fine. When we test it on a new Kubernetes cluster, the deployment is created well, but the pod is not created. No warning or error messages.

 "kubectl describe deployment"  does not show any hints. Pod security policy check is good, RBAC privilege check is good.

OldReplicaSets:    <none>

NewReplicaSet:     livesqlstg-admin-678df959b4 (0/1 replicas created)

Events:

  Type    Reason             Age   From                   Message

  ----    ------             ----  ----                   -------

  Normal  ScalingReplicaSet  16s   deployment-controller  Scaled up replica set livesqlstg-admin-678df959b4 to 1

Solution:

  The reason is we have resource quota implemented on the namespace. 

 spec:

  hard:

    configmaps: "10"

    limits.cpu: "10"

    limits.memory: 20Gi

    persistentvolumeclaims: "10"

    ....

By doing that, we need an additional resource section in the deployment yaml file.  ie

      resources:

              requests:

                  memory: "10Gi"

                  cpu: "1"

              limits:

                  memory: "10Gi"

                  cpu: "1"

 It would be good for Kubernetes to give users some warnings for that. 

Tip: No route to host issues in Kubernetes Pods

Symptom:

    We see intermittent the network issues in OKE (Oracle Kubernetes Engine). ingress controller pods have difficult to access other services.  We use curl to test the network port, we get an error like below:

 

$ curl -v telnet://10.244.97.24:9090

* Expire in 0 ms for 6 (transfer 0x560b9cdd7dd0)

*   Trying 10.244.97.24...

* TCP_NODELAY set

* Expire in 200 ms for 4 (transfer 0x560b9cdd7dd0)

* connect to 10.244.97.24 port 9090 failed: No route to host

Solution:

   There are quite a few reasons for that. Check my another blog 

   In this case, it is related to firewall ports open. 

  By default, the network team open all ingress and egress ports for the same worker nodes Subnet which means no firewall among all worker nodes.  However, it was set stateful.  As Kubernetes overly network heavily depends on UDP which is stateless, so we need to open ports as stateless. 

How To RMAN Backup Oracle Database 19c running in Kubernetes

Requirement:

   We have an Oracle Database 19c running in OKE( Oracle Kubernetes Engine). We would like to use rman to backup DB to Object storage of  Cloud. We use Oracle Cloud Infrasture (OCI) as an example. The same concept applied to other Clouds.

Steps:

• Create a docker image with python 3 and Oracle OCI CLI installed. Please refer official doc how to install Oracle OCI CLI. Also, Dockerfile can be found via  GitHub repo 
• Create a statefulset using the docker image. Yaml files can be found via GitHub Repo
• Download the rman backup module of OCI. link
• Follow the instructions of installation. link
• Attention: when we set up oci cli, the config file should not be in the docker image, but to the persistent block storage volume. ie /opt/oracle/diag/.oci/config and export OCI_CLI_CONFIG_FILE=/opt/oracle/diag/.oci/config
•  Attention: when we set up rman backup module and create wallet files,  all config files should not be put in the docker image, but to the persistent block storage volume. ie /opt/oracle/diag/
• java -jar oci_install.jar \
• -host https://objectstorage.us-phoenix-1.oraclecloud.com \
• -pvtKeyFile /opt/oracle/diag/.oci/testuser_ww-oci_api_key.pem \
• -pubFingerPrint 52:b6:0e:2e:***:a1 \
• -uOCID "ocid1.user.oc1..aaaaahjia***adfe" \
• -tOCID "ocid1.tenancy.oc1..aanh7gl5**dfe" \
• -walletDir /opt/oracle/diag/.oci/opc_wallet \
• -configFile /opt/oracle/diag/.oci/opc_wallet/opcAUTOCDB.ora \
• -libDir $ORACLE_HOME/lib \
• -bucket BUK-OBJECT-STORAGE-BAK-TEMP \
• -proxyHost yourproxy.com \
• -proxyPort 80
• Use java- jar oci_installer.jar -h for more details
• Tip:If you have libopc.so in place in $ORACLE_HOME/lib which is in docker image, we can ignore the warning of  downloading part of the process
• Tip: You can copy opc_wallet to other servers or OKE clusters without doing oci cli and java -jar oic_installer.jar steps .
• Tip: If you see error " KBHS-00713: HTTP client error '', check http_proxy and https_proxy settings. Rman backup to object storage module uses  HTTP HTTPS protocols. 
• Tip: If you see error " KBHS-01012: ORA-28759 occurred during wallet operation; WRL file:/home/oracle/opc_wallet ",  it maybe due to there are some old opc<sid>.ora config files in $ORACLE_HOME/dbs. DB always try to read the config file in ./dbs instead of using parameters. Remove the files should clear it
• To avoid error "KBHS-01006: Parameter OPC_HOST was not specified", we need to put all parameters in opcAUTOCDB.ora in the rman script.
• Test RMAN backup inside your statefulset DB pod
• rman target /
• SET ENCRYPTION ON IDENTIFIED BY 'testtest' ONLY;
• run {
• SET ENCRYPTION ON IDENTIFIED BY 'changeme' ONLY;
• ALLOCATE CHANNEL t1 DEVICE TYPE sbt PARMS "SBT_LIBRARY=/opt/oracle/product/19c/dbhome_1/lib/libopc.so ENV=(OPC_HOST=https://objectstorage.us-phoenix-1.oraclecloud.com/n/testnamespace, OPC_WALLET='LOCATION=file:/opt/oracle/diag/.oci/opc_wallet CREDENTIAL_ALIAS=alias_oci', OPC_CONTAINER=TEST-OBJECT-STORAGE-RMAN, OPC_COMPARTMENT_ID=ocid1.compartment.oc1..aa****sddfeq, OPC_AUTH_SCHEME=BMC)";
• backup current controlfile;
• }