Change the Reclaim Policy of a PersistentVolume In OKE

Symptom:

  By default OKE(Oracle Kubernete Engine)  storageclass is oci which is OCI block volume
  If we don't specify storageclass in yaml file,  OKE would automatically create block volumes as persistent volumes and attach to pods for us which is very convenient.
 However reclaim policy of persistent volumes is "DELETE"
 It means if we delete pv and pvc OKE created, OKE would delete block volumes in OCI as well.

Solution:

  To prevent potential data loss due to reclaim policy " DELETE" , we can update it to be "RETAIN"

kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Please refer kubernete doc for more details

Docker Network Related Issue When We Build Docker Images

Symptom:

  When I ran "docker build -t oracle/database:19c  .  "  , it always error out on yum repo issues as below
https://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/repodata/repomd.xml: [Errno 12] Timeout on https://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/repodata/repomd.xml: (28, 'Resolving timed out after 30540 milliseconds')

Solution:

I checked around proxy settings according to note. I didn't find any issues.
I added proxy settings into Dockerfile like below to inform yum to use proxy
RUN echo "proxy=http://<proxy server IP address>:80" >> /etc/yum.conf
"docker login iad.ocir.io" was working fine.
DNS entries in /etc/resolv.conf  of the container had correct settings
I  restarted docker daemon , didn't work

After checking around, I suspected it was related to docker network
We created a new bridge network in docker via "docker network create  henrynetwork"
In this way, we can force docker build to use new bridge network instead of default one
 "docker build --network=henrynetwork -t oracle/database:19c  .  "
It works.  So far I don't have any clue why default bridge network won't work.  But new bridge network will work around it. 

Tip For Size of Oracle DB Tempfiles

Symptom:

   You will see big difference between ls -l and du -sm  on the same Oracle DB Tempfiles
ie
# ls -l o1_mf_temp_fpq46223_.dbf
-rw-r-----. 1 54321 54321 34358697984 Oct 14 06:21 o1_mf_temp_fpq46223_.dbf
--> 32G
# du -sm o1_mf_temp_fpq46223_.dbf
6       o1_mf_temp_fpq46223_.dbf
-->6M

Reason:

 ls -l  reads the attributions of files, so when the tempfile was created, it was created as 32G. So both allocated and unallocated together will be 32G.
However du is counting the actual number of allocated blocks.The unallocated blocks read as 0, take up no space on disk.
In this way, du is closer to the truth about how much space was used by DB.
ls shows us total size of allocated and unallocated blocks

Tips of How to Understand CustomResourceDefinition in K8S

In Kubernetes world, CustomResourceDefinition (CRD) will play a big role when we extend K8S cluster.  In the future, CRD may include even core K8S components like pod, deployment, statefulset......etc.  see CNCF youtube link
The basic logic is :

• We define what CRD we would like to use , ie mysql-operator in github, postgres-operator.... 

Once they are defined, they will be stored in Etcd.  They are purely text file, We can define whatever we want. K8S just regard them as text config file and put them in Etcd. K8S won't check if the CRD has been implemented or not. It won't error out if CRD is not implemented.  Related kubectl commands:

lubectl get crd <name>  -o yaml    kubectl api-resources   kubectl api-versions

• We implement the business logic via GO

This part will implement how we handle events related to the CRD we define.  K8S will send events to this part, this part will do business logic for the events. K8S cluster is events driven system. All events are put in different queues inside K8S core. 

• We create yaml files to use the CRD we define

We create yaml to generate some events (create, update, delete....)  to K8S,  our 2nd part above will receive these events and do the business logic we implement.

Fortunately kubebuilder  does lots of scaffolding work for us. More details Refer kubebuilder book

How To Put Binary or Text Config Files into K8s ConfigMap

Requirement:

  We often have all kind of config files for our Apps.  From binary wallet files to simple text config files, we need to store them somewhere where apps can access.  It is not a good idea to put these config files inside docker images or PV(persistent volume) as it would make the pod less portable and hard to scale and migrate. Fortunately from K8S v1.10 , Configmap would support both binary and text config files .  Here are some examples how we utilize configmap to achieve that. Mounted configmap are updated automatically. More details please refer kubernetes configmap doc

Solution:

We plan to create 2 configmaps . One for binary files like ewallet and one for text config files  like sysctl.conf, init.ora....

• Use kubectl create configmap walletconfigmap --from-file=ewallet1 --from-file=ewallet2
• Use kubectl create configmap textconfigmap --from-file=sysctl.conf  --from-file=jdbc.xml  --from-file=init.ora  
• Use kubectl get configmap walletconfigmap -o yaml . You would see ewallet would be stored as binaryData

apiVersion: v1

items:

- apiVersion: v1

  binaryData:

    ewallet: sdfsweffewg.....

........

• Mount wallet configmap as a volume in the pod with correct config file path . ie wallet file is on /etc/oracle/tde
• Mount text configmap as a volume in the pod with correct config file path . ie text file is on /opt/oracle/dbs

volumes:

- name: wallet-volume

  configMap:

          name: walletconfigmap 

- name: textconfig-volume

   configMap:

          name: textconfigmap 

Under container section of yaml file

volumeMounts:

- name: wallet-volume

  mountPath: /etc/oracle/tde

- name: textconfig-volume

  mountPath: /opt/oracle/dbs

• In this way, when we into the pod via kubectl exec -it  <pod name> /bin/bash , we would see ewallet1 ewallet2 in /etc/oracle/tde  and  sysctl.conf jdbc.xml and init.ora are in /opt/oracle/dbs.
• If we do some updates on configmap , K8S will sync them periodically.