How To RMAN Backup Oracle Database 19c running in Kubernetes

Requirement:

   We have an Oracle Database 19c running in OKE( Oracle Kubernetes Engine). We would like to use rman to backup DB to Object storage of  Cloud. We use Oracle Cloud Infrasture (OCI) as an example. The same concept applied to other Clouds.

Steps:

• Create a docker image with python 3 and Oracle OCI CLI installed. Please refer official doc how to install Oracle OCI CLI. Also, Dockerfile can be found via  GitHub repo 
• Create a statefulset using the docker image. Yaml files can be found via GitHub Repo
• Download the rman backup module of OCI. link
• Follow the instructions of installation. link
• Attention: when we set up oci cli, the config file should not be in the docker image, but to the persistent block storage volume. ie /opt/oracle/diag/.oci/config and export OCI_CLI_CONFIG_FILE=/opt/oracle/diag/.oci/config
•  Attention: when we set up rman backup module and create wallet files,  all config files should not be put in the docker image, but to the persistent block storage volume. ie /opt/oracle/diag/
• java -jar oci_install.jar \
• -host https://objectstorage.us-phoenix-1.oraclecloud.com \
• -pvtKeyFile /opt/oracle/diag/.oci/testuser_ww-oci_api_key.pem \
• -pubFingerPrint 52:b6:0e:2e:***:a1 \
• -uOCID "ocid1.user.oc1..aaaaahjia***adfe" \
• -tOCID "ocid1.tenancy.oc1..aanh7gl5**dfe" \
• -walletDir /opt/oracle/diag/.oci/opc_wallet \
• -configFile /opt/oracle/diag/.oci/opc_wallet/opcAUTOCDB.ora \
• -libDir $ORACLE_HOME/lib \
• -bucket BUK-OBJECT-STORAGE-BAK-TEMP \
• -proxyHost yourproxy.com \
• -proxyPort 80
• Use java- jar oci_installer.jar -h for more details
• Tip:If you have libopc.so in place in $ORACLE_HOME/lib which is in docker image, we can ignore the warning of  downloading part of the process
• Tip: You can copy opc_wallet to other servers or OKE clusters without doing oci cli and java -jar oic_installer.jar steps .
• Tip: If you see error " KBHS-00713: HTTP client error '', check http_proxy and https_proxy settings. Rman backup to object storage module uses  HTTP HTTPS protocols. 
• Tip: If you see error " KBHS-01012: ORA-28759 occurred during wallet operation; WRL file:/home/oracle/opc_wallet ",  it maybe due to there are some old opc<sid>.ora config files in $ORACLE_HOME/dbs. DB always try to read the config file in ./dbs instead of using parameters. Remove the files should clear it
• To avoid error "KBHS-01006: Parameter OPC_HOST was not specified", we need to put all parameters in opcAUTOCDB.ora in the rman script.
• Test RMAN backup inside your statefulset DB pod
• rman target /
• SET ENCRYPTION ON IDENTIFIED BY 'testtest' ONLY;
• run {
• SET ENCRYPTION ON IDENTIFIED BY 'changeme' ONLY;
• ALLOCATE CHANNEL t1 DEVICE TYPE sbt PARMS "SBT_LIBRARY=/opt/oracle/product/19c/dbhome_1/lib/libopc.so ENV=(OPC_HOST=https://objectstorage.us-phoenix-1.oraclecloud.com/n/testnamespace, OPC_WALLET='LOCATION=file:/opt/oracle/diag/.oci/opc_wallet CREDENTIAL_ALIAS=alias_oci', OPC_CONTAINER=TEST-OBJECT-STORAGE-RMAN, OPC_COMPARTMENT_ID=ocid1.compartment.oc1..aa****sddfeq, OPC_AUTH_SCHEME=BMC)";
• backup current controlfile;
• }

Dockerfile for Oracle Database 19.5 image with patches applied

Summary:

Here is the github link for Dockerfile of Oracle Database 19.5 image with patches applied

https://github.com/HenryXie1/Dockerfile/tree/master/OracleDatabase

The docker image has 19.3 installed and apply below patches to 19.5
OCT_RU_DB_p30125133_190000_Linux-x86-64.zip  OCT_RU_OJVM_p30128191_190000_Linux-x86-64.zip  
p30083488_195000DBRU_Linux-x86-64.zip

The docker image has updates to facilitate automated block storage provision in  OKE (Oracle Kubernetes Engine)

The docker image creates three different volumes for  Oradata,  Fast Recovery Area (FRA)  and Diagnose area (diag). The three would help to keep datafiles safe, dedicated space for recovery and separated place for diagnosing avoid filling up Data and FRA places.

The testdb yaml files utilize oci-bv (Container Storage Interface -- CSI based)  of OKE

Tip: Sending build context to Docker daemon when Docker build

Symptom:

  When we run docker build

Sending build context to Docker daemon...

   After a while, we hit out of space issue.

Solution:

When docker build large image like oracle database, we better only keep only 1 version DB downloaded binary file in the docker build directory. 
By default docker daemon sending build context will include all the zip files in it (include unused version zip files), it may cause unnecessary space pressure.

Error : /usr/bin/postgresql-setup initdb no such file or directory

Symptom:

When we build postgresql 9.5 docker image, we
 /usr/bin/postgresql-setup initdb  no such file or directory

Solution:

It is due to initdb was not in default PATH . By default, yum installs initdb at /usr/pgsql-9.5/bin
To fix that we add below line into Dockerfile
RUN ln -s /usr/pgsql-9.5/bin/initdb  /usr/bin/initdb

The full Dockerfile details of Postgresql 9.5 is on github link

Dockerfile for Httpd on Oracle Linux 7

Please check details on Github OracleLinux Httpd Dockerfile Repo

Dockerfile for Postgresql9.5 on Oracle Linux 7

Please check details on Github OracleLinux Postgresql 9.5 Dockerfile 

Docker: Failed to get D-Bus connection: Operation not permitted

Symptom:

When we build Postgresql 9.5 docker image on  Oracle Linux 7-slim ( FROM oraclelinux:7-slim)
it error out
Docker: Failed to get D-Bus connection: Operation not permitted

Solution:

It is due to Oracle Linux 7-slim is very slim. It takes systemd function out of the image.  D-bus need systemd to function.  In order to fix it ,we need to use a bit heavy linux image which includes systemd.  To use Oracle Linux 7 would fix it (FROM oraclelinux:7)

The full Dockerfile details of Postgresql 9.5 is on github link