Monday, March 25, 2019

How To Create Oracle DB Dataguard in K8S

Please click github link for full details

How To Switch Service Without Changing Apps Connection Details in K8S


    In enterprise world, we often have  standby or DR services.  Sometimes we have active and active services too.  When we do switch over service, we need to modify Apps connection details to point to new service. It generates workloads and confusion when we touch Apps configurations  In Kubernetes world, the power and flexibility of service save us from these hassles . We can switch services without changing Apps connection details.


We can use selector in K8S service component to choose which Apps the service is sitting on.
For example:
Primary DB service is on label livesqlsb-db-service and Standby DB service is on label livesqlsb-db-dr-service

To update service to point to standby DB , we simple update selector from livesqlsb-db-service --> livesqlsb-db-dr-service in the yaml file
Then kubectl apply -f  <yaml file>

apiVersion: v1
kind: Service
     name: livesqlsb-db-service
  name: livesqlsb-db-service
  namespace: default
  - port: 1521
    protocol: TCP
    targetPort: 1521
     name: livesqlsb-db-service  or  livesqlsb-db-dr-service (update it to be livesqlsb-db-dr-service for standby DB service)

Monday, March 11, 2019

How To Migrate ORDS To Welogic Cluster In K8S


We have quite a few APEX applications which are running on ORDS in weblogic domain. Now with oracle weblogic kubernetes operator please refer  github link. We can deploy a weblogic cluster in OKE. Please refer How To Create Weblogic Cluster Service via Weblogic Operator In K8S
So we would like to migrate our ORDS and APEX workload to weblogic cluster in OKE to leverage the advantages of kubernetes. 


Most of hard work how to create weblogic domain has been done via How To Create Weblogic Cluster Service via Weblogic Operator In K8S
As ORDS needs some extra config,lib, static files other than just ords.war , it is not as simple as to deploy a war into the domain  like testwebapp .
The Key thing here is to build a docker image with ORDS in it 
  • Download ORDS file from OTN
  • unzip it to ords/  under  the weblogic domain creation build directory . The directory is a bit long as we git clone the weblogic kubernetes operator . All scripts are under it. 
  •  ie  /home/k8suser/yaml/weblogic-kubernetes-operator/kubernetes/samples/scripts/create-weblogic-domain/domain-home-in-image/docker-images/OracleWebLogic/samples/12213-domain-home-in-image
  • Update Dockerfile to add an  entry  to copy  ord/  we just unzip to docker image  "COPY ./ords/  /u01/app/oracle/product/ords/"
    • /u01/app/oracle/product/ords/   is the dir that weblogic cluster will deploy the war file to .  Each version of war would be a bit different , as the war has dependency on config files outside war file,  it makes deployment difficult . We have to maintain the same directory in the docker images to let ords find correct config files 
    • It is the same with APEX static image files , the images files are put  ./ords/i , so all images files are copied into docker image
  • Update myinputs.yaml file to add extra java options for ORDS. Details below
    • javaOptions: -Dweblogic.StdoutDebugEnabled=false -Djavax.xml.bind.JAXBContext=com.sun.xml.bind.v2.ContextFactory -Djava.awt.headless=true -Doracle.jdbc.fanEnabled=false
    • For PROD ,we may add big memory for jvm  like
      • javaOptions: -Xms3g -Xmx3g -verbose:memory,gcreport,gc -Doracle.jdbc.implicitStatementCacheSize=100 -Dweblogic.StdoutDebugEnabled=false -Djavax.xml.bind.JAXBContext=com.sun.xml.bind.v2.ContextFactory -Djava.awt.headless=true -Doracle.jdbc.fanEnabled=false  
  • To create a new image based on the new Dockerfile  "./ -i myinputs.yaml -o .  -u weblogic -p  ****"
  • Start domain using the new images refer details on  How To Create Weblogic Cluster Service via Weblogic Operator In K8S
  • Deploy ords.war  i.war and livesql.war 3 war files in /u01/app/oracle/product/ords/ from console . Refer ORDS Installation and Setup
    • ords.war is for ords service  
    • i.war is for /i  static image files    (  java -jar ords.war static --context-path /i   /u01/app/oracle/product/ords/    ) 
    • livesql.war is for /livesql static image files   (   java -jar ords.war static --context-path /livesql  /u01/app/oracle/product/ords/   )
  • Test url  http://<domain name>/apex/


  • 404 error  "The request could not be mapped to any database. Check the request URL is correct, and that URL to database mappings have been correctly configured"
    • weblogic cluster will use its own deployment path " /u01/app/oracle/product/ords/  " not the one we set on the console
      • We need to copy the ords files in the same path in docker image

Thursday, March 07, 2019

How To Create Weblogic Cluster Service via Weblogic Operator In K8S


   We have many workloads running on top of weblogic cluster domain. It is not an easy task to setup weblogic cluster with just only oracle weblogic docker images. Oracle is providing an open source new tool (Weblogic Kubernetes Operator) to make things easier. More details of this operator please refer  github link. This note is to follow the quickstart guide to create a stage weblogic cluster with domain in image (which is recommended)and  use  Traefik as load balancer and proxy server. Another useful blog is weblogic-kubernetes-support-with-operator-20-v2


Part1: Create Weblogic Cluster 

  • git clone
  • docker pull oracle/weblogic-kubernetes-operator:2.0
  • test "docker login  "  as official weblogic docker images in docker hub need an account to access. We can get free account from docker website
  • docker pull store/oracle/weblogic:
  • kubectl create namespace stage-weblogic-operator-ns
  • kubectl create serviceaccount -n stage-weblogic-operator-ns stage-weblogic-operator-sa
  • Install helm . Our tiller version is 2.8.2 . Please download correct helm version . Helm github link
  • cd directory of  git clone ie /home/k8suser/yaml/weblogic-kubernetes-operator/
  • Run helm to install weblogic operator
helm install kubernetes/charts/weblogic-operator \
--name stage-weblogic-operator \
--namespace stage-weblogic-operator-ns \
--set serviceAccount=stage-weblogic-operator-sa \
--set "domainNamespaces={}" \
  • kubectl create namespace stage-domain1-ns
  • Update the namespace in helm
helm upgrade \
--reuse-values \
--set "domainNamespaces={stage-domain1-ns}" \
--wait \
stage-weblogic-operator \
  • Create weblogic credentials 
./kubernetes/samples/scripts/create-weblogic-domain-credentials/ \ -u weblogic -p *****  -n stage-domain1-ns -d stage-domain1
  • Create ocir secret in OCI to access images of private repository .Please refer my other note
kubectl create secret docker-registry iad-ocir-secret --docker-username='testenacy/' --docker-password='*****' --docker-email=''   -n stage-domain1-ns
  • cd kubernetes/samples/scripts/create-weblogic-domain/domain-home-in-image
  • cp create-domain-inputs.yaml myinputs.yaml
  • Modify myinputs.yaml to add this entry "imagePullSecretName: iad-ocir-secret"
  • Modify myinputs.yaml to set exposeAdminNodePort to be true. "exposeAdminNodePort: true"
  • Next step is to create domain.yaml file
  • Check root user can access internet as it needs clone of docker files from github
  • run as root as scripts connects to docker daemon  to create new a docker image
  • #cd kubernetes/samples/scripts/create-weblogic-domain/domain-home-in-image
  • Create a new docker image and domain yaml file
#./ -i myinputs.yaml -o .  -u weblogic -p  ****
  • A new domain.yaml will be created in ./weblogic-domains/stage-domain1
  • Modify the domain.yaml to update the image details from "domain-home-in-image:" to ""
  • #docker tag domain-home-in-image:
  • #docker push  Please refer my other note
  • kubectl apply -f ./weblogic-domains/stage-domain1/domain.yaml
  • The weblogic console url would be like below. However as domain is in docker image, it is not recommended to start/stop managed server via console. Plus no configuration updates will be saved after pod is recreated. Modify domain configuration ,please refer this guide
 http://<ip or hostname >:30701/console/login/LoginForm.jsp
  • The output of pod status would like
$ kubectl get po -n stage-domain1-ns
NAME                            READY     STATUS    RESTARTS   AGE
stage-domain1-admin-server      1/1       Running   0          1h
stage-domain1-managed-server1   1/1       Running   0          1h
stage-domain1-managed-server2   1/1       Running   0          1h
$ kubectl get po -n stage-weblogic-operator-ns
NAME                                 READY     STATUS    RESTARTS   AGE
weblogic-operator-6c44c764b5-4ssmf   1/1       Running   0          19h
$ kubectl get svc -n  stage-domain1-ns
NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
stage-domain1-admin-server            ClusterIP   None            <none>        30012/TCP,7001/TCP   3m
stage-domain1-admin-server-external   NodePort   <none>        7001:30701/TCP       16m
stage-domain1-cluster-cluster-1       ClusterIP     <none>        8001/TCP             3m
stage-domain1-managed-server1         ClusterIP   None            <none>        8001/TCP             5h
stage-domain1-managed-server2         ClusterIP   None            <none>        8001/TCP             5h           
  • We can deploy testwebapp via WLS console to all cluster nodes. The sample web application is located in the kubernetes/samples/charts/application directory. test the url via curl
 curl http://<pod ip address>:8001/testwebapp/index.jsp

Part2: Create Load Balancer

  • Weblogic Operator supports quite a few load balancer. Full list can be accessed via github link
  • We choose traefik  .  Details in guide
  • cd kubernetes/samples/charts/traefik 
  • helm init --client-only
  • To install traefik in the same namespace of WLS domain
  • helm install --name traefik-operator --namespace stage-domain1-ns  --values values.yaml stable/traefik
  • Once traefik is installed,  Add one entry into local laptop hosts file to spoof hostname.  "<Host IP address>"
  • We can access its dashboard
  • Or use below curl test traefik dashboard
curl -H 'host:' http://${HOSTNAME}:30305/
  • Add host routing rule for testwebapp
apiVersion: extensions/v1beta1
kind: Ingress
  annotations: traefik
  name: traefik-hostrouting-1
  namespace: stage-domain1-ns
  - host:
      - path:
          serviceName: stage-domain1-cluster-cluster-1
          servicePort: 8001 
  • Add one entry into local laptop hosts file to spoof hostname.  "<Host IP address>"
  • We can access its dashboard
  • Or use below curl test traefik dashboard
  • curl -H 'host:' http://${HOSTNAME}:30305/test/webapp
  • Also you can see rule details on traefik dashboard 

Easy Way To Unplug and Plug PDB After 12.2

After Oracle DB 12.2 , it is much easier to move data or migrate DB to different places. 2 simple sqls can achieve that.



Copy the .pdb file to the target place......


file_name_convert =('/u110/datatarget/','/opt/oracle/oradata/testnew/DATAFILE/');


Wednesday, March 06, 2019

How To Create Private Load Balancer in OKE


    OKE(Oracle Kubernetes Engine) has well integration with OCI load balancer . We are going to create a private load balancer in OKE . Default a public load balancer is created with public address.


  We add some annoations in the metadata to address that. The full list of annotations could be found in github oracle oci clould controller manager load balancer section

apiVersion: v1
kind: Service
  annotations: 100Mbps

Tuesday, March 05, 2019

How To Fix Pod Mounted PV Permission Issues on OKE


  We are building DB services on OKE . By default OKE(Oracle Kubernete Engine) storageclass is oci which is OCI block volume.  If we don't specify storageclass in yaml file, OKE would automatically create block volumes as persistent volumes and attach to pods for us which is very convenient.
However we hit permission issue , by default the filesystem created by OKE is owned by root , the docker images user is oracle with id 54321 . It fails on creating DB.


  It is not a good practice for a Dockerfile to modify parent host mounted file permission. We can use yaml to tell OKE to mount the volume with correct permission.  More details refer Kubernetes security context doc
Add below in the spec of the yaml file, in this case 54321 is the id
         runAsUser: 54321
         fsGroup: 54321

Sunday, March 03, 2019

How To Build Docker Image of Oracle 19c DB


  We would like to build a new 19c DB docker image for testing.


It is based on how we build 18.4 docker images in oracle github

  • Download all scripts from oracle github
  • Update Dockerfile to use the zip which is 19c
# add proxy
RUN echo "proxy=http://<proxy server ip address>:80" >> /etc/yum.conf
  • docker build --network=default --force-rm=true --no-cache=true --build-arg DB_EDITION=ee -t  oracle/database:19.2v1  .