Tip: Use oci cli to reboot a VM

oci compute instance action --action SOFTRESET --region us-ashburn-1 --instance-id  <instance id you can get from kubectl describe node>

oci compute instance get  --region us-ashburn-1 --instance-id  <instance id you can get from kubectl describe node>

sometimes, you may get 404 error if you omit " --region us-ashburn-1"

Tip: Collect console serial Logs of Oracle Cloud Infrastructure

oci compute console-history capture   --region us-ashburn-1 --instance-id <instance-ocid>

--> oci compute console-history get  --region us-ashburn-1 --instance-console-history-id <OCID from the command before> 

--> oci compute console-history get-content --region us-ashburn-1  --length 1000000000 --file /tmp/logfile.txt --instance-console-history-id <OCID from the command before>

Error :no available volume zone in Kubernetes

Symptom:

   When we create deployment/statefulset/pod in OKE (Oracle Kubernete Engine), somehow we hit below error:
Warning  FailedScheduling  3s (x7 over 3m)   default-scheduler  0/3 nodes are available: 1 node(s) didn't match node selector, 2 node(s) had no available volume zone.

Solution:

   One of the reasons is the we use OKE auto provision for our block volume storage. It has a constraint that block volume need to be the same AD (availability zone) as VM. In that case the block volume is created in different AD, the pod can't access the block volume
To fix that, we just need to adjust the label to let pod be created in the same AD as block volume.

Change the Reclaim Policy of a PersistentVolume In OKE

Symptom:

  By default OKE(Oracle Kubernete Engine)  storageclass is oci which is OCI block volume
  If we don't specify storageclass in yaml file,  OKE would automatically create block volumes as persistent volumes and attach to pods for us which is very convenient.
 However reclaim policy of persistent volumes is "DELETE"
 It means if we delete pv and pvc OKE created, OKE would delete block volumes in OCI as well.

Solution:

  To prevent potential data loss due to reclaim policy " DELETE" , we can update it to be "RETAIN"

kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Please refer kubernete doc for more details

How To Add OKE Worker Nodes via OCI API

Requirement

  We have Kubernetes cluster running in OKE (Oracle Kubernetes Engine).  We would like to dig deeper to use OCI API to manager the node pool of our K8S cluster. ie we can add a new worker node into our new private subnet. With OCI API, we can have more flexibility to handle worker nodes in the cluster.  In this example, we plan to use curl and raw REST API calls to do that.  Later oci-cli or SDK would support such functions

Solution

• First to install oci-curl .  It is a Bash Shell script. Please refer OCI Doc
• Configure the oci-curl to use correct tenancyId , authUserId , keyFingerprint, privateKeyPath. More details Please refer OCI Credential Doc and  OCI API  details
• Test oci-curl . If you can get result like below , it means it works 

# .   ./oci-curl.sh

# oci-curl containerengine.us-ashburn-1.oraclecloud.com GET "/20180222/clusterOptions/all"

{

  "kubernetesVersions": [

    "v1.10.11",

    "v1.11.5"

  ]

• Prepare oke-nodepool.json The example of json file is like

{

  "compartmentId": "INSERT_YOUR_COMPARTMENT_ID",

  "clusterId": "INSERT_YOUR_CLUSTER_ID",

  "name": "INSERT_YOUR_NODE_POOL_NAME",

  "kubernetesVersion": "INSERT_YOUR_K8S_VERSION",

  "nodeImageName": "INSERT_YOUR_CUSTOM_IMAGE_OCID",

  "nodeShape": "INSERT_YOUR_SHAPE_NAME",

  "nodeMetadata": {

    "INSERT_YOUR_CUSTOM_KEY_1": "INSERT_YOUR_CUSTOM_VALUE_1",

    "INSERT_YOUR_CUSTOM_KEY_2": "INSERT_YOUR_CUSTOM_VALUE_2"

  },

  "initialNodeLabels": [

    {

      "key": "node.info/INSERT_YOUR_SAMPLE_KEY",

      "value": "INSERT_YOUR_SAMPLE_VALUE"

    }

  ],

  "sshPublicKey": "INSERT_YOUR_SSH_PUBLIC_KEY",

  "quantityPerSubnet": INSERT_YOUR_QTY_PER_SUBNET,

  "subnetIds": [

    "",

    "",

    ""

  ]

}

• Create a node pool 

oci-curl containerengine.us-ashburn-1.oraclecloud.com POST ./oke-nodepool.json "/20180222/nodePools"

• Go to the OCI ashburn console , a new nodepool would be created in OKE according to json file.

Tip To Upload Files Into OCI Object Storage with Pre-Authenticated Requests

Symptom:

 Sometimes we need to provide many files to customers and customers need to provide files to us.
We can use OCI Object Storage Pre-Authenticated Requests to share data. Please refer more details on  Oracle Blog
    However after I create bucket Pre-Authenticated Requests , copy and paste url into our browser, we get below error

{"code":"NotFound","message":"Not Found"}

Solution:

The object Pre-Authenticated Requests can be writing or reading or both. 
The bucket Pre-Authenticated Requests is for writing purpose only (not reading)
So we need to use curl or other tools to put objects into the bucket. ie

curl https://objectstorage.us-ashburn-1.oraclecloud.com/p/THVqAQC1oIiwmerj6aK1lY5q3xG-rK3iGZvM2b1kq0/n/testtest/b/testdata/o/ --upload-file  mydata.tar