Python3 OCI SDK Create Bucket and Upload Files Into OCI Object Storage

Requirement:

We need to use OCI object storage for our backup purpose.  First to create Bucket, then we put all our backup files into the bucket.
Before we do that, we need to setup config file for OCI SDK to get correct user credential, tenancy, compartment_id ...etc. Refer my blog for example:

Solution:

Create bucket example:

#!/u01/python3/bin/python3
import oci
import argparse
#set bhucket name to create
parser = argparse.ArgumentParser(description= 'Create Bucket in Oracle cloud Object Storage')
parser.add_argument('bucketname',help='The name of bucket to be created')
parser.add_argument('--version', '-v', action='version', version='%(prog)s 1.0')
args = parser.parse_args()
bucket_name = args.bucketname
config = oci.config.from_file()
identity = oci.identity.IdentityClient(config)
compartment_id = config["compartment_id"]
object_storage = oci.object_storage.ObjectStorageClient(config)
namespace = object_storage.get_namespace().data
from oci.object_storage.models import CreateBucketDetails
request = CreateBucketDetails()
request.compartment_id = compartment_id
request.name = bucket_name
bucket = object_storage.create_bucket(namespace, request)
print(f'Create bucket "{bucket_name}" in compartment_id "{compartment_id}')

Upload files into the bucket example:

#!/u01/python3/bin/python3
import oci
import glob
import os
import argparse
parser = argparse.ArgumentParser(description= 'Upload files into Oracle cloud Object Storage')
parser.add_argument('bucketname',help='The name of bucket to upload')
parser.add_argument('files',nargs='*',help='The Full path files to upload, wildcard can be used to match multiple files, ie *livesql-archivelog*')
parser.add_argument('--version', '-v', action='version', version='%(prog)s 1.0')
args = parser.parse_args()
mybucketname = args.bucketname
upload_files_loc = args.files
config = oci.config.from_file()
identity = oci.identity.IdentityClient(config)
compartment_id = config["compartment_id"]
object_storage = oci.object_storage.ObjectStorageClient(config)
namespace = object_storage.get_namespace().data
for path in upload_files_loc:
    with open(path,'rb') as f:
       obj = object_storage.put_object(namespace,mybucketname,os.path.basename(path),f)
       print(f'uploaded "{path}" to bucket "{mybucketname}"')

Make the Script executable without python intepeter

pip install pyinstaller

pyinstaller -F  < your python script>

in dist folder, you will see the executable file of your python script.

remember it needs  ~/.oci/config and ~/.oci/oci api key , these 2 file to login oracle OCI. 

otherwise you may get error like:

oci.exceptions.ConfigFileNotFound: Could not find config file at /root/.oci/config

Prepare Config File For Python3 OCI SDK

Refer official doc

• mkdir ~/.oci/
• vim config    -----example would be like

[DEFAULT]
user=ocid1.user.oc1..testaaaaaaaa2rwnm5zt2q3kvhjfgrd3w
fingerprint=85:7e:55:e3:cd:63:6a:87:d7:c5:e2:87:40:4e:71:95
key_file=/home/oracle/.oci/oci_api_key.pem
tenancy=ocid1.tenancy.oc1..testestjizbiv4zjm763rbrtd3dfhpjq
compartment_id=ocid1.compartment.oc1..testaaaaatesttest
region=us-phoenix-1

•  Use below code to load config file for python OCI SDK

>>> from oci.config import from_file
>>> config = from_file()
# Using a different profile from the default location
>>> config = from_file(profile_name="integ-beta")
# Using the default profile from a different file
>>> config = from_file(file_location="~/.oci/config")
#display config setting to see if they are in place
>>> config
{'log_requests': False, 'additional_user_agent': '', 'pass_phrase': None, 'user': 'ocid1.user.oc1..aaaaaaaa2rwntegn3vjf5eg5bp2zuvl7ria', 'fingerprint': '85:7e:03:e3:cd:63:6a:87:d7:66:e2:87:40:4e:71:95', 'key_file': '/home/oracle/.oci/oci_api_key.pem', 'tenancy': 'ocid1.tenancy.oc1..aaaaaatestsetiv4zjm76fhpjq7cmnka', 'compartment_id': 'ocid1.compartment.oc1..aaaaaaaanwshxt3acxczvfsy3ykq', 'region': 'us-phoenix-1'}

Python 3: TypeError: 'bytes' object does not support item assignment

Symptom:

    There is a function in the python code

mesg = { 'clientid':'eeeeeee','token':'ttttttt','channelid':'hhhhh','text':'' }
def mesg2slack(mesgtext):
    """
    send notification mesg to slack channel
    """
    global mesg
    mesg['text'] = mesgtext
    mesg = urllib.parse.urlencode(mesg).encode("utf-8")
    slackurl = 'https://test.test.com/apex/test/v1/push.message'
    req = urllib.request.Request(slackurl, data=mesg) # this will make the method "POST"
    resp = urllib.request.urlopen(req)

First invoke mesg2slack('first try '), it works fine

Second invoke mesg2slack('2nd try'), it error out with error : TypeError: 'bytes' object does not support item assignment

Diagonisis :

      Be careful when we use global variable in python. We use "global mesg"  in the function, it was set as dict type.  However when we do " mesg = urllib.parse.urlencode(mesg).encode("utf-8")  " , it changes the mesg type to be bytes. 

type(mesg)  was 'dict'  when mesg = { 'clientid':'eeeeeee','token':'ttttttt','channelid':'hhhhh','text':'' }

type(mesg)  was 'bytes' after mesg = urllib.parse.urlencode(mesg).encode("utf-8")

Solution:

    Use local variable mesgutf for urllib

def mesg2slack(mesgtext):
    """
    send notification mesg to slack channel
    """
    global mesg
    mesg['text'] = mesgtext
    mesgutf = urllib.parse.urlencode(mesg).encode("utf-8")
    slackurl = 'https://test.test.com/apex/smi/v1/push.message'
    req = urllib.request.Request(slackurl, data=mesgutf) # this will make the method "POST"
    resp = urllib.request.urlopen(req)

How To Make Your Own Container Tools To Debug Kubernetes network Issue

Requirement:

   Sometimes we need to get into docker container to check network, storage...etc all kind of things for debugging. However base image won't have such tools like ip , curl ,ssh,sftp, wget , netstat ,nc, ping..... installed as we mean to keen running  images as slim as possible. How can we debug into the container without such tools?

Solution:

   Create our own container with all tools we need and attach our container to network of apps container.
Here are some details

• docker run -itd --name debug oraclelinux:7-slim
• docker exec -it debug /bin/bash
• <debug container># yum install ssh,curl, iproute ....etc tools you need 
• exit
• docker commit debug henry-swiss-knife:v1
• later you can add more tools into your own container image

Then use this henry-swiss-knife to attach network stack of kubernetes

• use docker ps |grep apex   ( find out container id of K8S pod of apex which is the example). In this case it is 44c780d348bd  ( the pod with  "/pause")

[root@instance-cas-mt2 ~]# docker ps|grep apex
340722fe6f77        4b39de352b36                                                         "/bin/sh -c $ORDS_HO…"   18 hours ago        Up 18 hours                                  k8s_apexords_apexords_default_8b06d971-cb89-11e8-a112-000017010a8f_0
44c780d348bd        container-registry.oracle.com/kubernetes_developer/pause-amd64:3.1   "/pause"                 18 hours ago        Up 18 hours                                  k8s_POD_apexords_default_8b06d971-cb89-11e8-a112-000017010a8f_0

• docker run -itd --name debug --net=container:44c780d348bd henry-swiss-knife:v1

[root@instance-cas-mt2 ~]# docker run -itd --name debug --net=container:44c780d348bd henry-swiss-knife:v1
904180885ae527b3fc4f34a319ab6dfae39af960e29b2dbf7a5902ead55684e8

• docker exec -it 90418 /bin/bash    (get into the debug container to debug K8S network stack)

How To Change Kubernetes POD CIDR

Requirement:

  Sometimes default K8S Pod CIDR overlap hosts CIDR in the intranet. So there would be problems when pod to communicate to pod. So we need to update Pod CIDR not to overlap any  CIDR in the host network.

Solution:

   It is based on Oracle K8S v1.10.5+2.0.2.el7 ( from yum.oracle.com). In this case we would like to change Pod CIDR to 192.168.0.0/16 (It is advised to use /16 CIDR )
On the master node:

• ATTEN: The command below will stop/erase existing K8S cluster and recreate a new one. Thus all configurations of K8S will be lost, Please take a backup. However it won't touch  our own docker images. 
• as root 
• # export KUBE_REPO_PREFIX=container-registry.oracle.com/kubernetes_developer
• # kubeadm-setup.sh up --pod-network-cidr  192.168.0.0/16
• Follow the instructions to get all other worker node to join the new cluster

[===> PLEASE DO THE FOLLOWING STEPS BELOW: <===]
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You can now join any number of machines by running the following on each node
as root:
  export KUBE_REPO_PREFIX=container-registry.oracle.com/kubernetes_developer && kubeadm-setup.sh join 100.106.146.3:6443 --token xxiaxy.132mis6d38xg6y3b --discovery-token-ca-cert-hash sha256:dc958ac229e213ca5f04fd609a44aa09606a459da06800224b01fa