Apex Ords Operator for Kubernetes

Requirement:

We often need to provision Apex and Ords for Dev, Stage, Prod. 

This is the operator to automate Apex Oracle Application Express 19.1 and Ords oracle rest data service via Kubernetes CRD, it creates a brand new Oracle 19c database statefulset, apex, ords deployment plus load balancer in the Kubernetes cluster

Solution:

Full details and source codes are on GitHub repository

Demo:

OKE Admission Control Webhook Sample

Requirement:

We need to implement a policy requested by the security team that Kubernetes service should have an annotation : service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: None Thus no security list will be updated by Kubernetes. This is an example that how we build our own admission controller which implements various policies from security or other teams. ie we can add only internal load balancer is allowed for internal service.....etc

Solution:

• Please refer github repo
• git clone https://github.com/HenryXie1/oke-admission-webhook
• go build -o oke-admission-webhook
• docker build --no-cache -t repo-url/oke-admission-webhook:v1 .
• rm -rf oke-admission-webhook
• docker push repo-url/oke-admission-webhook:v1
• ./deployment/webhook-create-signed-cert.sh --service oke-admission-webhook-svc --namespace kube-system --secret oke-admission-webhook-secret
• kubectl replace --force -f deployment/validatingwebhook.yaml
• kubectl replace --force -f deployment/deployment.yaml
• kubectl replace --force -f deployment/service.yaml

Demo:

Tip: what are the GVK GVR CRD CR Scheme in Kuberentes Core API

GVK:

• GVK stands for Group Version Kind 
• Each Kind in K8S has Group and Version. i.e. Kind "Pod" is in Group "core" , Version "v1". Refer to official API doc
• GVK  is defined to associate Group, Version and Kind
• Each GVK map to a given root Go type in the package
• Source code definition is  here 

GVR:

• GVR stands for Group Version Resource
• GVR is a "use" or "instance" of GVK in the K8S API
• The command "kubectl api-resources"  gives us a list of GVR in the K8S cluster

CRD:

• CRD stands for Custom Resource Definition
• Each CRD is like Kind in K8S, so it also has Group, Version
• CRD is the extension of the K8S API. Refer to official doc
• Once it is defined, it acts like GVK in K8S API.

CR:

• CR stands for Custom Resource
• CR is a "use" or "instance" of CRD in the K8S API
• Once it is instantiated, it acts like GVR in K8S API.
• The command "kubectl api-resources"  gives us a list including both GVR and CR in the cluster.

Scheme:

• The scheme is defined to keep track of a given GO type mapping to a given GVK. 
• For example, we define   myexample.io/api/v1.mykind{}
• The scheme is going to map it to the API group we defined in CRD: batchv1.myexample.io/v1
• {
      kind:  mykind
      apiVersion: batchv1.myexample.io/v1
  }
• Source code definition is here.

Tip: Understand Golang methods and interface

Methods:

• methods are functions with a special receiver ( a normal or define type)
• so we define a type and create a method based on this type,  so the logic is the first type  ---> define method.
• As we have lots of common types, we use a type to group all sorts of methods
• So the type is the centrepiece to think thorough

Interface:

• Golang interface is still a type, but not normal concrete type like string, int ....etc. Instead, it is an abstract type built on top of those concrete types.
•  It has two elements:  a concrete type + value of the concrete type
• According to the interface definition, which concrete types implements this interface, Golang automatically matches which concrete types are bond to this interface.
• So in the same program, the same interface type can be different concrete types and values.
• Concrete types have methods. We can call these methods like my-type.my-methods(). It is the same for the interface. We can call these methods like my-interface.my-methods()
• A method can be called by both interface and its concrete type associated.
• Use type assertion to get the value from the interface, then use the value to invoke other methods which are not defined in the interface.
• One of the reasons why we have an interface is: since we also have lots of common methods, i.e. print string, play sports... all sorts of actions are considered as methods, so the genius part is we define an interface as a common signature of methods, so the centrepiece is not a TYPE, but methods,  it uses a common method (i.e. print string)  to group all sorts of types...

Tip: Understand Golang Channel Directions

• Channel is used to communicate among Goroutines.
• Always image a channel as a pipe in the brain. A pipe of any concrete type of Golang, like string, int..etc. This pipe connects Goroutines
• There is sending side of this pipe on one Goroutines, and there is an ending side of this pipe on another Goroutines
• main() is also a Goroutine. 
• We need to identify which Goroutine is sending side, which is the ending side.

• Sending side is like.

sends chan<- string 

the chan<- string  means there is a string on the sending side of pipe   
We use it like    sends <- "my sending string" 
Ending side is like.
ends <-chan string 
the <-chan string  means there is a string on the ending side of pipe   
We use it like    ending_message <-ends or <-ends
for range <channel name>: it's used on the ending side the pipe to fetch values
time.Ticker is a good example of ending side of pipe
type Ticker struct {
	C <-chan Time // The channel on which the ticks are delivered.
	// contains filtered or unexported fields
}

Below is another advance example of sending / ending side
os.Signal is the sending side of channel sigs
another process or thread from OS
main() goroutines is the ending side of channel sigs
// Registering signals - INT and TERM
sigs := make(chan os.Signal, 1)
signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)
// Process blocked here and waiting for signal(term/int)
<-sigs

Automation Tool to Create Http Ords and Loadbalancer in K8S

Requirement:

A kubectl plugin that create http and ords( Oracle Rest Data Services) based on Apex (oracle application express) 19.1

Once we have Apex ready . We often need to provision  http and ords for it. We would like to automate http ords and loadbalancer deployment in K8S. Once we have db hostname, port , sys password , apex /ords password. We can deployment a brand new http ords and loadbalancer deployment env via 1 command. We can also delete it via 1 command. ords image is based on docker images of oracle github.

Solution:

Full details and source codes are on github repository

Automation Tool to Create Database 19.2 in K8S

Requirement:

A kubectl plugin that create statefulset of oracle database 19.2 in your Kubernetes cluster or minikube.You get the full power of oracle database 19.2 in about 10-20 min (need more time of first time run to download docker image) and you can access it from laptop (assume ports are open)

Solution:

Full details and source codes are on github repository

Automation Tool to Create Apex 19.1 in K8S

Requirement:

A kubectl plugin to provision Apex(Oracle Application Express).  Apex is the foundation of many applications .  We often need to provision a apex for test, stage and prod. We would like to automate apex 19.1 deployment on a Oracle DB. 
This database can be a  DB in Cloud(AWS, Azure, GCP,OCI)  , it can be a DB in a VM, it can be DB pod in K8S.  Once we have db hostname, port , sys password , we can deployment a brand new Apex 19.1  env via 1 command.  We can also delete it via 1 command.

Solution:

Full details and source codes are on github repository

BPF Hello World Examples

What is BPF:

Refer from this doc
BPF is a highly flexible and efficient virtual machine-like construct in the Linux kernel allowing to execute bytecode at various hook points in a safe manner. It is used in a number of Linux kernel subsystems, most prominently networking, tracing and security (e.g. sandboxing).

BPF in the Linux kernel is allowing to execute bytecode at various hook points in a safe manner. It is used in a number of Linux kernel subsystems, most prominently networking, tracing and security (e.g. sandboxing).

Github BPF Hello World examples

Error: expected ';', found '{' in Golang

Symptom:

When we write go code for kubernetes OwnerReference , we get such error

expected ';', found '{' 

code is like

var oradbownerref = []metav1.ObjectMeta.OwnerReference{{
Kind:       apexords.TypeMeta.Kind,
APIVersion: apexords.TypeMeta.APIVersion,
Name:       apexords.ObjectMeta.Name,
UID:        apexords.ObjectMeta.UID,
}}

Solution:

It is due to OwnerReference  is on metav1 level ,not metav1.ObjectMeta level.
Correct code is

var oradbownerref = []metav1.OwnerReference{{
Kind:       apexords.TypeMeta.Kind,
APIVersion: apexords.TypeMeta.APIVersion,
Name:       apexords.ObjectMeta.Name,
UID:        apexords.ObjectMeta.UID,
}}

Example of Pod Struct with ConfigMap ImagePullSecrets in Client-GO

typeMetadata := metav1.TypeMeta{
Kind:       "Pod",
APIVersion: "v1",
}
objectMetadata := metav1.ObjectMeta{
Name: "ordspod",
Namespace:    o.UserSpecifiedNamespace,
}
configmapvolume := &corev1.ConfigMapVolumeSource{
LocalObjectReference: corev1.LocalObjectReference{Name: "test-configmap"},
}
podSpecs := corev1.PodSpec{
ImagePullSecrets: []corev1.LocalObjectReference{{
Name: "test-secret",
}},
Volumes:  []corev1.Volume{{
Name: "ords-config",
VolumeSource: corev1.VolumeSource{
ConfigMap: configmapvolume,
},
}},
Containers:    []corev1.Container{{
Name: "ordspod",
Image: "ords:v19",
VolumeMounts: []corev1.VolumeMount{{
Name: "ords-config",
MountPath: "/mnt/k8s",
}},
}},
}
pod := corev1.Pod{
TypeMeta:   typeMetadata,
ObjectMeta: objectMetadata,
Spec:       podSpecs,
}

Tip to Get Output From A Command Running Inside Pod

Symptom:

  We write a client-go program to run a simple command ie pwd inside a pod.  It runs fine, No error  but I don't see pwd output to the standard OS output. Partial sample code is like

func ExecPodCmd(o *KubeOperations,Podname,Podnamespace string,SimpleCommand []string) error {
       SimpleCommand := []string{"pwd"}

execReq := o.clientset.CoreV1().RESTClient().Post().
    Resource("pods").
Name(Podname).
Namespace(Podnamespace).
SubResource("exec")
    execReq.VersionedParams(&corev1.PodExecOptions{
Command:   PsqlCommand,
Stdin:     true,
Stdout:    true,
Stderr:    true,
}, scheme.ParameterCodec)
exec, err := remotecommand.NewSPDYExecutor(o.restConfig, "POST", execReq.URL())
if err != nil {
return fmt.Errorf("error while creating Executor: %v", err)
}
err = exec.Stream(remotecommand.StreamOptions{
Stdin:  os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
Tty:    false,
})
if err != nil {
return fmt.Errorf("error in Stream: %v", err)
} else {
return nil
}
}

Solution:

It turns out we need to use bash -c to run the simplecommand to get proper os output.
Replace   SimpleCommand := []string{"pwd"} with SimpleCommand := []string{"/bin/sh", "-c", "pwd"}

Error: unable to upgrade connection: you must specify at least 1 of stdin, stdout, stderr

Symptom:

  We would like to run a simple comand ie pwd in a pod via client-go.  It error on below part of code:

err = exec.Stream(remotecommand.StreamOptions{
Stdin:  stdin,
Stdout: stdout,
Stderr: stderr,
Tty:    false,
})

Solution:

  We can simply to use os.Stdin os.Stdout os.Stderr to get the comand output from pod to our standard os output.  Correct code is like:

err = exec.Stream(remotecommand.StreamOptions{
Stdin:  os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
Tty:    false,
})

Error: cannot use "k8s.io/api/core/v1".ConfigMapVolumeSource literal (type "k8s.io/api/core/v1".ConfigMapVolumeSource) as type *"k8s.io/api/core/v1".ConfigMapVolumeSource in field value

Symptom:

Related code:
o.pgreplicamaster.Spec.Template.Spec.Volumes = []corev1.Volume{
{Name: "pgreplica-config",
VolumeSource: corev1.VolumeSource{ 
   ConfigMap: corev1.ConfigMapVolumeSource{
    LocalObjectReference: corev1.LocalObjectReference{Name: o.UserSpecifiedCM},
   },
   },
    
},
}
 Compile Error
cannot use "k8s.io/api/core/v1".ConfigMapVolumeSource literal (type "k8s.io/api/core/v1".ConfigMapVolumeSource) as type *"k8s.io/api/core/v1".ConfigMapVolumeSource in field value

Solution:

    Refer VolumeSource k8s doc. ConfigMap suppose to have pointer  *ConfigMapVolumeSource, not value of  ConfigMapVolumeSource , We need to add &.
Meanwhile we can't mix field value and value , refer stackflow link
Correct code is:
o.pgreplicamaster.Spec.Template.Spec.Volumes = []corev1.Volume{
{Name: "pgreplica-config",
 VolumeSource: corev1.VolumeSource{ 
   ConfigMap: &corev1.ConfigMapVolumeSource{
    LocalObjectReference: corev1.LocalObjectReference{Name: o.UserSpecifiedCM},
   },
   },
 
},
}

Example of Updating ConfigMap name in Client-go

Option 1:

o.pgreplicamaster.Spec.Template.Spec.Volumes[0].VolumeSource.ConfigMap.LocalObjectReference = corev1.LocalObjectReference{Name: o.UserSpecifiedCM}

is doing the same thing as

Option 2:

o.pgreplicamaster.Spec.Template.Spec.Volumes = []corev1.Volume{
{Name: "pgreplica-config",
VolumeSource: corev1.VolumeSource{
   ConfigMap: &corev1.ConfigMapVolumeSource{
    LocalObjectReference: corev1.LocalObjectReference{Name: o.UserSpecifiedCM},
   },
   },
 
},
}

fmt.Printf("%#v\n",o.pgreplicamaster.Spec.Template.Spec.Volumes[0].VolumeSource.ConfigMap.LocalObjectReference)

Go Exercise: How To use yaml Unmarshal to convert yaml into k8s client go data struct

please see code on github link

Go Exercise: How To use decode yaml into k8s client go data struct

Please see code on github link

Go Excercise: Cobra Example

Please see code on github link

Error: cannot load k8s.io/client-go/pkg/api": cannot find module providing package k8s.io/client-go/pkg/api

Symptom:

  When we use client-go api ,there is line of code :
api.Codecs.UniversalDeserializer()
We often import "k8s.io/client-go/pkg/api". It was working. Then suddently we hit this error
Error: cannot load k8s.io/client-go/pkg/api": cannot find module providing package k8s.io/client-go/pkg/api

Solution:

  The reason of this error is due to code baseline has moved to a new location
Instead of import "k8s.io/client-go/pkg/api"
we should use
import "k8s.io/client-go/kubernetes/scheme"

Then change code to use scheme instead of api
scheme.Codecs.UniversalDeserializer()

Error:request.go:598:31: not enough arguments in call to watch.NewStreamWatcher have (*versioned.Decoder) want (watch.Decoder, watch.Reporter)

Symptom:

When we build a go program, we hit such error:
k8s.io/client-go/rest/request.go:598:31: not enough arguments in call to watch.NewStreamWatcher have (*versioned.Decoder) want (watch.Decoder, watch.Reporter)
It appears there are updates on the request.go which have new requirements
See details of changes log  on apimachinary 

Solution:

   We need to avoid to use the latest master branch of the client-go. Instead we can use stable version of client-go. Fortunately go-modules addresses these problems
see github go-modules
So here are the steps to fix it

• $ export GO111MODULE=on
• In your project location, run :  go mod init     ---- create go.mod file
• Go build  cmd/test.go    --- go mod will fetch related files which replace dep ensure
• You will still see the error, that's ok , next step we fix it 
• Edit go.mod and replace client-go with correct version
• In this case  we use :   k8s.io/client-go v0.0.0-20190425172711-65184652c889
• Go build cmd/test.go     ---error would be gone