Tip: Kubectl Delete Options

A few options to delete resources in K8S

• kubectl delete  ****  
• kubectl delete  ****   --force --grace-period=0
• kubectl delete  ****   --force --grace-period=0 --wait=false
• kubectl version  --- it will show both k8s client and server version

Tip:OPA gatekeeper REGO nodeSelector Constraint Template

Symptom:

    We start to use OPA gatekeeper for our kubernetes clusters. Refer https://github.com/open-policy-agent/gatekeeper
    We try to enforce all pods and deployment...etc to have a assigned nodeSelector.  We had some issues. The details of the issue can be found in github link

Solutions:

Rego template is like this:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8sallowednodeselector
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedNodeselector
        listKind: K8sAllowedNodeselectorList
        plural: k8sallowednodeselector
        singular: k8sallowednodeselector
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items:
                type: object
                properties:
                  key:
                    type: string
                  allowedvalue:
                    type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowednodeselector
        key1 := { k | input.review.object.spec.nodeSelector[k] }
        key2 := { k | input.review.object.spec.template.spec.nodeSelector[k]  }
        mykey := key1 | key2

        # Make sure all required selectors are implemented in template including deployment, replicaset,sts...
        violation[{"msg": msg}] {
          provided := mykey
          required := {label | label := input.parameters.labels[_].key}
          missing := required - provided
          expected :=  input.parameters.labels[_]
          count(missing) > 0
          msg := sprintf("Missing nodeSelector label <%v: %v>, or too many nodeSelector labels,only 1 nodeSelector lable is allowed.",[expected.key,expected.allowedvalue])
        }
        #Make sure that ONLY required selectors are used
        violation[{"msg": msg}] {
          provided := mykey
          required := {label | label := input.parameters.labels[_].key}
          missing := provided - required
          expected :=  input.parameters.labels[_]
          count(missing) > 0
          msg := sprintf("Missing nodeSelector label <%v: %v>, or too many nodeSelector labels,only 1 nodeSelector lable is allowed.",[expected.key,expected.allowedvalue])
        }
        #Make sure all required selectors are implemented in template including deployment, replicaset,sts...
        violation[{"msg": msg}] {
          value :=  input.review.object.spec.template.spec.nodeSelector[key]
          expected :=  input.parameters.labels[_]
          expected.key == key
          not  expected.allowedvalue == value
          msg := sprintf("Value in Label <%v: %v> does not satisfy allowed value:<%v: %v>", [key,value,expected.key,expected.allowedvalue])
        }
        #Make sure all required selectors are implemented in pod
        violation[{"msg": msg}] {
          value :=  input.review.object.spec.nodeSelector[key]
          expected :=  input.parameters.labels[_]
          expected.key == key
          not  expected.allowedvalue == value
          msg := sprintf("nodSelector of Pod <%v: %v> does not satisfy allowed value:<%v: %v>", [key,value,expected.key,expected.allowedvalue])
        }

How to Refer Key and Value in Key-Value pair in OPA Gatekeeper in Rego

Symptom:

    We start to use OPA gatekeeper for our kubernetes clusters. Refer https://github.com/open-policy-agent/gatekeeper for more details.
    When we code some policies for kubernetes using OPA (open policy agent) Rego ,we would like to reference "key" name and "value"  in nodeSelector key-value pair.  ie we have

nodeSelector:
    app: mytest

I would like to refererence "app" which is key and "test" which is value in our OPA gatekeeper policy .

Solution:

   The easy way to do it is

 myvalue := input.review.object.spec.nodeSelector[mykey]

The value of varible mykey will have  "app"
The value variable myvalue will have  "mytest"
And they are strings

To get "set" , we need to use special way to achieve it:
To get "set" for key : 

  provided := {mykey | input.review.object.spec.nodeSelector[mykey]}

To get set for value: 

  provided := {myvalue | myvalue := input.review.object.spec.nodeSelector[_]}

Tip: OPA Rego error minus: operand 1 must be one of {number, set} but got string

Symptom:

    We start to use OPA gatekeeper for our kubernetes clusters. Refer https://github.com/open-policy-agent/gatekeeper
    When we code some policies for kubernetes using OPA (open policy agent) Rego , the part of code is like below

violation[{"msg": msg}] {
          provided := input.review.object.spec.nodeSelector[label]
          required := input.parameters.labels[_].key
          missing := required - provided
          expected :=  input.parameters.labels[_]
          count(missing) > 0
          msg := sprintf("Missing nodeSelector label <%v: %v>, or too many nodeSelector labels,only 1 nodeSelector lable is allowed.< %v:%v>",[expected.key,expected.allowedvalue,provided,required])

We get error:

eval_type_error: minus: operand 1 must be one of {number, set} but got string): error when creating "access-pod.yaml": admission webhook "validation.gatekeeper.sh" denied the request: admission.k8s.gatekeeper.sh: templates["admission.k8s.gatekeeper.sh"]["K8sAllowedNodeselector"]:5: eval_type_error: minus: operand 1 must be one of {number, set} but got string

Solution:

        missing := required - provided , all variables are string, minus operator can't deal with string, so we need to convert them into number or set
So the right code is

provided := {label | input.review.object.spec.nodeSelector[label]}
required := {label | label := input.parameters.labels[_].key}