Tip: OPA Rego error minus: operand 1 must be one of {number, set} but got string

Symptom:

    We start to use OPA gatekeeper for our kubernetes clusters. Refer https://github.com/open-policy-agent/gatekeeper
    When we code some policies for kubernetes using OPA (open policy agent) Rego , the part of code is like below

violation[{"msg": msg}] {
          provided := input.review.object.spec.nodeSelector[label]
          required := input.parameters.labels[_].key
          missing := required - provided
          expected :=  input.parameters.labels[_]
          count(missing) > 0
          msg := sprintf("Missing nodeSelector label <%v: %v>, or too many nodeSelector labels,only 1 nodeSelector lable is allowed.< %v:%v>",[expected.key,expected.allowedvalue,provided,required])

We get error:

eval_type_error: minus: operand 1 must be one of {number, set} but got string): error when creating "access-pod.yaml": admission webhook "validation.gatekeeper.sh" denied the request: admission.k8s.gatekeeper.sh: templates["admission.k8s.gatekeeper.sh"]["K8sAllowedNodeselector"]:5: eval_type_error: minus: operand 1 must be one of {number, set} but got string

Solution:

        missing := required - provided , all variables are string, minus operator can't deal with string, so we need to convert them into number or set
So the right code is

provided := {label | input.review.object.spec.nodeSelector[label]}
required := {label | label := input.parameters.labels[_].key}

Error: You must be logged in to the server (Unauthorized)

Symptom:

   When users try to list pod of OKE (oracle kubernete engine) via kubectl get po. It error out as below

error: You must be logged in to the server (Unauthorized)

Solution:

  It is quite possible the users don't have correct privilege in  Oracle OCI IAM.  Users need to be in a group which has a policy "USE"  or higher "MANAGE"  for OKE clusters.

ie  Allow group <group-name> to use  cluster-family in <location>

Tip: RBAC Comparison Oracle DB vs Kubernetes

This is for Oracle DBA to better understand how Kubernetes RBAC works. They both have similar RBAC concepts

Oracle Database	Kubernetes
dba role	cluster-admin role
grant dba role	grant cluster-admin role
create apps-user role to access tablespace example only	create apps-user role to access namespace example only
create apps-user	create apps-user or service account
grant apps-user role to apps-user	role-binding apps-user role to apps-user
apps-users work happily in tablespace example	apps-users work happily in namespace example

How to Segregate Applications in Kubernetes Cluster without Compromise Cluster-Admin Role

Requirement:

   In enterprise world, we often have a few applications running on same Kubernete cluster. Each application owners would like to operate actions on his own applications without interfering other applications.  We would not like to grant cluster-admin to application owners for security reasons. Meanwhile application owner would have fully privilege in their own application scope.
This is for Oracle DBA to better understand how Kubernetes RBAC works. They both have similar RBAC concepts

Oracle Database	Kubernetes
dba role	cluster-admin role
grant dba role	grant cluster-admin role
create apps-user role to access tablespace example only	create apps-user role to access namespace example only
create apps-user	create apps-user or service account
grant apps-user role to apps-user	role-binding apps-user role to apps-user
apps-users work happily in tablespace example	apps-users work happily in namespace example

Solution:

• Create namespace for each application

kubectl create namespace  test-apps-ns

• Cluster admin create  role, serviceaccount, rolebinding for each application . Below is an example yaml file

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace:  test-apps-ns
  name: test-role
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: oke-test-user
  namespace:  test-apps-ns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace:  test-apps-ns
  name: test-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: test-role
subjects:
- kind: ServiceAccount
  name: oke-test-user
  namespace:  test-apps-ns

Example of OKE ClusterRolebinding for User OCID of Oracle Cloud

Commands:

$ kubectl create rolebinding hxie-rolebinding --role=livesql-apps --user=ocid1.user.oc1..aaaaa...tx5a
$ kubectl create clusterrolebinding <my-cluster-admin-binding> --clusterrole=cluster-admin  --user=<user_OCID>

Yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2019-10-23T23:24:30Z"
  name: hxie_clst_adm
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: ocid1.user.oc1......uvl7ria


Refer doc: https://docs.cloud.oracle.com/iaas/Content/ContEng/Concepts/contengaboutaccesscontrol.htm