Error: The required information to complete authentication was not provided

Symptom:

    When we run  "oci os ns get"  in Oracle Cloud, we get below error:
WARNING: Your computer time: 2020-03-19T08:21:13.667808+00:00 differs from the server time: 2020-03-19T08:27:28+00:00 by more than 5 minutes. This can cause authentication errors connecting to services.
ServiceError:
{
    "code": "NotAuthenticated",
    "message": "The required information to complete authentication was not provided.",
    "opc-request-id": "iad-1:LadKpOv52VZyJpLcapW1oD_MfXrxSEpICkJh90iR5Xke2k437wa7PQUaP99kuGSQ",
    "status": 401
}

Solution:

   We often ignore the warning message. The box OS we run oci is indeed had more than 5 minutes time differences with the server.

   Sync the local time and fix the issue

How To Add TLS Certificate Into Ingress in OKE

Requirement:

         In order to secure the traffic, we need to deploy  TLS certificates into our ingress running in OKE.  We are going to use self-signed certificates to demonstrate it. 

Solution:

• Generate self-signed certificates via openssl
  • openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout tls.key -out tls.crt  -config req.conf -extensions 'v3_req'

  • req.conf:
    
    [req]
    distinguished_name = ingress_tls_prometheus_test
    x509_extensions = v3_req
    prompt = no
    [ingress_tls_prometheus_test]
    C = US
    ST = VA
    L = NY
    O = BAR
    OU = BAR
    CN = www.bar.com
    [v3_req]
    keyUsage = keyEncipherment, dataEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = prometheus.bar.com
    DNS.2 = grafana.bar.com
    DNS.3 = alertmanager.bar.com

    
    
  • To verify it:    openssl x509 -in tls.crt -noout -text
• Create Kubernetes TLS secret for that
  • kubectl create secret tls tls-prometheus-test --key tls.key --cert tls.crt -n monitoring
• Add TLS section into the ingress yaml file. Example:

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: prometheus-ingress
      namespace: monitoring
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      tls:
      - hosts:
        - prometheus.bar.com
        secretName: tls-prometheus-test
      - hosts:
        - grafana.bar.com
        secretName: tls-prometheus-test
      - hosts:
        - alertmanager.bar.com
        secretName: tls-prometheus-test
      rules:
      - host: prometheus.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: prometheus-k8s
              servicePort: 9090
      - host: grafana.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: grafana
              servicePort: 3000
      - host: alertmanager.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: alertmanager-main
              servicePort: 9093

• Ingress controller would redirect http traffic to https traffic automatically for these 3 domains 
• Spoof IP address for DNS names via the below entry and take off www proxy of the browser if necessary.
  • 123.123.123.123         prometheus.bar.com  grafana.bar.com  alertmanager.bar.com

Example of Pod Security Policy for Apache Httpd in OKE

Requirement:

As Pod Security Policy is enabled in Kubernetes Cluster, we need a PSP (Pod Security Policy) for Apache Httpd Server. How to create an Apache Httpd docker image, please refer to note. Http Server needs some special features other than normal applications.
Here is a PSP example which is tested in OKE (Oracle Kubernetes Engine).

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: oke-restricted-psp
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default'
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  'runtime/default'
spec:
  privileged: false
  allowedCapabilities:
  - NET_BIND_SERVICE
  # Required to prevent escalations to root.
  allowPrivilegeEscalation: true
  # Allow core volume types.
  volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    # Assume that persistentVolumes set up by the cluster admin are safe to use.
    - 'persistentVolumeClaim'
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    # Require the container to run without root privileges.
    rule: 'MustRunAsNonRoot'
  seLinux:
    # This policy assumes the nodes are using AppArmor rather than SELinux.
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  fsGroup:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  readOnlyRootFilesystem: false
---
# Cluster role which grants access to the restricted pod security policy
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: oke-restricted-psp-clsrole
rules:
- apiGroups:
  - extensions
  resourceNames:
  - oke-restricted-psp
  resources:
  - podsecuritypolicies
  verbs:
  - use

Error: cannot delete Pods with local storage when kubectl drain in OKE

Symptom

In the Kubernetes world, we often need to upgrade Kubernetes Master nodes and worker nodes themselves.
In OKE (Oracle Kubernetes Engine)  we follow Master node upgrade guide  and Worker node upgrade guide

When we run kubectl drain <node name>  --ignore-daemonsets

We got :
error: cannot delete Pods with local storage (use --delete-local-data to override): monitoring/grafana-65b66797b7-d8gzv, monitoring/prometheus-adapter-8bbfdc6db-pqjck

Solution:

 The error is due to there is local storage (  emptyDir: {} ) attached in the pods.

For Statefulset 

Please use volumeClaimTemplates . OKE supports the automatic movement of PV PVC to a new worker node. It will detach and reattach the block storages to the new work node(assume they are in the same Availablity domain).  We don't need to worry about data migration which is excellent for statefulset.  example is:

volumeClaimTemplates:
  - metadata:
      name: prometheus-storage
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 50Gi
      selector:
        matchLabels:
          app: grafana
      storageClassName: oci
      volumeMode: Filesystem

For Deployment,

Because deployment is stateless, we can delete local data
kubectl drain  <node name> --ignore-daemonsets --delete-local-data
If the deployment has PV,PVC attached, same as statefulset,  OKE supports the automatic movement of PV PVC to a new worker node. It will detach and reattach the block storages to the new work node(assume they are in the same Availablity domain).  We don't need to worry about data migration.

Tip: A few commands to triage Kubernetes Network related Issues

tcptraceroute 100.95.96.3 53

nc -vz 147.154.4.38 53TCP port

nc -vz -u 147.154.4.38 53 UDP port

tcpdump -ni ens3 port 31530

for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done