Tip: error: failed to load key pair tls: failed to parse private key

 Symptom:

    When we kubectl create secret tls ..., we hit below error

error: failed to load key pair tls: failed to parse private key

Reason:

    It is likely the private key file is encrypted with a passphrase.

   Use openssl to unencrypt it and use the new key for kubectl 

openssl rsa -in encrypted-private.key -out unencrypted.key

 Enter pass phrase for ...... 

 

 

Tip: OpenSSL SSL_connect: SSL_ERROR_SYSCALL

 Symptoms:

We use curl -v https://<domain> to test if the network traffic is allowed
The expected result would be like 

*  Trying 12.12.12.12:443...

* TCP_NODELAY set

* Connected to ***  port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

*   CAfile: /etc/ssl/certs/ca-certificates.crt

  CApath: /etc/ssl/certs

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

* TLSv1.3 (IN), TLS handshake, Server hello (2):

* TLSv1.2 (IN), TLS handshake, Certificate (11):

* TLSv1.2 (OUT), TLS alert, unknown CA (560):

However, we see this error :

*  Trying 12.12.12.12:443...

* TCP_NODELAY set

* Connected to *** port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

*   CAfile: /etc/ssl/certs/ca-certificates.crt

  CApath: /etc/ssl/certs

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to *** :443

Solution:

From the output, we see 443 is open but the TLS handshake, Server hello is missing. We have mid-tiers to handle TLS certificates. So it is very likely that the network is interrupted between  LB and mid-tiers where TLS is being handled.  It would be a good approach to double-check firewall ports between them. :)

Another reason is ingress controller pods were stuck may bounce or scale up to workaround it

Tip: Add Password Into Existing Private Keys

Add Password:

openssl rsa -aes256 -in your-private-key.pem -out your-private-key-encrypted.pem
writing RSA key
Enter PEM pass phrase:   ****
Verifying - Enter PEM pass phrase: ****

Remove Password:

openssl rsa -in  your-private-key-encrypted.pem -out your-private-key.pem
Private key passphrase:  ****

How To Add TLS Certificate Into Ingress in OKE

Requirement:

         In order to secure the traffic, we need to deploy  TLS certificates into our ingress running in OKE.  We are going to use self-signed certificates to demonstrate it. 

Solution:

• Generate self-signed certificates via openssl
  • openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout tls.key -out tls.crt  -config req.conf -extensions 'v3_req'

  • req.conf:
    
    [req]
    distinguished_name = ingress_tls_prometheus_test
    x509_extensions = v3_req
    prompt = no
    [ingress_tls_prometheus_test]
    C = US
    ST = VA
    L = NY
    O = BAR
    OU = BAR
    CN = www.bar.com
    [v3_req]
    keyUsage = keyEncipherment, dataEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = prometheus.bar.com
    DNS.2 = grafana.bar.com
    DNS.3 = alertmanager.bar.com

    
    
  • To verify it:    openssl x509 -in tls.crt -noout -text
• Create Kubernetes TLS secret for that
  • kubectl create secret tls tls-prometheus-test --key tls.key --cert tls.crt -n monitoring
• Add TLS section into the ingress yaml file. Example:

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: prometheus-ingress
      namespace: monitoring
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      tls:
      - hosts:
        - prometheus.bar.com
        secretName: tls-prometheus-test
      - hosts:
        - grafana.bar.com
        secretName: tls-prometheus-test
      - hosts:
        - alertmanager.bar.com
        secretName: tls-prometheus-test
      rules:
      - host: prometheus.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: prometheus-k8s
              servicePort: 9090
      - host: grafana.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: grafana
              servicePort: 3000
      - host: alertmanager.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: alertmanager-main
              servicePort: 9093

• Ingress controller would redirect http traffic to https traffic automatically for these 3 domains 
• Spoof IP address for DNS names via the below entry and take off www proxy of the browser if necessary.
  • 123.123.123.123         prometheus.bar.com  grafana.bar.com  alertmanager.bar.com

Cross Namespace Ingress Usage Example in OKE

Requirement:

      The normal use case to create ingress is to create one in the application namespace where application services and TLS certificates/keys are sitting.

      In the enterprise world, the security team is not comfortable to store TLS private keys in the application namespace. TLS private keys need to be stored securely in the namespace of the ingress controller.   In this case, we need to create ingress in "ingress controller" namespace instead of the application namespace.   We need to find a way to let ingress in "ingress controller" namespace to point to services in the application namespace (cross namespace service ).  Below is the solution of how we can achieve that in OKE ( Oracle Kubernetes Engine).

Solution:

• Create TLS secrets in ingress controller namespace. Refer doc

  • $ openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout tls.key -out tls.crt  -config req.conf -extensions 'v3_req'
    
    req.conf:
    [req]
    distinguished_name = ingress_tls_prometheus_test
    x509_extensions = v3_req
    prompt = no
    [ingress_tls_prometheus_test]
    C = US
    ST = VA
    L = NY
    O = BAR
    OU = BAR
    CN = www.bar.com
    [v3_req]
    keyUsage = keyEncipherment, dataEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = prometheus.bar.com
    DNS.2 = grafana.bar.com
    DNS.3 = alertmanager.bar.com

  • kubectl create secret tls tls-prometheus-test  --key tls.key --cert tls.crt -n ingress-nginx

• The key to using services in different namespaces is ExternalName.  It is working in OKE, but may not be working other  Cloud providers. One of the externalname examples is:

  • apiVersion: v1
    kind: Service
    metadata:
      annotations:
      name: prometheus-k8s-svc
      namespace: ingress-nginx
    spec:
      externalName: prometheus-k8s.monitoring.svc.cluster.local
      ports:
      - port: 9090
        protocol: TCP
        targetPort: 9090
      type: ExternalName

• Create ingress in ingress controller namespace.

  • apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: prometheus-ingress
      namespace: ingress-nginx
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      tls:
      - hosts:
        - prometheus.bar.com
        - grafana.bar.com
        - alertmanager.bar.com
        secretName: tls-prometheus-test
      rules:
      - host: prometheus.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: prometheus-k8s-svc
              servicePort: 9090
      - host: grafana.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: grafana-svc
              servicePort: 3000
      - host: alertmanager.bar.com
        http:
          paths:
          - path: /
            backend:
              serviceName: alertmanager-main-svc
              servicePort: 9093

How To Generate Self-Signed Multiple SAN Certificate Using OpenSSL

Requirement:

    In our development services, we often need to have self-signed certificates. Sometimes we need to add SAN into such a certificate.  Below is how we use OpenSSL to achieve it.

Solution:

$ openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout tls.key -out tls.crt  -config req.conf -extensions 'v3_req'

req.conf:

[req]
distinguished_name = ingress_tls_prometheus_test
x509_extensions = v3_req
prompt = no
[ingress_tls_prometheus_test]
C = US
ST = VA
L = NY
O = BAR
OU = BAR
CN = www.bar.com
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = prometheus.bar.com
DNS.2 = grafana.bar.com
DNS.3 = alertmanager.bar.com

To verify it:    openssl x509 -in tls.crt -noout -text

Tip of How to Verify CA chain of Kubernetes

Find root CA file of the Kubernetes cluster
Find Client certificate file
They could be in /etc/kubernetes/pki  or  /var/lib/kubelet

# openssl  verify  -CAfile ca.crt  apiserver-kubelet-client.crt
apiserver-kubelet-client.crt: OK

Tip of How To Find When Kubernetes CA Expires

Find the locations of CA files
They could be in /etc/kubernetes/pki  or  /var/lib/kubelet
Use openssl

# openssl x509 -in  ca.crt  -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Oct 30 08:01:56 2018 GMT
            Not After : Oct 27 08:01:56 2028 GMT
        Subject: CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:cd:ab:2d:f3:86:f3:57:28:43:4f:08:93:a6:e7:
                    22:b0:51:73:74:cb:5a:3c:ee:01:2d:ad:1c:a4:97:
                 .........