Thursday, April 22, 2021

Tip: curl: (23) Failed writing body


When we run 

curl -sSL -o /usr/local/bin/argocd$VERSION/argocd-linux-amd64

We get error

curl: (23) Failed writing body (0 != 1369)


 Tt is due to  "/usr/local/bin/argocd"  is on the /usr/local/bin  directory which is owned by root user while we use normal user to run curl.

To fix it , change "/usr/local/bin/argocd" to be "/tmp/argocd"

Wednesday, April 14, 2021

Tip: git can't communicate with github after unset http.proxy


    We used to have an HTTP proxy to access Github. It was working fine. When we take off HTTP proxy via "git config --global -e", use "git config --global -l" to confirm it is taken off.

   However, it still can't communicate with GitHub. Error like 

 kex_exchange_identification: Connection closed by remote host fatal: Could not read from remote repository


   It is due to we use ssh to communicate with GitHub, while there are extra HTTP proxy settings in ~/.ssh/config file

ProxyCommand=socat - PROXY:<proxy-server>:%h:%p,proxyport=80

Take them off will fix the issue. 

Tuesday, April 13, 2021

Tip: When OPA gatekeeper stuck


    We hit issues that all kubectl command stuck like kubectl get pod...etc

    initially, we thought it is a Kubernetes control plane issue but confirmed with the cloud provider, the control plane has some communication issues with the webhook


  It turns out the OPA gatekeeper was stuck and cause webhook issues with the control plane.


1. Delete webhook

kubectl delete gatekeeper-validating-webhook-configuration

2. It will stabilize the communications with the control plane

3. Delete and redeploy opa keeper deployment 

Thursday, April 08, 2021

Tip: error: failed to load key pair tls: failed to parse private key


    When we kubectl create secret tls ..., we hit below error

error: failed to load key pair tls: failed to parse private key


    It is likely the private key file is encrypted with a passphrase.

   Use openssl to unencrypt it and use the new key for kubectl 

openssl rsa -in encrypted-private.key -out unencrypted.key

 Enter pass phrase for ...... 



Wednesday, April 07, 2021

Tip: Pods keep crashloopbackoff


 Pods always crashloopbackoff 

"kubectl describe pod..."  does not give meaningful info, as well as "kubectl get events"


One of the likely reason is related to pod security policy. My situation is the existing pod security policy does not allow Nginx or Apache to run. It does not have



  # apache or nginx need escalation to root to function well

  allowPrivilegeEscalation: true

So the pods keep crashloopbackoff. To fix it is to add the above into the pod security policy.

Saturday, April 03, 2021

Tip: Istio TLS secrets, Gateway, VirtualService namespace scope

There is some confusion about where we should put istio objects. Is it in the istio-system or users namespace?

Here are some tips:

For TLS,mTLS CA, certs, key management in istio, the Kubernetes secrets should be created in the istio-system. Not in users' namespace

Gateway and VirtualService need to be created on the users' namespace