Sunday, October 11, 2020

Tip: Error Http 504 gateway timeout on ingress controller


    We have micro-services behind our ingress controller in our Kubernetes cluster. We are hitting HTTP 504 error in our ingress controller logs intermittently. - - [01/Oct/2020:20:32:13 +0000] "GET /mos/products?limit=50&offset=0&orderBy=Name%3Aasc HTTP/2.0" 504 173 "https://ep******" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Firefox/78.0" 1578 180.004 [ingress-nginx-external2-mag-oke-products-svc-8080] [],, 0, 0, 0 60.001, 60.001, 60.002 504, 504, 504 c5b8cb67927d3997b4019e9830762694

  Bounce ingress controller would fix the issues temporarily.


  We find the issues are caused parameters of nginx which stated

Add below annotations into ingress rules to fix it "5" "10"

Friday, October 02, 2020

Tip:Node and Namespace drop down menu missing node names in Grafana


      We have Prometheus and Grafana setup running well. Suddenly the node and namespace drop-down list disappeared.  No config changes were made. 


   It is very likely the kube-state-metrics service have problems. That's the place grafana get the info from.

   Bounce the pod or recreate the deployment to fix it