How To Restart Fluentd Process in K8S Without Changing Pod Name

Requirement:

   When we debug K8S apps running in Pods, we often delete pods to force restart apps in the Pods to reload configuration.  For Statefulset , the name of Pod won't be changed. However for deployment and Daemonset, new pod names are generated after we delete the pods, it add some difficulty to track which pod we are debug the apps. We can find an easier way to restart apps in the pods while keeping same pod name.

Solution:

We use fluentd as an example.

• We update the config file of fluentd in the pod. ie  update /etc/fluent/fluent.conf file
• We backup existing docker images via docker tag

docker tag k8s.gcr.io/fluentd-elasticsearch:v.2.0.4 k8s.gcr.io/fluentd-elasticsearch:backup

• We commit the changes on conf file into the docker images. Othewise all your changes will be lost after bounce. Use docker images|grep fluent to find correct name of K8S pod.

docker commit  <full k8s pod name in docker>   k8s.gcr.io/fluentd-elasticsearch:v.2.0.4

• Use kubectl exec -it  <pod name>  /bin/bash  to get into the pod
• As ps is not installed by default in many pods standard. We can use "find" command to find which process fluentd is running. 

find /proc -mindepth 2 -maxdepth 2 -name exe -exec ls -lh {} \; 2>/dev/null 

• We can use kill to send signals to Fluentd process . see doc link . ie we send SIGHUP to ask process to reload the conf file. It is quite possible the fluend just restart and trigger pod bounce itself. It would be fine as we have committed changes in the docker images.

Kill -SIGHUP  8

• In this way, the pod name would be kepted, you can have the same pod name after pod bounce.