In our last blog, we explained how to handle rolling deployments of Rails applications with no downtime.
In this article we will walk you through how to handle graceful shutdown of processes in Kubernetes.
When we deploy Rails applications on kubernetes it stops existing pods and spins up new ones. When old pod is terminated by Replicaset, then active Sidekiq processes are also terminated. We run our batch jobs using sidekiq and it is possible that sidekiq jobs might be running when deployment is being performed. Terminating old pod during deployment can kill the already running jobs.
As per default pod termination policy of kubernetes, kubernetes sends command to delete pod with a default grace period of 30 seconds. At this time kubernetes sends TERM signal. When the grace period expires, any processes still running in the Pod are killed with SIGKILL.
We can adjust the terminationGracePeriodSeconds timeout as per our need and can change it from 30 seconds to 2 minutes.
However there might be cases where we are not sure how much time a process takes to gracefully shutdown. In such cases we should consider using PreStop hook which is our next solution.
Kubernetes provides many Container lifecycle hooks.
PreStop hook is called immediately before a container is terminated. It is a blocking call. It means it is synchronous. It also means that this hook must be completed before the container is terminated.
Note that unlike solution1 this solution is not time bound. Kubernetes will wait as long as it takes for PreStop process to finish. It is never a good idea to have a process which takes more than a minute to shutdown but in real world there are cases where more time is needed. Use PreStop for such cases.
We decided to use preStop hook to stop Sidekiq because we had some really long running processes.
Using PreStop hooks in Sidekiq deployment
This is a simple deployment template which terminates Sidekiq process when pod is terminated during deployment.
1apiVersion: v1 2kind: Deployment 3metadata: 4 name: test-staging-sidekiq 5 labels: 6 app: test-staging 7 namespace: test 8spec: 9 template: 10 metadata: 11 labels: 12 app: test-staging 13 spec: 14 containers: 15 - image: <your-repo>/<your-image-name>:latest 16 name: test-staging 17 imagePullPolicy: Always 18 env: 19 - name: REDIS_HOST 20 value: test-staging-redis 21 - name: APP_ENV 22 value: staging 23 - name: CLIENT 24 value: test 25 volumeMounts: 26 - mountPath: /etc/sidekiq/config 27 name: test-staging-sidekiq 28 ports: 29 - containerPort: 80 30 volumes: 31 - name: test-staging-sidekiq 32 configMap: 33 name: test-staging-sidekiq 34 items: 35 - key: config 36 path: sidekiq.yml 37 imagePullSecrets: 38 - name: registrykey
Next we will use PreStop lifecycle hook to stop Sidekiq safely before pod termination.
We will add the following block in deployment manifest.
1lifecycle: 2 preStop: 3 exec: 4 command: 5 [ 6 "/bin/bash", 7 "-l", 8 "-c", 9 "cd /opt/myapp/current; for f in tmp/pids/sidekiq*.pid; do bundle exec sidekiqctl stop $f; done", 10 ]
PreStop hook stops all the Sidekiq processes and does graceful shutdown of Sidekiq before terminating the pod.
We can add this configuration in original deployment manifest.
1apiVersion: v1 2kind: Deployment 3metadata: 4 name: test-staging-sidekiq 5 labels: 6 app: test-staging 7 namespace: test 8spec: 9 replicas: 1 10 template: 11 metadata: 12 labels: 13 app: test-staging 14 spec: 15 containers: 16 - image: <your-repo>/<your-image-name>:latest 17 name: test-staging 18 imagePullPolicy: Always 19 lifecycle: 20 preStop: 21 exec: 22 command: 23 [ 24 "/bin/bash", 25 "-l", 26 "-c", 27 "cd /opt/myapp/current; for f in tmp/pids/sidekiq*.pid; do bundle exec sidekiqctl stop $f; done", 28 ] 29 env: 30 - name: REDIS_HOST 31 value: test-staging-redis 32 - name: APP_ENV 33 value: staging 34 - name: CLIENT 35 value: test 36 volumeMounts: 37 - mountPath: /etc/sidekiq/config 38 name: test-staging-sidekiq 39 ports: 40 - containerPort: 80 41 volumes: 42 - name: test-staging-sidekiq 43 configMap: 44 name: test-staging-sidekiq 45 items: 46 - key: config 47 path: sidekiq.yml 48 49 imagePullSecrets: 50 - name: registrykey
Let's launch this deployment and monitor the rolling deployment.
1 2$ kubectl apply -f test-deployment.yml 3deployment "test-staging-sidekiq" configured 4
We can confirm that existing Sidekiq jobs are completed before termination of old pod during the deployment process. In this way we handle a graceful shutdown of Sidekiq process. We can apply this technique to other processes as well.