Kubernetes has a couple of different ways to scale deployments automatically based on the load the application receives. The Horizontal Pod Autoscaler (HPA) can be used out of the box in a Kubernetes cluster to increase or decrease the number of Pods of your deployment. By default, HPA supports scaling based on CPU and memory usage, served by the metrics server.

While building NeetoDeploy initially, we'd set up to scale deployments based on CPU and memory usage, since these were the default metrics supported by the HPA. However, later we wanted to scale deployments based on the average response time of our application.

This is an example of a case where the metric we want to scale is not directly related to the CPU or the memory usage. Other examples of this could be network metrics from the load balancer, like the number of requests received in the application. In this blog, we will discuss how we achieved autoscaling of deployments in Kubernetes based on the average response time using prometheus-adapter.

When an application receives a lot of requests suddenly, this creates a spike in the average response time. The CPU and memory metrics also spike, but they take longer to catch up. In such cases, being able to scale deployments based on the response time will ensure that the spike in traffic is handled gracefully.

Prometheus is one of the most popular cloud native monitoring tools and the Kubernetes HPA can be extended to scale deployments based on metrics exposed by Prometheus. We used the prometheus-adapter to build autoscaling based on the average response time in NeetoDeploy.

Setting up the custom metrics

We took the following steps to make our HPAs work with Prometheus metrics.

1. Installed prometheus-adapter in our cluster.
2. Configured the metric we wanted for our HPAs as a custom metric in the prometheus-adapter.
3. Confirmed that the metric is added to the custom.metrics.k8s.io API endpoint.
4. Configured an HPA with the custom metric.

Install prometheus-adapter in the cluster

prometheus-adapter is an implementation of the custom.metrics.k8s.io API using Prometheus. We used the prometheus-adapter to set up Kubernetes metrics APIs for our Promtheus metrics, which then can be used with our HPAs.

We installed prometheus-adapter in our cluster using Helm. We got a template for the values file for the Helm installation here.

We made a few changes to the file before we applied it to our cluster and deployed prometheus-adapter:

1. We made sure that the Prometheus deployment is configured properly by giving the correct service url and port.

# values.yaml
prometheus:
  # Value is templated
  url: http://prometheus.monitoring.svc.cluster.local
  port: 9090
  path: ""
# ... rest of the file

2. We made sure that the custom metrics that we needed for our HPA are configured under rules.custom in the values.yaml file. In the following example, we are using the custom metric traefik_service_avg_response_time since we'll be using that to calculate the average response time for each deployment.

# values.yaml
rules:
  default: false

  custom:
    - seriesQuery:'{__name__=~"traefik_service_avg_response_time", service!=""}'
      resources:
        overrides:
          app_name:
            resource: service
          namespace:
            resource: namespace
      metricsQuery: traefik_service_avg_response_time{<<.LabelMatchers>>}

Once we configured our values.yaml file properly, we installed prometheus-adapter in our cluster with Helm.

helm repo add prometheus https://prometheus-community.github.io/helm-charts
helm repo update
helm install prom-adapter prometheus-community/prometheus-adapter --values values.yaml

Query for custom metric

Once we got prometheus-adapter running, we queried our cluster to check if the custom metric is coming up in the custom.metrics.k8s.io API endpoint.

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq

The response looked like this:

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "services/traefik_service_avg_response_time",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": ["get"]
    },
    {
      "name": "namespaces/traefik_service_avg_response_time",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": ["get"]
    }
  ]
}

We also queried the metric API for a particular service we've configured the metric for. Here, we're querying the traefik_service_avg_response_time metric for the neeto-chat-web-staging app in the default namespace.

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/neeto-chat-web-staging/traefik_service_avg_response_time | jq

The API response gave the following.

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "describedObject": {
        "kind": "Service",
        "namespace": "default",
        "name": "neeto-chat-web-staging",
        "apiVersion": "/v1"
      },
      "metricName": "traefik_service_avg_response_time",
      "timestamp": "2024-02-26T19:31:33Z",
      "value": "19m",
      "selector": null
    }
  ]
}

From the response, we can see that the average response time at the instant is reported as 19ms.

Create the HPA

Now that we're sure that prometheus-adapter is able to serve custom metrics under the custom.metrics.k8s.io API, we wired this up with a Horizontal Pod Autoscaler to scale our deployments based on our custom metric.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-name-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app-name-deployment
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Object
      object:
        metric:
          name: traefik_service_avg_response_time
          selector: { matchLabels: { app_name: my-app-name } }
        describedObject:
          apiVersion: v1
          kind: Service
          name: my-app-name
        target:
          type: Value
          value: 0.03

With everything set up, the HPA was able to fetch the custom metric scraped by Prometheus and scale our Pods up and down based on the value of the metric. We also created a recording rule in Prometheus for storing our custom metric queries and dropped the unwanted labels as a best practice. We can use the custom metric stored with the recording rule directly with prometheus-adapter to expose the metrics as an API endpoint in Kubernetes. This is helpful when your custom metric queries are complex.

If your application runs on Heroku, you can deploy it on NeetoDeploy without any change. If you want to give NeetoDeploy a try, then please send us an email at invite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, follow NeetoDeploy on X. You can also join our community Slack to chat with us about any Neeto product.

Prometheus along with Grafana is a highly scalable open-source monitoring framework for container orchestration platform. Prometheus probes the application and collects various data. It stores all this data in its time series database. Grafana is a visualization tool. It uses the data from the database to show the data that is meaningful to the user.

Both Prometheus and Grafana are gaining popularity in the observability space as it helps with metrics and alerts. Learning to integrate them using Helm will allow us to monitor our Kubernetes cluster and troubleshoot problems easily. Furthermore, we can deep dive into our cluster's well-being and efficiency, focusing on resource usage and performance metrics within our Kubernetes environment.

We will also learn how to create a simple dashboard on Grafana.

Why using Prometheus and Grafana for monitoring is good

Using Prometheus and Grafana for monitoring has many benefits:

• Scalability: Both tools are highly scalable and can handle the monitoring needs of small to large Kubernetes clusters.
• Flexibility: They allow us to create custom dashboards tailored to our specific monitoring requirements.
• Real-time Monitoring: Prometheus provides real-time monitoring, helping us to quickly detect and respond to issues.
• Alerting: Prometheus enables us to set up alerts based on specific metrics, so we can be notified when issues arise.
• Data Visualization: Grafana offers powerful data visualization capabilities, making it easier to understand complex data.
• Open Source: Both Prometheus and Grafana are open-source, reducing monitoring costs.
• Community Support: We can benefit from active communities, ensuring continuous development and support.
• Integration: They seamlessly integrate with other Kubernetes components and applications, simplifying setup.
• Historical Data: Grafana allows us to explore historical data, aiding in long-term analysis and trend identification.
• Extensible: Both tools are extensible, allowing us to integrate additional data sources and plugins.
• Efficient Resource Usage: Prometheus efficiently utilizes resources, ensuring minimal impact on our cluster's performance.

Two common ways to use Prometheus and Grafana on Kubernetes:

1. Manual Kubernetes deployment: In this method, we need to write  Kubernetes Deployments  and  Services for both Prometheus and Grafana. In the YAML file, we need to put all the settings for Prometheus and Grafana on Kubernetes. Then we send these files to our Kubernetes cluster. But we can end up with many YAML files, which can be hard. If we make a mistake in any YAML file, Prometheus and Grafana won't work on Kubernetes.
2. Using Helm: This is an easy way to send any application container to Kubernetes. Helm is the official package manager for Kubernetes. With Helm, we can make installing, sending, and managing Kubernetes makes applications easier.

A Helm Chart has all the YAML files:

• Deployments.
• Services.
• Secrets.
• ConfigMaps manifests.

We use these files to send the application container to Kubernetes. Instead of making individual YAML files for each application container, Helm lets us download Helm charts that already have YAML files.

Setting up Prometheus and Grafana using Helm chart

We will use ArtifactHub, which offers public and private repositories for Helm Charts. We will use these Helm Charts to arrange the pods and services in our Kubernetes cluster.

To get Prometheus and Grafana working on Kubernetes with Helm, we will start by installing Helm.

Installing Helm on Linux

sudo apt-get install helm

Installing Helm on Windows

choco install Kubernetes-helm

Installing Helm on macOS

brew install helm

We can check out the official  Helm documentation  if we run into any issues while installing Helm.

The image below represents the successful Helm installation on macOS.

For this blog, we’re going to install Helm chart and by default, this chart also installs additional, dependent charts (including Grafana):

• prometheus-community/kube-state-metrics
• prometheus-community/prometheus-node-exporter
• grafana/grafana

To get this Helm chart, let's run this command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

We have downloaded the latest version of Prometheus & Grafana.

To install the Prometheus Helm Chart on a Kubernetes Cluster, let's run the following command:

helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack

We have successfully installed Prometheus & Grafana on the Kubernetes Cluster. We can access the Prometheus & Grafana servers via ports 9090 & 80, respectively.

Now, let's run the following command to view all the resources created by the Helm Chart in our Kubernetes cluster:

kubectl get all

The Helm chart created the following resources:

• Pods: It hosts the deployed Prometheus Kubernetes application inside the cluster.
• Replica Sets: A collection of instances of the same application inside the Kubernetes cluster. It enhances application reliability.
• Deployments: It is the blueprint for creating the application pods.
• Services: It exposes the pods running inside the Kubernetes cluster. We use it to access the deployed Kubernetes application.
• Stateful Sets: They manage the deployment of the stateful application components and ensure stable and predictable network identities for these components.
• Daemon Sets: They ensure that all (or a specific set of) nodes run a copy of a pod, which is useful for tasks such as logging, monitoring, and other node-specific operations.

Run this command to view all the Kubernetes Services for Prometheus & Grafana:

kubectl get service

Listed services for Prometheus and Grafana are:

• alertmanager-operated
• kube-prometheus-stack-alertmanager
• kube-prometheus-stack-grafana
• kube-prometheus-stack-kube-state-metrics
• kube-prometheus-stack-operator
• kube-prometheus-stack-prometheus
• kube-prometheus-stack-prometheus-node-exporter
• prometheus-operated

kube-prometheus-stack-grafana and kube-prometheus-stack-prometheus are the ClusterIP type services, which means we can only access them within the Kubernetes cluster.

To expose the Prometheus and Grafana to be accessed outside the Kubernetes cluster , we can either use the NodeIP or LoadBalance service.

Exposing Prometheus and Grafana using NodePort services

Let's run the following command to expose the Prometheus Kubernetes service:

kubectl expose service kube-prometheus-stack-prometheus --type=NodePort --target-port=9090 --name=prometheus-node-port-service

kubectl expose service kube-prometheus-stack-grafana --type=NodePort --target-port=3000 --name=grafana-node-port-service

That command will create new services of NodePort type & make the Prometheus and Grafana is accessible outside the Kubernetes Cluster on ports 9090 and 80.

As we can see, the grafana-node-port-service and prometheus-node-port-service are successfully created and are being exposed on node ports 32489 & 30905

Now, we can run this command and get the external IP of any node to access the Prometheus and Grafana:

kubectl get nodes -o wide

We can use the External-IP and the node ports to access the Prometheus and Grafana dashboards outside the cluster environment.

Prometheus Dashboard

Grafana Dashboard

Run this command, to get the password for the admin user of the Grafana dashboard:

kubectl get secret --namespace default kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Grafana Dashboard

Upon login to the Grafana dashboard, use admin as the username and our generated password. We will see "Welcome to Grafana" homepage as shown below.

Since we used the Kube Prometheus Stack helm chart, the data source for Prometheus and Alert Manager is added by default.

We can add more data sources by clicking on the Add new data source button on the top right side.

By default, this Helm chart adds multiple dashboards to monitor the health of the Kubernetes cluster and its resources.

Additionally, we also have the option of creating our dashboards from scratch as well as importing multiple Grafana dashboards provided by the Grafana library.

To import a Grafana Dashboard, let's follow these steps:

• From this Grafana library, we can add any dashboard

  

• Select Dashboard and copy the Dashboard ID

  

• Under Dashboards page we can get the Import option

  

• Under "Import Dashboard" page, we need to paste the Dashboard IP that we copied earlier & click on the Load button.

  

• After clicking on the Load button, it will auto-load the dashboard from the library after which we can import the dashboard by clicking on the Import button.

  

• Once the import is complete, we’ll be redirected to the new imported dashboard which’ll also be visible under the Dashboards page.

  

  We can use this Node Exporter dashboard to monitor & observe the health of our nodes present in our Kubernetes Cluster.

Conclusion

In this blog, we learned how to integrate Prometheus and Grafana using the helm chart. We also learned how to import dashboards into Grafana from the Grafana library.

In the next blog, we will explore how to integrate Grafana Loki with Grafana and collect and store event-related metrics using the Kubernetes Event Exporter.

While using Kubernetes as a distributed system for container deployments, modularity and reusability can be achieved using parameterizing containers to deploy micro services.

Parameterized containers

Assuming container as a function in a program, how many parameters does it have? Each parameter represents an input that can customize a generic container to a specific situation.

Let's assume we have a Rails application isolated in services like puma, sidekiq/delayed-job and websocket. Each service runs as a separate deployment on a separate container for the same application. When deploying the change we should be building the same image for all the three containers but they should be different function/processes. In our case, we will be running 3 pods with the same image. This can be achieved by building a generic image for containers. The Generic container must be accepting parameters to run different services.

We need to expose parameters and consume them inside the container. There are two ways to pass parameters to our container.

1. Using environment variables.
2. Using command line arguments.

In this article, we will use environment variables to run parameterized containers like puma, sidekiq/delayed-job and websocket for Rails applications on kubernetes.

We will deploy wheel on kubernetes using parametrized container approach.

Pre-requisite

• Understanding of Dockerfile and image building.

• Access to working kubernetes cluster.

• Understanding of Kubernetes terms like pods, deployments, services, configmap, and annotations.

Building a generic container image.

Dockerfile (Link is not available) in wheel uses bash script setup_while_container_init.sh as a command to start a container. The script is self-explanatory and, as we can see, it consists of two functions web and background. Function web starts the puma service and background starts the delayed_job service.

We create two different deployments on kubernetes for web and background services. Deployment templates are identical for both web and background. The value of environment variable POD_TYPE to init-script runs the particular service in a pod.

Once we have docker image built, let's deploy the application.

Creating kubernetes deployment manifests for wheel application

Wheel uses PostgreSQL database and we need postgres service to run the application. We will use the postgres image from docker hub and will deploy it as deployment.

Note: For production deployments, database should be deployed as a statefulset or use managed database services.

K8s manifest for deploying PostgreSQL.

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: db
  name: db
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - image: postgres:9.4
          name: db
          env:
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              value: welcome

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: db
  name: db
spec:
  ports:
    - name: headless
      port: 5432
      targetPort: 5432
  selector:
    app: db

Create Postgres DB and the service.

$ kubectl create -f db-deployment.yml -f db-service.yml
deployment db created
service db created

Now that DB is available, we need to access it from the application using database.yml.

We will create configmap to store database credentials and mount it on the config/database.yml in our application deployments.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: database-config
data:
  database.yml: |
    development:
      adapter: postgresql
      database: wheel_development
      host: db
      username: postgres
      password: welcome
      pool: 5

    test:
      adapter: postgresql
      database: wheel_test
      host: db
      username: postgres
      password: welcome
      pool: 5

    staging:
      adapter: postgresql
      database: postgres
      host: db
      username: postgres
      password: welcome
      pool: 5

Create configmap for database.yml.

$ kubectl create -f database-configmap.yml
configmap database-config created

We have the database ready for our application, now let's proceed to deploy our Rails services.

Deploying Rails micro-services using the same docker image

In this blog, we will limit our services to web and background with kubernetes deployment.

Let's create a deployment and service for our web application.

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: wheel-web
  labels:
    app: wheel-web
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: wheel-web
    spec:
      containers:
      - image: bigbinary/wheel:generic
        name: web
        imagePullPolicy: Always
        env:
        - name: DEPLOY_TIME
          value: $date
          value: staging
        - name: POD_TYPE
          value: WEB
        ports:
        - containerPort: 80
        volumeMounts:
          - name: database-config
            mountPath: /wheel/config/database.yml
            subPath: database.yml
      volumes:
        - name: database-config
          configMap:
            name: database-config

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app: wheel-web
  name: web
spec:
  ports:
  - name: puma
    port: 80
    targetPort: 80
  selector:
    app: wheel-web
  type: LoadBalancer

Note that we used POD_TYPE as WEB, which will start the puma process from the container startup script.

Let's create a web/puma deployment and service.

kubectl create -f web-deployment.yml -f web-service.yml
deployment wheel-web created
service web created

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: wheel-background
  labels:
    app: wheel-background
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: wheel-background
    spec:
      containers:
        - image: bigbinary/wheel:generic
          name: background
          imagePullPolicy: Always
          env:
            - name: DEPLOY_TIME
              value: $date
            - name: POD_TYPE
              value: background
          ports:
            - containerPort: 80
          volumeMounts:
            - name: database-config
              mountPath: /wheel/config/database.yml
              subPath: database.yml
      volumes:
        - name: database-config
          configMap:
            name: database-config

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: wheel-background
  name: background
spec:
  ports:
    - name: background
      port: 80
      targetPort: 80
  selector:
    app: wheel-background

For background/delayed-job we set POD_TYPE as background. It will start delayed-job process.

Let's create background deployment and the service.

kubectl create -f background-deployment.yml -f background-service.yml
deployment wheel-background created
service background created

Get application endpoint.

$ kubectl get svc web -o wide | awk '{print $4}'
a55714dd1a22d11e88d4b0a87a399dcf-2144329260.us-east-1.elb.amazonaws.com

We can access the application using the endpoint.

Now let's see pods.

$ kubectl get pods
NAME                                READY     STATUS    RESTARTS   AGE
db-5f7d5c96f7-x9fll                 1/1       Running   0          1h
wheel-background-6c7cbb4c75-sd9sd   1/1       Running   0          30m
wheel-web-f5cbf47bd-7hzp8           1/1       Running   0          10m

We see that db pod is running postgres, wheel-web pod is running puma and wheel-background pod is running delayed job.

If we check logs, everything coming to puma is handled by web pod. All the background jobs are handled by background pod.

Similarly, if we are using websocket, separate API pods, traffic will be routed to respective services.

This is how we can deploy Rails micro services using parametrized containers and a generic image.

We deploy one of our Rails applications on a dedicated Kubernetes cluster. Kubernetes is a good fit for us since as per the load and resource consumption, Kubernetes horizontally scales the containerized application automatically. The prerequisite to deploy any kind of application on Kubernetes is that the application needs to be containerized. We use Docker to containerize our application.

We have been successfully containerizing and deploying our Rails application on Kubernetes for about a year now. Although containerization was working fine, we were not happy with the overall time spent to containerize the application whenever we changed the source code and deployed the app.

We use Jenkins for building on-demand Docker images of our application with the help of CloudBees Docker Build and Publish plugin.

We observed that the average build time of a Jenkins job to build a Docker image was about 9 to 10 minutes.

Investigating what takes most time

We wipe the workspace folder of the Jenkins job after finishing each Jenkins build to avoid any unintentional behavior caused by the residue left from a previous build. The application's folder is about 500 MiB in size. Each Jenkins build spends about 20 seconds to perform a shallow Git clone of the latest commit of the specified git branch from our remote GitHub repository.

After cloning the latest source code, Jenkins executes docker build command to build a Docker image with a unique tag to containerize the cloned source code of the application.

Jenkins build spends another 10 seconds invoking docker build command and sending build context to Docker daemon.

01:05:43 [docker-builder] $ docker build --build-arg RAILS_ENV=production -t bigbinary/xyz:production-role-management-feature-1529436929 --pull=true --file=./Dockerfile /var/lib/jenkins/workspace/docker-builder
01:05:53 Sending build context to Docker daemon 489.4 MB

We use the same Docker image on a number of Kubernetes pods. Therefore, we do not want to execute bundle install and rake assets:precompile tasks while starting a container in each pod which would prevent that pod from accepting any requests until these tasks are finished.

The recommended approach is to run bundle install and rake assets:precompile tasks while or before containerizing the Rails application.

Following is a trimmed down version of our actual Dockerfile which is used by docker build command to containerize our application.

FROM bigbinary/xyz-base:latest

ENV APP_PATH /data/app/

WORKDIR $APP_PATH

ADD . $APP_PATH

ARG RAILS_ENV

RUN bin/bundle install --without development test

RUN bin/rake assets:precompile

CMD ["bin/bundle", "exec", "puma"]

The RUN instructions in the above Dockerfile executes bundle install and rake assets:precompile tasks while building a Docker image. Therefore, when a Kubernetes pod is created using such a Docker image, Kubernetes pulls the image, starts a Docker container using that image inside the pod and runs puma server immediately.

The base Docker image which we use in the FROM instruction contains necessary system packages. We rarely need to update any system package. Therefore, an intermediate layer which may have been built previously for that instruction is reused while executing the docker build command. If the layer for FROM instruction is reused, Docker reuses cached layers for the next two instructions such as ENV and WORKDIR respectively since both of them are never changed.

01:05:53 Step 1/8 : FROM bigbinary/xyz-base:latest
01:05:53 latest: Pulling from bigbinary/xyz-base
01:05:53 Digest: sha256:193951cad605d23e38a6016e07c5d4461b742eb2a89a69b614310ebc898796f0
01:05:53 Status: Image is up to date for bigbinary/xyz-base:latest
01:05:53  ---> c2ab738db405
01:05:53 Step 2/8 : ENV APP_PATH /data/app/
01:05:53  ---> Using cache
01:05:53  ---> 5733bc978f19
01:05:53 Step 3/8 : WORKDIR $APP_PATH
01:05:53  ---> Using cache
01:05:53  ---> 0e5fbc868af8

Docker checks contents of the files in the image and calculates checksum for each file for an ADD instruction. Since source code changes often, the previously cached layer for the ADD instruction is invalidated due to the mismatching checksums. Therefore, the 4th instruction ADD in our Dockerfile has to add the local files in the provided build context to the filesystem of the image being built in a separate intermediate container instead of reusing the previously cached instruction layer. On an average, this instruction spends about 25 seconds.

01:05:53 Step 4/8 : ADD . $APP_PATH
01:06:12  ---> cbb9a6ac297e
01:06:17 Removing intermediate container 99ca98218d99

We need to build Docker images for our application using different Rails environments. To achieve that, we trigger a parameterized Jenkins build by specifying the needed Rails environment parameter. This parameter is then passed to the docker build command using --build-arg RAILS_ENV=production option. The ARG instruction in the Dockerfile defines RAILS_ENV variable and is implicitly used as an environment variable by the rest of the instructions defined just after that ARG instruction. Even if the previous ADD instruction didn't invalidate build cache; if the ARG variable is different from a previous build, then a "cache miss" occurs and the build cache is invalidated for the subsequent instructions.

01:06:17 Step 5/8 : ARG RAILS_ENV
01:06:17  ---> Running in b793b8cc2fe7
01:06:22  ---> b8a70589e384
01:06:24 Removing intermediate container b793b8cc2fe7

The next two RUN instructions are used to install gems and precompile static assets using sprockets. As earlier instruction(s) already invalidates the build cache, these RUN instructions are mostly executed instead of reusing cached layer. The bundle install command takes about 2.5 minutes and the rake assets:precompile task takes about 4.35 minutes.

01:06:24 Step 6/8 : RUN bin/bundle install --without development test
01:06:24  ---> Running in a556c7ca842a
01:06:25 bin/bundle install --without development test
01:08:22  ---> 82ab04f1ff42
01:08:40 Removing intermediate container a556c7ca842a
01:08:58 Step 7/8 : RUN bin/rake assets:precompile
01:08:58  ---> Running in b345c73a22c
01:08:58 bin/bundle exec rake assets:precompile
01:09:07 ** Invoke assets:precompile (first_time)
01:09:07 ** Invoke assets:environment (first_time)
01:09:07 ** Execute assets:environment
01:09:07 ** Invoke environment (first_time)
01:09:07 ** Execute environment
01:09:12 ** Execute assets:precompile
01:13:20  ---> 57bf04f3c111
01:13:23 Removing intermediate container b345c73a22c

Above both RUN instructions clearly looks like the main culprit which were slowing down the whole docker build command and thus the Jenkins build.

The final instruction CMD which starts the puma server takes another 10 seconds. After building the Docker image, the docker push command spends another minute.

01:13:23 Step 8/8 : CMD ["bin/bundle", "exec", "puma"]
01:13:23  ---> Running in 104967ad1553
01:13:31  ---> 35d2259cdb1d
01:13:34 Removing intermediate container 104967ad1553
01:13:34 [0mSuccessfully built 35d2259cdb1d
01:13:35 [docker-builder] $ docker inspect 35d2259cdb1d
01:13:35 [docker-builder] $ docker push bigbinary/xyz:production-role-management-feature-1529436929
01:13:35 The push refers to a repository [docker.io/bigbinary/xyz]
01:14:21 d67854546d53: Pushed
01:14:22 production-role-management-feature-1529436929: digest: sha256:07f86cfd58fac412a38908d7a7b7d0773c6a2980092df416502d7a5c051910b3 size: 4106
01:14:22 Finished: SUCCESS

So, we found the exact commands which were causing the docker build command to take so much time to build a Docker image.

Let's summarize the steps involved in building our Docker image and the average time each needed to finish.

Often, people build Docker images from a single Git branch, like master. Since changes in a single branch are incremental and hardly has differences in the Gemfile.lock file across commits, bundler cache need not be managed explicitly. Instead, Docker automatically reuses the previously built layer for the RUN bundle install instruction if the Gemfile.lock file remains unchanged.

In our case, this does not happen. For every new feature or a bug fix, we create a separate Git branch. To verify the changes on a particular branch, we deploy a separate review app which serves the code from that branch. To achieve this workflow, everyday we need to build a lot of Docker images containing source code from varying Git branches as well as with varying environments. Most of the times, the Gemfile.lock and assets have different versions across these Git branches. Therefore, it is hard for Docker to cache layers for bundle install and rake assets:precompile tasks and reuse those layers during every docker build command run with different application source code and a different environment. This is why the previously built Docker layer for the RUN bin/bundle install instruction and the RUN bin/rake assets:precompile instruction was often not being used in our case. This reason was causing the RUN instructions to be executed without reusing the previously built Docker layer cache while performing every other Docker build.

Before discussing the approaches to speed up our Docker build flow, let's familiarize with the bundle install and rake assets:precompile tasks and how to speed up them by reusing cache.

Speeding up "bundle install" by using cache

By default, Bundler installs gems at the location which is set by Rubygems. Also, Bundler looks up for the installed gems at the same location.

This location can be explicitly changed by using --path option.

If Gemfile.lock does not exist or no gem is found at the explicitly provided location or at the default gem path then bundle install command fetches all remote sources, resolves dependencies if needed and installs required gems as per Gemfile.

The bundle install --path=vendor/cache command would install the gems at the vendor/cache location in the current directory. If the same command is run without making any change in Gemfile, since the gems were already installed and cached in vendor/cache, the command will finish instantly because Bundler need not to fetch any new gems.

The tree structure of vendor/cache directory looks like this.

vendor/cache
├── aasm-4.12.3.gem
├── actioncable-5.1.4.gem
├── activerecord-5.1.4.gem
├── [...]
├── ruby
│   └── 2.4.0
│       ├── bin
│       │   ├── aws.rb
│       │   ├── dotenv
│       │   ├── erubis
│       │   ├── [...]
│       ├── build_info
│       │   └── nokogiri-1.8.1.info
│       ├── bundler
│       │   └── gems
│       │       ├── activeadmin-043ba0c93408
│       │       [...]
│       ├── cache
│       │   ├── aasm-4.12.3.gem
│       │   ├── actioncable-5.1.4.gem
│       │   ├── [...]
│       │   ├── bundler
│       │   │   └── git
│       └── specifications
│           ├── aasm-4.12.3.gemspec
│           ├── actioncable-5.1.4.gemspec
│           ├── activerecord-5.1.4.gemspec
│           ├── [...]
│           [...]
[...]

It appears that Bundler keeps two separate copies of the .gem files at two different locations, vendor/cache and vendor/cache/ruby/VERSION_HERE/cache.

Therefore, even if we remove a gem in the Gemfile, then that gem will be removed only from the vendor/cache directory. The vendor/cache/ruby/VERSION_HERE/cache will still have the cached .gem file for that removed gem.

Let's see an example.

We have 'aws-sdk', '2.11.88' gem in our Gemfile and that gem is installed.

$ ls vendor/cache/aws-sdk-*
vendor/cache/aws-sdk-2.11.88.gem
vendor/cache/aws-sdk-core-2.11.88.gem
vendor/cache/aws-sdk-resources-2.11.88.gem

$ ls vendor/cache/ruby/2.4.0/cache/aws-sdk-*
vendor/cache/ruby/2.4.0/cache/aws-sdk-2.11.88.gem
vendor/cache/ruby/2.4.0/cache/aws-sdk-core-2.11.88.gem
vendor/cache/ruby/2.4.0/cache/aws-sdk-resources-2.11.88.gem

Now, we will remove the aws-sdk gem from Gemfile and run bundle install.

$ bundle install --path=vendor/cache
Using rake 12.3.0
Using aasm 4.12.3
[...]
Updating files in vendor/cache
Removing outdated .gem files from vendor/cache
  * aws-sdk-2.11.88.gem
  * jmespath-1.3.1.gem
  * aws-sdk-resources-2.11.88.gem
  * aws-sdk-core-2.11.88.gem
  * aws-sigv4-1.0.2.gem
Bundled gems are installed into `./vendor/cache`

$ ls vendor/cache/aws-sdk-*
no matches found: vendor/cache/aws-sdk-*

$ ls vendor/cache/ruby/2.4.0/cache/aws-sdk-*
vendor/cache/ruby/2.4.0/cache/aws-sdk-2.11.88.gem
vendor/cache/ruby/2.4.0/cache/aws-sdk-core-2.11.88.gem
vendor/cache/ruby/2.4.0/cache/aws-sdk-resources-2.11.88.gem

We can see that the cached version of gem(s) remained unaffected.

If we add the same gem 'aws-sdk', '2.11.88' back to the Gemfile and perform bundle install, instead of fetching that gem from remote Gem repository, Bundler will install that gem from the cache.

$ bundle install --path=vendor/cache
Resolving dependencies........
[...]
Using aws-sdk 2.11.88
[...]
Updating files in vendor/cache
  * aws-sigv4-1.0.3.gem
  * jmespath-1.4.0.gem
  * aws-sdk-core-2.11.88.gem
  * aws-sdk-resources-2.11.88.gem
  * aws-sdk-2.11.88.gem

$ ls vendor/cache/aws-sdk-*
vendor/cache/aws-sdk-2.11.88.gem
vendor/cache/aws-sdk-core-2.11.88.gem
vendor/cache/aws-sdk-resources-2.11.88.gem

What we understand from this is that if we can reuse the explicitly provided vendor/cache directory every time we need to execute bundle install command, then the command will be much faster because Bundler will use gems from local cache instead of fetching from the Internet.

Speeding up "rake assets:precompile" task by using cache

JavaScript code written in TypeScript, Elm, JSX etc cannot be directly served to the browser. Almost all web browsers understands JavaScript (ES4), CSS and image files. Therefore, we need to transpile, compile or convert the source asset into the formats which browsers can understand. In Rails, Sprockets is the most widely used library for managing and compiling assets.

In development environment, Sprockets compiles assets on-the-fly as and when needed using Sprockets::Server. In production environment, recommended approach is to pre-compile assets in a directory on disk and serve it using a web server like Nginx.

Precompilation is a multi-step process for converting a source asset file into a static and optimized form using components such as processors, transformers, compressors, directives, environments, a manifest and pipelines with the help of various gems such as sass-rails, execjs, etc. The assets need to be precompiled in production so that Sprockets need not resolve inter-dependencies between required source dependencies every time a static asset is requested. To understand how Sprockets work in great detail, please read this guide.

When we compile source assets using rake assets:precompile task, we can find the compiled assets in public/assets directory inside our Rails application.

$ ls public/assets
manifest-15adda275d6505e4010b95819cf61eb3.json
icons-6250335393ad03df1c67eafe138ab488.eot
icons-6250335393ad03df1c67eafe138ab488.eot.gz
cons-b341bf083c32f9e244d0dea28a763a63.svg
cons-b341bf083c32f9e244d0dea28a763a63.svg.gz
application-8988c56131fcecaf914b22f54359bf20.js
application-8988c56131fcecaf914b22f54359bf20.js.gz
xlsx.full.min-feaaf61b9d67aea9f122309f4e78d5a5.js
xlsx.full.min-feaaf61b9d67aea9f122309f4e78d5a5.js.gz
application-adc697aed7731c864bafaa3319a075b1.css
application-adc697aed7731c864bafaa3319a075b1.css.gz
FontAwesome-42b44fdc9088cae450b47f15fc34c801.otf
FontAwesome-42b44fdc9088cae450b47f15fc34c801.otf.gz
[...]

We can see that the each source asset has been compiled and minified along with its gunzipped version.

Note that the assets have a unique and random digest or fingerprint in their file names. A digest is a hash calculated by Sprockets from the contents of an asset file. If the contents of an asset is changed, then that asset's digest also changes. The digest is mainly used for busting cache so a new version of the same asset can be generated if the source file is modified or the configured cache period is expired.

The rake assets:precompile task also generates a manifest file along with the precompiled assets. This manifest is used by Sprockets to perform fast lookups without having to actually compile our assets code.

An example manifest file, in our case public/assets/manifest-15adda275d6505e4010b95819cf61eb3.json looks like this.

{
  "files": {
    "application-8988c56131fcecaf914b22f54359bf20.js": {
      "logical_path": "application.js",
      "mtime": "2018-07-06T07:32:27+00:00",
      "size": 3797752,
      "digest": "8988c56131fcecaf914b22f54359bf20"
    },
    "xlsx.full.min-feaaf61b9d67aea9f122309f4e78d5a5.js": {
      "logical_path": "xlsx.full.min.js",
      "mtime": "2018-07-05T22:06:17+00:00",
      "size": 883635,
      "digest": "feaaf61b9d67aea9f122309f4e78d5a5"
    },
    "application-adc697aed7731c864bafaa3319a075b1.css": {
      "logical_path": "application.css",
      "mtime": "2018-07-06T07:33:12+00:00",
      "size": 242611,
      "digest": "adc697aed7731c864bafaa3319a075b1"
    },
    "FontAwesome-42b44fdc9088cae450b47f15fc34c801.otf": {
      "logical_path": "FontAwesome.otf",
      "mtime": "2018-06-20T06:51:49+00:00",
      "size": 134808,
      "digest": "42b44fdc9088cae450b47f15fc34c801"
    },
    [...]
  },
  "assets": {
    "icons.eot": "icons-6250335393ad03df1c67eafe138ab488.eot",
    "icons.svg": "icons-b341bf083c32f9e244d0dea28a763a63.svg",
    "application.js": "application-8988c56131fcecaf914b22f54359bf20.js",
    "xlsx.full.min.js": "xlsx.full.min-feaaf61b9d67aea9f122309f4e78d5a5.js",
    "application.css": "application-adc697aed7731c864bafaa3319a075b1.css",
    "FontAwesome.otf": "FontAwesome-42b44fdc9088cae450b47f15fc34c801.otf",
    [...]
  }
}

Using this manifest file, Sprockets can quickly find a fingerprinted file name using that file's logical file name and vice versa.

Also, Sprockets generates cache in binary format at tmp/cache/assets in the Rails application's folder for the specified Rails environment. Following is an example tree structure of the tmp/cache/assets directory automatically generated after executing RAILS_ENV=environment_here rake assets:precompile command for each Rails environment.

$ cd tmp/cache/assets && tree
.
├── demo
│   ├── sass
│   │   ├── 7de35a15a8ab2f7e131a9a9b42f922a69327805d
│   │   │   ├── application.css.sassc
│   │   │   └── bootstrap.css.sassc
│   │   ├── [...]
│   └── sprockets
│       ├── 002a592d665d92efe998c44adc041bd3
│       ├── 7dd8829031d3067dcf26ffc05abd2bd5
│       └── [...]
├── production
│   ├── sass
│   │   ├── 80d56752e13dda1267c19f4685546798718ad433
│   │   │   ├── application.css.sassc
│   │   │   └── bootstrap.css.sassc
│   │   ├── [...]
│   └── sprockets
│       ├── 143f5a036c623fa60d73a44d8e5b31e7
│       ├── 31ae46e77932002ed3879baa6e195507
│       └── [...]
└── staging
    ├── sass
    │   ├── 2101b41985597d41f1e52b280a62cd0786f2ee51
    │   │   ├── application.css.sassc
    │   │   └── bootstrap.css.sassc
    │   ├── [...]
    └── sprockets
        ├── 2c154d4604d873c6b7a95db6a7d5787a
        ├── 3ae685d6f922c0e3acea4bbfde7e7466
        └── [...]

Let's inspect the contents of an example cached file. Since the cached file is in binary form, we can forcefully see the non-visible control characters as well as the binary content in text form using cat -v command.

$ cat -v tmp/cache/assets/staging/sprockets/2c154d4604d873c6b7a95db6a7d5787a

^D^H{^QI"
class^F:^FETI"^SProcessedAsset^F;^@FI"^Qlogical_path^F;^@TI"^]components/Comparator.js^F;^@TI"^Mpathname^F;^@TI"T$root/app/assets/javascripts/components/Comparator.jsx^F;^@FI"^Qcontent_type^F;^@TI"^[application/javascript^F;^@TI"
mtime^F;^@Tl+^GM-gM-z;[I"^Klength^F;^@Ti^BM-L^BI"^Kdigest^F;^@TI"%18138d01fe4c61bbbfeac6d856648ec9^F;^@FI"^Ksource^F;^@TI"^BM-L^Bvar Comparator = function (props) {
  var comparatorOptions = [React.createElement("option", { key: "?", value: "?" })];
  var allComparators = props.metaData.comparators;
  var fieldDataType = props.fieldDataType;
  var allowedComparators = allComparators[fieldDataType] || allComparators.integer;
  return React.createElement(
    "select",
    {
      id: "comparator-" + props.id,
      disabled: props.disabled,
      onChange: props.handleComparatorChange,
      value: props.comparatorValue },
    comparatorOptions.concat(allowedComparators.map(function (comparator, id) {
      return React.createElement(
        "option",
        { key: id, value: comparator },
        comparator
      );
    }))
  );
};^F;^@TI"^Vdependency_digest^F;^@TI"%d6c86298311aa7996dd6b5389f45949f^F;^@FI"^Srequired_paths^F;^@T[^FI"T$root/app/assets/javascripts/components/Comparator.jsx^F;^@FI"^Udependency_paths^F;^@T[^F{^HI"   path^F;^@TI"T$root/app/assets/javascripts/components/Comparator.jsx^F;^@F@^NI"^^2018-07-03T22:38:31+00:00^F;^@T@^QI"%51ab9ceec309501fc13051c173b0324f^F;^@FI"^M_version^F;^@TI"%30fd133466109a42c8cede9d119c3992^F;^@F

We can see that there are some weird looking characters in the above file because it is not a regular file to be read by humans. Also, it seems to be holding some important information such as mime-type, original source code's path, compiled source, digest, paths and digests of required dependencies, etc. Above compiled cache appears to be of the original source file located at app/assets/javascripts/components/Comparator.jsx having actual contents in JSX and ES6 syntax as shown below.

const Comparator = props => {
  const comparatorOptions = [<option key="?" value="?" />];
  const allComparators = props.metaData.comparators;
  const fieldDataType = props.fieldDataType;
  const allowedComparators =
    allComparators[fieldDataType] || allComparators.integer;
  return (
    <select
      id={`comparator-${props.id}`}
      disabled={props.disabled}
      onChange={props.handleComparatorChange}
      value={props.comparatorValue}
    >
      {comparatorOptions.concat(
        allowedComparators.map((comparator, id) => (
          <option key={id} value={comparator}>
            {comparator}
          </option>
        ))
      )}
    </select>
  );
};

If similar cache exists for a Rails environment under tmp/cache/assets and if no source asset file is modified then re-running the rake assets:precompile task for the same environment will finish quickly. This is because Sprockets will reuse the cache and therefore will need not to resolve the inter-assets dependencies, perform conversion, etc.

Even if certain source assets are modified, Sprockets will rebuild the cache and re-generate compiled and fingerprinted assets just for the modified source assets.

Therefore, now we can understand that that if we can reuse the directories tmp/cache/assets and public/assets every time we need to execute rake assets:precompile task, then the Sprockets will perform precompilation much faster.

Speeding up "docker build" -- first attempt

As discussed above, we were now familiar about how to speed up the bundle install and rake assets:precompile commands individually.

We decided to use this knowledge to speed up our slow docker build command. Our initial thought was to mount a directory on the host Jenkins machine into the filesystem of the image being built by the docker build command. This mounted directory then can be used as a cache directory to persist the cache files of both bundle install and rake assets:precompile commands run as part of docker build command in each Jenkins build. Then every new build could reuse the previous build's cache and therefore could finish faster.

Unfortunately, this wasn't possible due to no support from Docker yet. Unlike the docker run command, we cannot mount a host directory into docker build command. A feature request for providing a shared host machine directory path option to the docker build command is still open here.

To reuse cache and perform faster, we need to carry the cache files of both bundle install and rake assets:precompile commands between each docker build (therefore, Jenkins build). We were looking for some place which can be treated as a shared cache location and can be accessed during each build.

We decided to use Amazon's S3 service to solve this problem.

To upload and download files from S3, we needed to inject credentials for S3 into the build context provided to the docker build command.

Alternatively, these S3 credentials can be provided to the docker build command using --build-arg option as discussed earlier.

We used s3cmd command-line utility to interact with the S3 service.

Following shell script named as install_gems_and_precompile_assets.sh was configured to be executed using a RUN instruction while running the docker build command.

set -ex

# Step 1.
if [ -e s3cfg ]; then mv s3cfg ~/.s3cfg; fi

bundler_cache_path="vendor/cache"
assets_cache_path="tmp/assets/cache"
precompiled_assets_path="public/assets"
cache_archive_name="cache.tar.gz"
s3_bucket_path="s3://docker-builder-bundler-and-assets-cache"
s3_cache_archive_path="$s3_bucket_path/$cache_archive_name"

# Step 2.
# Fetch tarball archive containing cache and extract it.
# The "tar" command extracts the archive into "vendor/cache",
# "tmp/assets/cache" and "public/assets".
if s3cmd get $s3_cache_archive_path; then
  tar -xzf $cache_archive_name && rm -f $cache_archive_name
fi

# Step 3.
# Install gems from "vendor/cache" and pack up them.
bin/bundle install --without development test --path $bundler_cache_path
bin/bundle pack --quiet

# Step 4.
# Precompile assets.
# Note that the "RAILS_ENV" is already defined in Dockerfile
# and will be used implicitly.
bin/rake assets:precompile

# Step 5.
# Compress "vendor/cache", "tmp/assets/cache"
# and "public/assets" directories into a tarball archive.
tar -zcf $cache_archive_name $bundler_cache_path \
                             $assets_cache_path  \
                             $precompiled_assets_path

# Step 6.
# Push the compressed archive containing updated cache to S3.
s3cmd put $cache_archive_name $s3_cache_archive_path || true

# Step 7.
rm -f $cache_archive_name ~/.s3cfg

Let's discuss the various steps annotated in the above script.

1. The S3 credentials file injected by Jenkins into the build context needs to be placed at ~/.s3cfg location, so we move that credentials file accordingly.
2. Try to fetch the compressed tarball archive comprising directories such as vendor/cache, tmp/assets/cache and public/assets. If exists, extract the tarball archive at respective paths and remove that tarball.
3. Execute the bundle install command which would reuse the extracted cache from vendor/cache.
4. Execute the rake assets:precompile command which would reuse the extracted cache from tmp/assets/cache and public/assets.
5. Compress the cache directories vendor/cache, tmp/assets/cache and public/assets in a tarball archive.
6. Upload the compressed tarball archive containing updated cache directories to S3.
7. Remove the compressed tarball archive and the S3 credentials file.

Please note that, in our actual case we had generated different tarball archives depending upon the provided RAILS_ENV environment. For demonstration, here we use just a single archive instead.

The Dockerfile needed to update to execute the install_gems_and_precompile_assets.sh script.

FROM bigbinary/xyz-base:latest

ENV APP_PATH /data/app/

WORKDIR $APP_PATH

ADD . $APP_PATH

ARG RAILS_ENV

RUN install_gems_and_precompile_assets.sh

CMD ["bin/bundle", "exec", "puma"]

With this setup, average time of the Jenkins builds was now reduced to about 5 minutes. This was a great achievement for us.

We reviewed this approach in a great detail. We found that although the approach was working fine, there was a major security flaw. It is not at all recommended to inject confidential information such as login credentials, private keys, etc. as part of the build context or using build arguments while building a Docker image using docker build command. And we were actually injecting S3 credentials into the Docker image. Such confidential credentials provided while building a Docker image can be inspected using docker history command by anyone who has access to that Docker image.

Due to above reason, we needed to abandon this approach and look for another.

Speeding up "docker build" -- second attempt

In our second attempt, we decided to execute bundle install and rake assets:precompile commands outside the docker build command. Outside meaning the place to execute these commands was Jenkins build itself. So with the new approach, we had to first execute bundle install and rake assets:precompile commands as part of the Jenkins build and then execute docker build as usual. With this approach, we could now avail the inter-build caching benefits provided by Jenkins.

The prerequisite was to have all the necessary system packages installed on the Jenkins machine required by the gems enlisted in the application's Gemfile. We installed all the necessary system packages on our Jenkins server.

Following screenshot highlights the things that we needed to configure in our Jenkins job to make this approach work.

1. Running the Jenkins build in RVM managed environment with the specified Ruby version

Sometimes, we need to use different Ruby version as specified in the .ruby-version in the cloned source code of the application. By default, the bundle install command would install the gems for the system Ruby version available on the Jenkins machine. This was not acceptable for us. Therefore, we needed a way to execute the bundle install command in Jenkins build in an isolated environment which could use the Ruby version specified in the .ruby-version file instead of the default system Ruby version. To address this, we used RVM plugin for Jenkins. The RVM plugin enabled us to run the Jenkins build in an isolated environment by using or installing the Ruby version specified in the .ruby-version file. The section highlighted with green color in the above screenshot shows the configuration required to enable this plugin.

2. Carrying cache files between Jenkins builds required to speed up "bundle install" and "rake assets:precompile" commands

We used Job Cacher Jenkins plugin to persist and carry the cache directories such as vendor/cache, tmp/cache/assets and public/assets between builds. At the beginning of a Jenkins build just after cloning the source code of the application, the Job Cacher plugin restores the previously cached version of these directories into the current build. Similarly, before finishing a Jenkins build, the Job Cacher plugin copies the current version of these directories at /var/lib/jenkins/jobs/docker-builder/cache on the Jenkins machine which is outside the workspace directory of the Jenkins job. The section highlighted with red color in the above screenshot shows the necessary configuration required to enable this plugin.

3. Executing the "bundle install" and "rake assets:precompile" commands before "docker build" command

Using the "Execute shell" build step provided by Jenkins, we execute bundle install and rake assets:precompile commands just before the docker build command invoked by the CloudBees Docker Build and Publish plugin. Since the Job Cacher plugin already restores the version of vendor/cache, tmp/cache/assets and public/assets directories from the previous build into the current build, the bundle install and rake assets:precompile commands reuses the cache and performs faster.

The updated Dockerfile has lesser number of instructions now.

FROM bigbinary/xyz-base:latest

ENV APP_PATH /data/app/

WORKDIR $APP_PATH

ADD . $APP_PATH

CMD ["bin/bundle", "exec", "puma"]

With this approach, average Jenkins build time is now between 3.5 to 4.5 minutes.

Following graph shows the build time trend of some of the recent builds on our Jenkins server.

Please note that the spikes in the above graphs shows that certain Jenkins builds took more than 5 minutes sometimes due to concurrently running builds at that time. Because our Jenkins server has a limited set of resources, concurrently running builds often run longer than estimated.

We are still looking to improve the containerization speed even more and still maintaining the image size small. Please let us know if there's anything else we can do to improve the containerization process.

Note that that our Jenkins server runs on the Ubuntu OS which is based on Debian. Our base Docker image is also based on Debian. Some of the gems in our Gemfile are native extensions written in C. The pre-installed gems on Jenkins machine have been working without any issues while running inside the Docker containers on Kubernetes. It may not work if both of the platforms are different since native extension gems installed on Jenkins host may fail to work inside the Docker container.

In this article we will discuss how to setup path based routing for a Ruby on Rails application in kubernetes using HAProxy ingress.

This post assumes that you have basic understanding of Kubernetes terms like pods, deployments, services, configmap and ingress.

Typically our Rails app has services like unicorn/puma, sidekiq/delayed-job/resque, Websockets and some dedicated API services. We had one web service exposed to the world using load balancer and it was working well. But as the traffic increased it became necessary to route traffic based on URLs/path.

However Kubernetes does not supports this type of load balancing out of the box. There is work in progress for alb-ingress-controller to support this but we could not rely on it for production usage as it is still in alpha.

The best way to achieve path based routing was to use ingress controller.

We researched and found that there are different types of ingress available in k8s world.

1. nginx-ingress
2. ingress-gce
3. HAProxy-ingress
4. traefik
5. voyager

We experimented with nginx-ingress and HAProxy and decided to go with HAProxy. HAProxy has better support for Rails websockets which we needed in the project.

We will walk you through step by step on how to use haproxy ingress in a Rails app.

Configuring Rails app with HAProxy ingress controller

Here is what we are going to do.

• Create a Rails app with different services and deployments.
• Create tls secret for SSL.
• Create HAProxy ingress configmap.
• Create HAProxy ingress controller.
• Expose ingress with service type LoadBalancer
• Setup app DNS with ingress service.
• Create different ingress rules specifying path based routing.
• Test the path based routing.

Now let's build Rails application deployment manifest for services like web(unicorn),background(sidekiq), Websocket(ruby thin),API(dedicated unicorn).

Here is our web app deployment and service template.

---
apiVersion: v1
kind: Deployment
metadata:
  name: test-production-web
  labels:
    app: test-production-web
  namespace: test
spec:
  template:
    metadata:
      labels:
        app: test-production-web
    spec:
      containers:
      - image: <your-repo>/<your-image-name>:latest
        name: test-production
        imagePullPolicy: Always
       env:
        - name: POSTGRES_HOST
          value: test-production-postgres
        - name: REDIS_HOST
          value: test-production-redis
        - name: APP_ENV
          value: production
        - name: APP_TYPE
          value: web
        - name: CLIENT
          value: test
        ports:
        - containerPort: 80
      imagePullSecrets:
        - name: registrykey
---
apiVersion: v1
kind: Service
metadata:
  name: test-production-web
  labels:
    app: test-production-web
  namespace: test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: test-production-web

Here is background app deployment and service template.

---
apiVersion: v1
kind: Deployment
metadata:
  name: test-production-background
  labels:
    app: test-production-background
  namespace: test
spec:
  template:
    metadata:
      labels:
        app: test-production-background
    spec:
      containers:
      - image: <your-repo>/<your-image-name>:latest
        name: test-production
        imagePullPolicy: Always
       env:
        - name: POSTGRES_HOST
          value: test-production-postgres
        - name: REDIS_HOST
          value: test-production-redis
        - name: APP_ENV
          value: production
        - name: APP_TYPE
          value: background
        - name: CLIENT
          value: test
        ports:
        - containerPort: 80
      imagePullSecrets:
        - name: registrykey
---
apiVersion: v1
kind: Service
metadata:
  name: test-production-background
  labels:
    app: test-production-background
  namespace: test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: test-production-background

Here is websocket app deployment and service template.

---
apiVersion: v1
kind: Deployment
metadata:
  name: test-production-websocket
  labels:
    app: test-production-websocket
  namespace: test
spec:
  template:
    metadata:
      labels:
        app: test-production-websocket
    spec:
      containers:
      - image: <your-repo>/<your-image-name>:latest
        name: test-production
        imagePullPolicy: Always
       env:
        - name: POSTGRES_HOST
          value: test-production-postgres
        - name: REDIS_HOST
          value: test-production-redis
        - name: APP_ENV
          value: production
        - name: APP_TYPE
          value: websocket
        - name: CLIENT
          value: test
        ports:
        - containerPort: 80
      imagePullSecrets:
        - name: registrykey
---
apiVersion: v1
kind: Service
metadata:
  name: test-production-websocket
  labels:
    app: test-production-websocket
  namespace: test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: test-production-websocket

Here is API app deployment and service info.

---
apiVersion: v1
kind: Deployment
metadata:
  name: test-production-api
  labels:
    app: test-production-api
  namespace: test
spec:
  template:
    metadata:
      labels:
        app: test-production-api
    spec:
      containers:
      - image: <your-repo>/<your-image-name>:latest
        name: test-production
        imagePullPolicy: Always
       env:
        - name: POSTGRES_HOST
          value: test-production-postgres
        - name: REDIS_HOST
          value: test-production-redis
        - name: APP_ENV
          value: production
        - name: APP_TYPE
          value: api
        - name: CLIENT
          value: test
        ports:
        - containerPort: 80
      imagePullSecrets:
        - name: registrykey
---
apiVersion: v1
kind: Service
metadata:
  name: test-production-api
  labels:
    app: test-production-api
  namespace: test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: test-production-api

Let's launch this manifest using kubectl apply.

$ kubectl apply -f test-web.yml -f test-background.yml -f test-websocket.yml -f test-api.yml
deployment "test-production-web" created
service "test-production-web" created
deployment "test-production-background" created
service "test-production-background" created
deployment "test-production-websocket" created
service "test-production-websocket" created
deployment "test-production-api" created
service "test-production-api" created

Once our app is deployed and running we should create HAProxy ingress. Before that let's create a tls secret with our SSL key and certificate.

This is also used to enable HTTPS for app URL and to terminate it on L7.

$ kubectl create secret tls tls-certificate --key server.key --cert server.pem

Here server.key is our SSL key and server.pem is our SSL certificate in pem format.

Now let's Create HAProxy controller resources.

HAProxy configmap

For all the available configuration parameters from HAProxy refer here.

apiVersion: v1
data:
  dynamic-scaling: "true"
  backend-server-slots-increment: "4"
kind: ConfigMap
metadata:
  name: haproxy-configmap
  namespace: test

HAProxy Ingress controller deployment

Deployment template for the Ingress controller with at-least 2 replicas to manage rolling deploys.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: haproxy-ingress
  name: haproxy-ingress
  namespace: test
spec:
  replicas: 2
  selector:
    matchLabels:
      run: haproxy-ingress
  template:
    metadata:
      labels:
        run: haproxy-ingress
    spec:
      containers:
        - name: haproxy-ingress
          image: quay.io/jcmoraisjr/haproxy-ingress:v0.5-beta.1
          args:
            - --default-backend-service=$(POD_NAMESPACE)/test-production-web
            - --default-ssl-certificate=$(POD_NAMESPACE)/tls-certificate
            - --configmap=$(POD_NAMESPACE)/haproxy-configmap
            - --ingress-class=haproxy
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
            - name: stat
              containerPort: 1936
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace

Notable fields in above manifest are arguments passed to controller.

--default-backend-service is the service when No rule is matched your request will be served by this app.

In our case it is test-production-web service, But it can be custom 404 page or whatever better you think.

--default-ssl-certificate is the SSL secret we just created above this will terminate SSL on L7 and our app is served on HTTPS to outside world.

HAProxy Ingress service

This is the LoadBalancer type service to allow client traffic to reach our Ingress Controller.

LoadBalancer has access to both public network and internal Kubernetes network while retaining the L7 routing of the Ingress Controller.

apiVersion: v1
kind: Service
metadata:
  labels:
    run: haproxy-ingress
  name: haproxy-ingress
  namespace: test
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
    - name: https
      port: 443
      protocol: TCP
      targetPort: 443
    - name: stat
      port: 1936
      protocol: TCP
      targetPort: 1936
  selector:
    run: haproxy-ingress

Now let's apply all the manifests of HAProxy.

$ kubectl apply -f haproxy-configmap.yml -f haproxy-deployment.yml -f haproxy-service.yml
configmap "haproxy-configmap" created
deployment "haproxy-ingress" created
service "haproxy-ingress" created

Once all the resources are running get the LoadBalancer endpoint using.

$ kubectl -n test get svc haproxy-ingress -o wide

NAME               TYPE           CLUSTER-IP       EXTERNAL-IP                                                            PORT(S)                                     AGE       SELECTOR
haproxy-ingress   LoadBalancer   100.67.194.186   a694abcdefghi11e8bc3b0af2eb5c5d8-806901662.us-east-1.elb.amazonaws.com   80:31788/TCP,443:32274/TCP,1936:32157/TCP   2m        run=ingress

DNS mapping with application URL

Once we have ELB endpoint of ingress service, map the DNS with URL like test-rails-app.com.

Ingress Implementation

Now after doing all the hard work it is time to configure ingress and path based rules.

In our case we want to have following rules.

https://test-rails-app.com requests to be served by test-production-web.

https://test-rails-app.com/websocket requests to be served by test-production-websocket.

https://test-rails-app.com/api requests to be served by test-production-api.

Let's create a ingress manifest defining all the rules.

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  namespace: test
spec:
  tls:
    - hosts:
        - test-rails-app.com
      secretName: tls-certificate
  rules:
    - host: test-rails-app.com
      http:
        paths:
          - path: /
            backend:
              serviceName: test-production-web
              servicePort: 80
          - path: /api
            backend:
              serviceName: test-production-api
              servicePort: 80
          - path: /websocket
            backend:
              serviceName: test-production-websocket
              servicePort: 80

Moreover there are Ingress Annotations for adjusting configuration changes.

As expected, now our default traffic on / is routed to test-production-web service.

/api is routed to test-production-api service.

/websocket is routed to test-production-websocket service.

Thus ingress implementation solves our purpose of path based routing and terminating SSL on L7 on Kubernetes.

Problem

We want to deploy a Rails application on Kubernetes. We assume that the assets:precompile task would be run as part of the Docker image build process.

We want to run rake tasks such as db:migrate and db:seed on the initial deployment, and just db:migrate task on each later deployment.

We cannot run these tasks while building the Docker image as it would not be able to connect to the database at that moment.

So, how to run these tasks?

Solution

We assume that we have a Docker image named myorg/myapp:v0.0.1 which contains the source code for our Rails application.

We also assume that we have included database.yml manifest in this Docker image with the required configuration needed for connecting to the database.

We need to create a Kubernetes deployment template with the following content.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
        - image: myorg/myapp:v0.0.1
          name: myapp
          imagePullPolicy: IfNotPresent
          env:
            - name: DB_NAME
              value: myapp
            - name: DB_USERNAME
              value: username
            - name: DB_PASSWORD
              value: password
            - name: DB_HOST
              value: 54.10.10.245
          ports:
            - containerPort: 80
      imagePullSecrets:
        - name: docker_pull_secret

Let's save this template file as myapp-deployment.yml.

We can change the options and environment variables in above template as per our need. The environment variables specified here will be available to our Rails application.

To apply above template for the first time on Kubernetes, we will use the following command.

$ kubectl create -f myapp-deployment.yml

Later on, to apply the same template after modifications such as change in the Docker image name or change in the environment variables, we will use the following command.

$ kubectl apply -f myapp-deployment.yml

After applying the deployment template, it will create a pod for our application on Kubernetes.

To see the pods, we use the following command.

$ kubectl get pods

Let's say that our app is now running in the pod named myapp-4007005961-1st7s.

To execute a rake task, for e.g. db:migrate on this pod, we can run the following command.

$ kubectl exec myapp-4007005961-1st7s                              \
          -- bash -c                                               \
          'cd ~/myapp && RAILS_ENV=production bin/rake db:migrate'

Similarly, we can execute db:seed rake task as well.

If we already have an automated flow for deployments on Kubernetes, we can make use of this approach to programmatically or conditionally run any rake task as per the needs.

Why not to use Kubernetes Jobs to solve this?

We faced some issues while using Kubernetes Jobs to run migration and seed rake tasks.

1. If the rake task returns a non-zero exit code, the Kubernetes job keeps spawning pods until the task command returns a zero exit code.

2. To get around the issue mentioned above we needed to unnecessarily implement additional custom logic of checking job status and the status of all the spawned pods.

3. Capturing the command's STDOUT or STDERR was difficult using Kubernetes job.

4. Some housekeeping was needed such as manually terminating the job if it wasn't successful. If not done, it will fail to create a Kubernetes job with the same name, which is bound to occur when we perform later deployments.

Because of these issues, we choose not to rely on Kubernetes jobs to solve this problem.