Kubernetes hasa couple of different waysto scale deployments automatically based on the load the application receives.TheHorizontal Pod Autoscaler (HPA)can be used out of the box in a Kubernetes cluster to increase or decrease thenumber of Pods of your deployment. By default, HPA supports scaling based on CPUand memory usage, served by themetrics server.

While building NeetoDeploy initially, we'd setup to scale deployments based on CPU and memory usage, since these were thedefault metrics supported by the HPA. However, later we wanted to scaledeployments based on the average response time of our application.

This is an example of a case where the metric we want to scale is not directlyrelated to the CPU or the memory usage. Other examples of this could be networkmetrics from the load balancer, like the number of requests received in theapplication. In this blog, we will discuss how we achieved autoscaling ofdeployments in Kubernetes based on the average response time usingprometheus-adapter.

When an application receives a lot of requests suddenly, this creates a spike inthe average response time. The CPU and memory metrics also spike, but they takelonger to catch up. In such cases, being able to scale deployments based on theresponse time will ensure that the spike in traffic is handled gracefully.

Prometheus is one of the most popular cloud nativemonitoring tools and the Kubernetes HPA can be extended to scale deploymentsbased on metrics exposed by Prometheus. We used the prometheus-adapter tobuild autoscaling based on the average response time inNeetoDeploy.

Setting up the custom metrics

We took the following steps to make our HPAs work with Prometheus metrics.

1. Installed prometheus-adapter in our cluster.
2. Configured the metric we wanted for our HPAs as a custom metric in theprometheus-adapter.
3. Confirmed that the metric is added to the custom.metrics.k8s.io APIendpoint.
4. Configured an HPA with the custom metric.

Install prometheus-adapter in the cluster

prometheus-adapter isan implementation of the custom.metrics.k8s.io API using Prometheus. We usedthe prometheus-adapter to set up Kubernetes metrics APIs for our Promtheusmetrics, which then can be used with our HPAs.

We installed prometheus-adapter in our cluster using Helm.We got a template for the values file for the Helm installationhere.

We made a few changes to the file before we applied it to our cluster anddeployed prometheus-adapter:

1. We made sure that the Prometheus deployment is configured properly by givingthe correct service url and port.

# values.yamlprometheus:  # Value is templated  url: http://prometheus.monitoring.svc.cluster.local  port: 9090  path: ""# ... rest of the file

2. We made sure that the custom metrics that we needed for our HPA areconfigured under rules.custom in the values.yaml file. In the followingexample, we are using the custom metric traefik_service_avg_response_timesince we'll be using that to calculate the average response time for eachdeployment.

# values.yamlrules:  default: false  custom:    - seriesQuery:'{__name__=~"traefik_service_avg_response_time", service!=""}'      resources:        overrides:          app_name:            resource: service          namespace:            resource: namespace      metricsQuery: traefik_service_avg_response_time{<<.LabelMatchers>>}

Once we configured our values.yaml file properly, we installedprometheus-adapter in our cluster with Helm.

helm repo add prometheus https://prometheus-community.github.io/helm-chartshelm repo updatehelm install prom-adapter prometheus-community/prometheus-adapter --values values.yaml

Query for custom metric

Once we got prometheus-adapter running, we queried our cluster to check if thecustom metric is coming up in the custom.metrics.k8s.io API endpoint.

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq

The response looked like this:

{  "kind": "APIResourceList",  "apiVersion": "v1",  "groupVersion": "custom.metrics.k8s.io/v1beta1",  "resources": [    {      "name": "services/traefik_service_avg_response_time",      "singularName": "",      "namespaced": true,      "kind": "MetricValueList",      "verbs": ["get"]    },    {      "name": "namespaces/traefik_service_avg_response_time",      "singularName": "",      "namespaced": false,      "kind": "MetricValueList",      "verbs": ["get"]    }  ]}

We also queried the metric API for a particular service we've configured themetric for. Here, we're querying the traefik_service_avg_response_time metricfor the neeto-chat-web-staging app in the default namespace.

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/neeto-chat-web-staging/traefik_service_avg_response_time | jq

The API response gave the following.

{  "kind": "MetricValueList",  "apiVersion": "custom.metrics.k8s.io/v1beta1",  "metadata": {},  "items": [    {      "describedObject": {        "kind": "Service",        "namespace": "default",        "name": "neeto-chat-web-staging",        "apiVersion": "/v1"      },      "metricName": "traefik_service_avg_response_time",      "timestamp": "2024-02-26T19:31:33Z",      "value": "19m",      "selector": null    }  ]}

From the response, we can see that the average response time at the instant isreported as 19ms.

Create the HPA

Now that we're sure that prometheus-adapter is able to serve custom metricsunder the custom.metrics.k8s.io API, we wired this up with a Horizontal PodAutoscaler to scale our deployments based on our custom metric.

apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata:  name: my-app-name-hpaspec:  scaleTargetRef:    apiVersion: apps/v1    kind: Deployment    name: my-app-name-deployment  minReplicas: 1  maxReplicas: 10  metrics:    - type: Object      object:        metric:          name: traefik_service_avg_response_time          selector: { matchLabels: { app_name: my-app-name } }        describedObject:          apiVersion: v1          kind: Service          name: my-app-name        target:          type: Value          value: 0.03

With everything set up, the HPA was able to fetch the custom metric scraped byPrometheus and scale our Pods up and down based on the value of the metric. Wealso created a recording rule in Prometheus for storing our custom metricqueries and dropped the unwanted labels as a best practice. We can use thecustom metric stored with the recording rule directly with prometheus-adapterto expose the metrics as an API endpoint in Kubernetes. This is helpful whenyour custom metric queries are complex.

If your application runs on Heroku, you can deploy it on NeetoDeploy without anychange. If you want to give NeetoDeploy a try, then please send us an email atinvite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, followNeetoDeploy on X. You can also join ourcommunity Slack to chat with us about anyNeeto product.

At neeto we are building 20+ applications, and most ofour applications are running in NeetoDeploy. Once we migrated from Heroku toNeetoDeploy, we started getting 520 response code for our applications. Thisissue was occurring randomly and rarely.

What is 520 response code?

A 520 response code happens when the connection is started on the origin webserver, but the request is not completed. This could be due to server crashes orthe inability to handle the incoming requests because of insufficient resources.

When we looked at our logs closely, we found that all the 520 response codesituations occurred when we restarted or deployed the app. From this, weconcluded that the new pods are failing to handle requests from the clientinitially and working fine after some time.

What is wrong with new pods?

Once our investigation narrowed down to the new pods, we quickly realized thatrequests are arriving at the server even when the server is not fully ready yetto take new requests.

When we create a new pod in Kubernetes, it is marked as "Ready", and requestsare sent to it as soon as its containers start. However, the servers initiatedwithin these containers may require additional time to boot up and to becomeready to accept the requests fully.

Let's try restarting the application

$ kubectl rollout restart deployment bling-staging-web

As we can see, a new container is getting created for the new pod. The READYstatus for the new pod is 0. It means it's not yet READY.

NAME                               READY  STATUS             RESTARTS  AGEbling-staging-web-656f74d9d-6kpzz  1/1    Running            0         2m8sbling-staging-web-79fc6f978-cdjf5  0/1    ContainerCreating  0         5s

Now we can see that the new pod is marked as READY (1 out of 1), and the old oneis terminating.

NAME                               READY  STATUS             RESTARTS  AGEbling-staging-web-656f74d9d-6kpzz  0/1    Terminating        0         2m9sbling-staging-web-79fc6f978-cdjf5  1/1    Running            0         6s

The new pod is shown as READY as soon as the container was created. But onchecking the logs, we could see that the server was still starting up and notready yet.

[1] Puma starting in cluster mode...[1] Installing dependencies...

From the above observation, we understood that the pod is marked as "READY"right after the container is created. Consequently, requests are received evenbefore the server is fully prepared to serve them, and they get a 520 responsecode.

Solution

To fix this issue, we must ensure that pods are marked as "Ready" only after theserver is up and ready to accept the requests. We can do this by usingKuberneteshealth probes.More than six years ago we wrotea blogon how we can leverage the readiness and liveness probes of Kubernetes.

Adding Startup probe

Initially, we only addedStartup probesince we had a problem with the boot-up phase. You can read more about theconfiguration settingshere.

The following configuration will add the Startup probe for the deployments:

startupProbe:  failureThreshold: 10  httpGet:    path: /health_check    port: 3000    scheme: HTTP  periodSeconds: 5  successThreshold: 1  timeoutSeconds: 60  initialDelaySeconds: 10

/health_check is a route in the application that is expected to return a 200response code if all is going well. Now, let's restart the application againafter adding the Startup probe.

Container is created for the new pod, but the pod is still not "Ready".

NAME                               READY  STATUS            RESTARTS  AGEbling-staging-web-656f74d9d-6kpzz  1/1    Running           0         2m8sbling-staging-web-79fc6f978-cdjf5  0/1    Running           0         5s

The new pod is marked as "Ready", and the old one is "Terminating".

NAME                               READY  STATUS            RESTARTS  AGEbling-staging-web-656f74d9d-6kpzz  0/1    Terminating       0         2m38sbling-staging-web-79fc6f978-cdjf5  1/1    Running           0         35s

If we check the logs, we can see the health check request:

  [1] Puma starting in cluster mode...  [1] Installing dependencies...  [1] * Puma version: 6.3.1 (ruby 3.2.2-p53) ("Mugi No Toki Itaru")  [1] *  Min threads: 5  [1] *  Max threads: 5  [1] *  Environment: heroku  [1] *   Master PID: 1  [1] *      Workers: 1  [1] *     Restarts: () hot () phased [1] * Listening on http://0.0.0.0:3000 [1] Use Ctrl-C to stop [2024-02-10T02:40:48.944785 #23]  INFO -- : [bb9e756a-51cc-4d6b-9a4a-96b0464f6740] Started GET "/health_check" for 192.168.120.195 at 2024-02-10 02:40:48 +0000 [2024-02-10T02:40:48.946148 #23]  INFO -- : [bb9e756a-51cc-4d6b-9a4a-96b0464f6740] Processing by HealthCheckController#healthy as */* [2024-02-10T02:40:48.949292 #23]  INFO -- : [bb9e756a-51cc-4d6b-9a4a-96b0464f6740] Completed 200 OK in 3ms (Allocations: 691)

Now, the pod is marked as "Ready" only after the health check succeeds, in otherwords, only when the server is prepared to accept the requests.

Fixing the Startup probe for production applications

Once we released the health check for our deployments, we found that healthchecks were failing for all production applications but working for staging andreview applications.

We were getting the following error in our production applications.

Startup probe failed: Get "https://192.168.43.231:3000/health_check": http: server gave HTTP response to HTTPS client2024-02-12 06:40:04 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>

From the above logs, it was clear that the issue was related to SSLconfiguration. On comparing the production environment configuration with theothers, we figured out that we had enabledforce_sslfor production applications. The force_ssl=true setting ensures that allincoming requests are SSL encrypted and will automatically redirect to their SSLcounterparts.

The following diagram broadly shows the path of an incoming request.

From the above diagram, we can infer the following things:

• SSL verification is happening in the ingress controller and not in the server.
• Client requests are going through the ingress controller before reaching theserver.
• Request from ingress controller to the pod is an HTTP request.
• The HTTP health check requests are directly sent from Kubelet to the pod anddo not go through the ingress controller.

Here is how our health check request works.

1. Kubeletsends an HTTP request to the server directly.
2. Since force_ssl is enabled,ActionDispatch::SSLmiddleware redirects the request to HTTPS.
3. When the HTTPS request reaches the server, Puma throwsAre you trying to open an SSL connection to a non-SSL Puma? error since noSSL certificates are configured with the server.

The solution to our problem lies in understanding why only the health checkrequest is rejected, whereas the request from the ingress controller is not,even though both are HTTP requests. This is because ingress controller sets someheaders before forwarding to the pod, and the header we are concerned about isX-FORWARDED-PROTO.The X-Forwarded-Proto header contains the HTTP/HTTPS scheme the client used toaccess the application. When a client makes an HTTPS request, the ingresscontroller terminates the SSL/TLS connection and forwards the request to thebackend service using plain HTTP after adding theX-Forwarded-Proto along withthe other headers.

Everything started working after adding the X-Forwarded-Proto header to ourstartup probe request.

startupProbe:  failureThreshold: 10  httpGet:    httpHeaders:      - name: X-FORWARDED-PROTO        value: https    path: <%= health_check_url %>    port: <%= port %>    scheme: HTTP  periodSeconds: 5  successThreshold: 1  timeoutSeconds: 60  initialDelaySeconds: 10

We also addedReadinessandLivenessprobes for our deployments.

If your application runs on Heroku, you can deploy it on NeetoDeploy without anychange. If you want to give NeetoDeploy a try, then please send us an email atinvite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, followNeetoDeploy on Twitter. You can also join ourSlack community to chat with us about any Neeto product.

What is sleep when idle feature

"Sleep when idle" is a feature of NeetoDeploy, which puts the deployedapplication to sleep when there is no hit to the server for 5 minutes. Thishelps reduce the cost of the server.

"Sleep when idle" feature can be enabled not only for the pull request reviewapplications, but for staging and production applications too. Many folks buildapplications to learn and for hobby. In such cases, there is no point in runningthe server when the server is not likely to get any traffic. Since NeetoDeploybilling is based on the usage "Sleep when idle" feature helps keep the bill lowfor the users.

Let's say you build something and you deployed to production. You shared it withyour friends. For a day or two you got a bit of traffic, and after that youmoved on to other things. If "sleep when idle" is enabled then you don't need toworry about anything. If the server is not getting any traffic then you will notbe billed.

How is Neeto using sleep when idle feature

At neeto, we are building 20+ applications at the sametime. It means lots of pull requests for all these products and thus lots of PRreview apps are created.

For a long time, we were using Heroku to build the review apps. However whenNeetoDeploy started to become stable, we movedto generating PR review apps fromHeroku to NeetoDeploy. This helped reduce cost.

How to make deployments sleep when idle?

This video describes how "sleep when idle" feature is implemented.

<iframewidth="560"height="315"src="https://youtube.com/embed/trn2DJyTjnw"frameborder="0"title="How we designed NeetoDeploy's 'sleep when idle' feature"allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"allowfullscreen

</iframe>

Keeping the apps running only when they're being used involves two steps:

1. Scaling the deployments down and bringing them back up again
2. Figuring out when to do the scaling

The deployments can be scaled easily using the kubectl scale command. Forexample, if we want to turn our deployment off, we can run the following toupdate our deployment to zero replicas, essentially destroying all the pods.

kubectl scale deployment/nginx --replicas=0

We can also delete our service, ingress or any other resource we might havecreated for our deployment. The configuration of the deployment itself wouldstill be present in the cluster even when we make it sleep, since the KubernetesDeployment is not deleted.

When we want to bring our app back up again, we can use the same command to spinup new pods:

kubectl scale deployment/nginx --replicas=1

The challenge was to figure out when to do this. We decided that we'd have athreshold based on the time the app is last accessed by users. If theapplication is not accessed for more than five minutes, we consider theapplication to be idle and we will scale it down. It'll be brought back up whena user tries to access it again.

Exploring existing solutions

There are existing CNCF projects like Knative andKeda, which can potentially be used to achieve what we wanthere. We spent some time exploring these but realized that these solutionsweren't exactly suitable for our requirements. Kubernetes also natively has aHPAScaleToZerofeature gatewhich enables theHorizontal Pod Autoscalerto scale down deployments to zero pods, but this is still in alpha and, hence isnot available in EKS yet.

Ultimately, we decided to write our own service for achieving this. The entirebackend of NeetoDeploy was designed asa collection of microservicesfrom day one. So it made sense to build our pod idling service as anothermicroservice that runs in our cluster.

Figuring out when to make applications sleep

To know when applications can be idled, we need to know when people areaccessing the applications from their browsers. Since all the requests toapplications deployed on NeetoDeploy would go through our load balancer, itwould contain the information of when every app was last accessed.

We use Traefik as our load balancer and we usedTraefik's middlewares toretrieve and process the information of when apps are being accessed. We wrote acustom middleware to send all the request information to the pod idling service,whenever an app is being accessed. The pod idling service would store all theURLs, along with the timestamp at which they were accessed, in a Redis cache.The following graphic shows how the request information would be collected andstored by the pod idling service into its Redis cache, both of which are runningwithin the cluster.

The pod idling service would then filter the apps that were last accessed morethan five minutes ago. It then sends a request to the cluster to scale all theseapps down. We'd also delete any related resources like the Services and theIngressRoutes used to configure networking for the deployments.

We first tested this by running the service manually, and sure enough, all theinactive deployments are filtered and scaled properly. We then added this as acron job in the pod idling service, which would run every five minutes. Thismeans that no app would run for more than five minutes if they're not beingused.

But wait! How would we bring the app back up after scaling it down?

Building the downtime service

As we discussed above, we use Traefik's IngressRoutes to route traffic to theapplication being accessed. We made use of thepriority parameterof IngressRoutes to boot up apps that are sleeping. Essentially, we created awildcard Traefik IngressRoute that points to a "downtime service" deployment,which is a React app that serves a message of There's nothing here, yet to letusers know that the app they're trying to access doesn't exist. You can see thisin action if you visit a random URL in NeetoDeploy, say something likenonexistent-appname.neetodeployapp.com

Wildcard IngressRoutes have the least priority by default. So if we create a"catch-all" wildcard IngressRoute, any invalid url without an IngressRoute ofits own, can be redirected to a single Service in Kubernetes. This is how we'reredirecting non-existent apps to the page shown above. In the following graphic,we can see how a request to a random URL is routed to the downtime service withthe wildcard IngressRoute.

This also means that if an app is scaled down by the pod idling service and getsits IngressRoute deleted, the next time a user tries to access the app, therequest would instead be routed to the downtime service. We need to handle thescale up logic from the downtime service.

Whenever a user requests a URL that doesn't have an IngressRoute, there are twopossibilities.

1. The app doesn't exist.
2. The app exists, but is currently scaled down.

The downtime service would first check the cluster if the requested app ispresent in the cluster in a sleeping state. If not then the user will be servedthe "There's nothing here, yet" page. If there is a sleeping deployment,however, we boot it back up. The downtime service sends the scale up request tothe cluster. We keep redirecting the user back to the url till the app is up andrunning. This redirection would keep happening until the app is scaled up sincewe create the Service and IngressRoute only after the pods of the app arerunning. At this point, the request will be routed to the correct pod by theapp's IngressRoute, since it has a higher priority than the wildcardIngressRoute of the downtime service. All of these steps are illustrated in theGIF below:

This design worked flawlessly and we were able to bring back scaled downapplications with as low as 20-30 seconds of wait time.

Conclusion

We've been running this setup for almost a year now, and it has been workingsmoothly so far. Pod idling service and the downtime service started as simplermicroservices and continue to evolve, adapting to the increasing demand as wegrow.

If your application runs on Heroku, you can deploy it on NeetoDeploy without anychange. If you want to give NeetoDeploy a try, then please send us an email atinvite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, followNeetoDeploy on X. You can also join ourcommunity Slack to chat with us about anyNeeto product.

One of the features that we wanted in our cloud deployment platform,NeetoDeploy was an application metrics. We decided to usePrometheus for building this feature. Prometheus is anopen source monitoring and alerting toolkit and is a CNCF graduated project.Venturing into the Cloud Native ecosystem of projects apart from Kubernetes wassomething we had never done before. We ended up learning a lot about Prometheusand how to use it during the course of building this feature.

Initial setup

We installed Prometheus in our Kubernetes cluster by writing a deploymentconfiguration YAML and applying it to our cluster. We also provisioned an AWSElastic Block Store volume using a PersistentVolumeClaim to store the metricsdata collected by Prometheus. Prometheus needed aconfiguration filewhere we defined what all targets it will be scraping metrics from. This is aYAML file which we stored in a ConfigMap in our cluster.

Targets in Prometheus can be anything that exposes metrics data in thePrometheus format at a /metrics endpoint. This can be your applicationservers, Kubernetes API servers or even Prometheus itself. Prometheus wouldscrape the data at the defined scrape_interval and store it in the volume astime series data. This can be queried and visualized in the Prometheus dashboardthat comes bundled in the Prometheus deployment.

We used kubectl port-forward command to test that Prometheus is workinglocally. Once everything was tested and we confirmed, we exposed Prometheus withan ingress so that we can hit its APIs with that url.

Initially we had configured the following targets:

1. node_exporter from Prometheus,which would scrape the metrics of the machine the deployment is running on.
2. kube-state-metrics whichwould listen to the Kubernetes API and store metrics of all the objects.
3. Traefik for all the network-related metrics(like the number of requests etc.) since we are using Traefik as our ingresscontroller.
4. kubernetes-nodes
5. kubernetes-pods
6. kubernetes-cadvisor
7. kubernetes-service-endpoints

The last 4 scrape jobs would be collecting metrics from the Kubernetes REST APIrelated to nodes, pods, containers and services respectively.

For scraping metrics from all of these targets, we had set a resource request of500 MB of RAM and 0.5 vCPU to our Prometheus deployment.

After setting up all of this, the Prometheus deployment was running fine, and wewere able to see the data from the Prometheus dashboard. Seeing this, we weresatisfied and happily started hacking with PromQL, Prometheus's query language.

The CrashLoopBackOff crime scene

CrashLoopBackOff is when a Kubernetes pod is going into a loop of crashing,restarting itself and then crashing again - and this was what was happening tothe Prometheus deployment we had created. From what we could see, the pod hadcrashed, and when it gets recreated, Prometheus would initialize itself and do areload of theWrite Ahead Log (WAL).

The WAL is there for adding additional durability to the database. Prometheusstores the metrics it scrapes in-memory before persisting them to the databaseas chunks, and the WAL makes sure that the in-memory data will not be lost inthe case of a crash. In our case, the Prometheus deployment was crashing and itwould get recreated. It would try to load the data from WAL into memory, andthen crash again before this was completed, leading to the CrashLoopBackOffstate.

We tried deleting the WAL blocks manually from the volume, even though thiswould incur some data loss. This was able to bring the deployment back up againsince WAL replay needn't be done. The deployment went into CrashLoopBackOffagain after a while.

Investigating the error

The first line of approach we took was to monitor the CPU, memory, and diskusage of the deployment. The disk usage seemed to be normal. We had provisioneda 100GB volume and it wasn't anywhere near getting used up. The CPU usage alsoseemed normal. The memory usage, however, was suspicious.

After the pods had crashed initially, we recreated the deployment and monitoredit using kubectl's --watch flag for following all the pod updates. While doingthis we were able to see that the pods were going into CrashLoopBackOff becausethey were getting OOMKilled first. The OOMKilled error in Kubernetes is when apod is terminated because it tries to use more memory than it is allotted in itsresource limits. We were consistently seeing the OOMKilled error so memorymust be the culprit here.

We added Prometheus itself as a target in Prometheus so that we could monitorthe memory usage of the Prometheus deployment. The following was the generaltrend of how Prometheus's memory was increasing over time. This would go onuntil the memory would cross the specified limit, and then the pod would go intoCrashLoopBackOff.

Now that we knew that memory was the issue, we started looking into what wascausing the memory leak. After talking with some folks from the Kubernetes Slackworkspace, we were asked to look at the TSDB status of the Prometheusdeployment. We monitored the stats in real time and saw that the number of timeseries data stored in the database was growing in tens of thousands by eachsecond! This lined up with the increase in the memory usage graph from earlier.

How we fixed it

We can calculate the memory requirement for Prometheus based on the number oftargets we are scraping metrics from and the frequency at which we are scrapingthe data. The memory requirement of the deployment is a function of both ofthese parameters. In our case ,this was definitely higher than what we couldafford to allocate (based on the nodegroup's machine type) since we werescraping a lot of data at a scrape interval of 15 seconds, which was set in thedefault configuration for Prometheus.

We increased the scrape interval to 60 seconds and removed all the targets fromthe Prometheus configuration whose metrics we didn't need for building thedashboard. Within the targets that we were scraping from, we used themetric_relabel_configs option to persist in the database only those metricswhich we needed and to drop everything else. We only needed thecontainer_cpu_usage_seconds_total, container_memory_usage_bytes and thetraefik_service_requests_total metrics - so we configured Prometheus so thatonly these three would be stored in our database, and by extension the WAL.

We redeployed Prometheus after making these changes and the memory showed greatstability afterwards. The following is the memory usage of Prometheus over thelast few days. It has not exceeded 1GB.

The aftermath

Once Prometheus was stable we were able to build the metrics dashboard with thePrometheus API in a straightforward manner. The metrics dashboard came to usewithin a couple of days, when the staging deployment ofNeetoCode had faced a downtime. You can see thechanges in the metrics from the time when the outage had occurred

The quintessential learning that we got from this experience is to always bewary of the resources that are being used up when it comes to tasks likescraping metrics over an extended period of time. We were scraping all themetrics initially in order to explore everything, even though all the metricswere not being used. But because of this, we were able to read a lot about howPrometheus works internally, and also learn some Prometheus best practices thehard way.

If your application runs on Heroku, you can deploy it on NeetoDeploy without anychange. If you want to give NeetoDeploy a try, then please send us an email atinvite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, followNeetoDeploy on Twitter. You can also join ourcommunity Slack to chat with us about any Neetoproduct.

PostgreSQL was running on the machine at the default port of 5432. Ports 443and 80 were open to everyone, for handling HTTP/S traffic. Port 22 was alsoopen to everyone, so that anyone with their public SSH keys added in theauthorized_keys file in the remote server, or having access to the private keyfile of the server could log into the machine remotely.

For development, I needed to access this remote database locally, so I editedthe pg_hba.conf file and opened PostgreSQL to the network. I added a new ruleto open port 5432 so that anyone could connect to the PostgreSQL instanceremotely if they had all the credentials. If you notice in the screenshot, youwill see all the ports that are open to the public network. This was all workinggreat for me, until one fine day it wasnt.

I realized that something was wrong when I couldnt connect to the PostgreSQLinstance remotely one day. The response I was getting was the standardis PostgreSQL running? error.

psql: could not connect to server: No such file or directoryConnection refused Is the server running on host ${hostname}and accepting TCP/IP connections on port 5432?

I was still able to SSH into the VM so I tried to restart PostgreSQL. After someinvestigation I figured out that PostgreSQL was back up momentarily when I dosystemctl restart postgresql, but it goes down again.

Inspecting the processes with htop I was able to see that all the CPU coreswere at 100% usage. Something didnt feel right. Sorting the processes based onthe percentage of CPU and memory used, I came across two peculiar processes -kdevtmpfsi and kinsing. A quick Google search showed that this was a cryptomining malware that spreads by exploiting flaws in resources that are exposed tothe public. Killing the process was of no use since the malware also adds a cronjob to replicate itself so that it cant be stopped.

Removing the malware

I found all files in the system with kdevtmpfsi and kinsing in their namesusing the unix find command and deleted them. The malwares files was insidethe /tmp directory.

find / -name kdevtmpfsi*find / -name kinsing*

Then I checked if there were any cron jobs running on the machine with thecrontab command. There were some jobs running that were there to reload themalware script, even if you delete it. I deleted the jobs related tokdevtmpfsi and kinsing. Another information I learnt was that in Unix, eachuser will have their own crontab which can run jobs as that particular user.

crontab -l  #To list all running cron jobscrontab -e #To delete running jobs

Things to pay attention to

I made all the passwords stronger, especially for the resources that were beingexposed to the public. One of the lessons I learnt was that you can always bemore secure, and that you should never compromise on your passwords. Thepasswords that I had set for my users were weak, with just a dictionary word, adigit and a special character - something like the format of himalaya7!

Instead of opening the required ports to the public network, I exposed them toonly the IP addresses from which I needed to access it.

Notice how the ports for SSH and PostgreSQL are only exposed to the required IPaddresses now.

I moved the application database to a managed PostgreSQL service rather thanrunning it in a VM by myself. This also means that I need not worry about theperformance or uptime, as all of this will be taken care of by AWS itself.

For extra security, I also set up a reverse proxy so that no one can ping mydeployed URL and get the IP address of the VM where the application is running.

Securing your deployments is as important as any other step when deploying yourapplication and it needs to be a priority right from when you are designing thearchitecture of your application. Taking care of such small details duringdevelopment will facilitate you in writing good code and following the rightpatterns from the start.

We have a bunch of files hosted in S3 which are served through CloudFront. Toreduce the CloudFront bandwidth cost and to make use of a global CDN (we usePrice Class 100 in CloudFront), we decided to use Cloudflare for filedownloads. This would help us cache files in Cloudflare edges and willeventually reduce the bandwidth costs at origin (CloudFront). But to do this, wehad to solve a few problems.

We had been signing CloudFront download URLs to restrict their usage after aperiod of time. This means the file download URLs are always unique. SinceCloudflare caches files based on URLs, caching will not work when the URLs aresigned. We had to remove the URL signing to get it working with Cloudflare, butwe can't allow people to continuously use the same download URL. CloudflareWorkers helped us with this.

We negotiated a deal with Cloudflare and upgraded the subscription to Enterpriseplan. Enterprise plan helps us define aCustom Cache Keyusing which we can configure Cloudflare to cache based on user defined key.Enterprise plan also increased cache file size limits. We wrote following Workercode which configures a custom cache key and authenticates URLs using HMAC.

Cloudflare worker starts with attaching a method to "fetch" event.

addEventListener("fetch", event => {  event.respondWith(verifyAndCache(event.request));});

verifyAndCache function can be defined as follows.

async function verifyAndCache(request) {  /**  source:  https://jameshfisher.com/2017/10/31/web-cryptography-api-hmac.html  https://github.com/diafygi/webcrypto-amples#hmac-verify  https://stackoverflow.com/questions/17191945/conversion-between-utf-8-arraybuffer-and-string  **/  // Convert the string to array of its ASCII values  function str2ab(str) {    let uintArray = new Uint8Array(      str.split("").map(function (char) {        return char.charCodeAt(0);      })    );    return uintArray;  }  // Retrieve to token from query string which is in the format "<time>-<auth_code>"  function getFullToken(url, query_string_key) {    let full_token = url.split(query_string_key)[1];    return full_token;  }  // Fetch the authentication code from token  function getAuthCode(full_token) {    let token = full_token.split("-");    return token[1].split("/")[0];  }  // Fetch timestamp from token  function getExpiryTimestamp(full_token) {    let timestamp = full_token.split("-");    return timestamp[0];  }  // Fetch file path from URL  function getFilePath(url) {    let url_obj = new URL(url);    return decodeURI(url_obj.pathname);  }  const full_token = getFullToken(request.url, "&verify=");  const token = getAuthCode(full_token);  const str =    getFilePath(encodeURI(request.url)) + "/" + getExpiryTimestamp(full_token);  const secret = "< HMAC KEY >";  // Generate the SHA-256 hash from the secret string  let key = await crypto.subtle.importKey(    "raw",    str2ab(secret),    { name: "HMAC", hash: { name: "SHA-256" } },    false,    ["sign", "verify"]  );  // Sign the "str" with the key generated previously  let sig = await crypto.subtle.sign({ name: "HMAC" }, key, str2ab(str));  // convert the Arraybuffer "sig" in string and then, in Base64 digest, and then URLencode it  let verif = encodeURIComponent(    btoa(String.fromCharCode.apply(null, new Uint8Array(sig)))  );  // Get time in Unix epoch  let time = Math.floor(Date.now() / 1000);  if (time > getExpiryTimestamp(full_token) || verif != token) {    // Render error response    const init = {      status: 403,    };    const modifiedResponse = new Response(`Invalid token`, init);    return modifiedResponse;  } else {    let url = new URL(request.url);    // Generate a cache key from URL excluding the unique query string    let cache_key = url.host + url.pathname;    let headers = new Headers(request.headers);    /**    Set an optional header/auth token for additional security in origin.    For example, using AWS Web Application Firewall (WAF), it is possible to create a filter    that allows requests only with a custom header to pass through CloudFront distribution.    **/    headers.set("X-Auth-token", "< Optional Auth Token >");    /**    Fetch the file using cache_key. File will be served from cache if it's already there,    or it will send the request to origin. Please note 'cacheKey' is available only in    Enterprise plan.    **/    const response = await fetch(request, {      cf: { cacheKey: cache_key },      headers: headers,    });    return response;  }}

Once the worker is added, configure an associated route in"Workers -> Routes -> Add Route" in Cloudflare.

Now, all requests will go through the configured Cloudflare worker. Each requestwill be verified using HMAC authentication and all files will be cached inCloudflare edges. This would reduce bandwidth costs at the origin.

It can be configured in EC2 -> Auto Scaling Groups -> Scaling Policies.

We can also configure a warm-up period so that it would wait before it launchesmore instances to keep the metric at the configured value.

Internally, we use terraform to manage AWS resources. We can configure TargetTracking Policy using terraform as follows.

resource "aws_launch_configuration" "web_cluster" {name_prefix = "staging-web-cluster"image_id = "<image ID>"instance_type = "<instance type>"key_name = "<ssh key name>"security_groups = ["<security group>"]user_data = "<user_data script>"root_block_device {volume_size = "<volume size>"}lifecycle {create_before_destroy = true}}resource "aws_autoscaling_group" "web_cluster" {name = "staging-web-cluster-asg"min_size = "<min ASG size>"max_size = "<max ASG size>"default_cooldown = "300"launch_configuration = "\${ aws_launch_configuration.web_cluster.name }"vpc_zone_identifier = ["<subnet ID>"]health_check_type = "EC2"health_check_grace_period = 300target_group_arns = ["<target group arn>"]}resource "aws_autoscaling_policy" "web_cluster_target_tracking_policy" {name = "staging-web-cluster-target-tracking-policy"policy_type = "TargetTrackingScaling"autoscaling_group_name = "\${aws_autoscaling_group.web_cluster.name}"estimated_instance_warmup = 200target_tracking_configuration {predefined_metric_specification {predefined_metric_type = "ASGAverageCPUUtilization"}    target_value = "60"}}

Target Tracking Policy allows us to easily configure and manage autoscaling inEC2. It's particularly helpful while running services like web servers.

Staging environment helps us in testing the code before pushing the code toproduction. However it becomes hard to manage the staging environment when morepeople work on different parts of the application. This can be solved byimplementing a system where feature branch can have its own individual stagingenvironment.

Heroku hasReview Apps featurewhich can deploy different branches separately. Gumroad,doesn't use Heroku so we built a custom in-house solution.

The first step was to build the infrastructure. We created a new Auto ScalingGroup, Application Load Balancer and route in AWS for the review apps. Loadbalancer and route are common for all review apps, but a new EC2 instance iscreated in the ASG when a new review app is commissioned.


The main challenge was to forward the incoming requests to the correct serverrunning the review app. This was made possible usingLua in nginx andconsul. When a review app is deployed, it writes itsIP and port to consul along with the hostname. Each review app server runs aninstance of OpenResty (Nginx + Lua modules) withthe following configuration.

server {  listen                   80;  server_name              _;  server_name_in_redirect  off;  port_in_redirect         off;  try_files $uri/index.html $uri $uri.html @app;  location @app {    set $upstream "";    rewrite_by_lua '      http   = require "socket.http"      json   = require "json"      base64 = require "base64"      -- read upstream from consul      host          = ngx.var.http_host      body, c, l, h = http.request("http://172.17.0.1:8500/v1/kv/" .. host)      data          = json.decode(body)      upstream      = base64.decode(data[1].Value)      ngx.var.upstream = upstream    ';    proxy_buffering   off;    proxy_set_header  Host $host;    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;    proxy_redirect    off;    proxy_pass        http://$upstream;  }}

It forwards all incoming requests to the correct IP:PORT after looking up inconsul with the hostname.

The next task was to build a system to deploy the review apps to thisinfrastructure. We were already using docker in both production and stagingenvironments. We decided to extend it to deploy branches by building dockerimage for every branch with deploy- prefix in the branch name. When such abranch is pushed to GitHub, a CircleCI job is run to build a docker image withthe code and all the necessary packages. This can be configured using aconfiguration template like this.

jobs:  build_image:    <<: *defaults    parallelism: 2    steps:      - checkout      - setup_remote_docker:          version: 17.09.0-ce      - run:          command: |            ci_scripts/2.0/build_docker_image.sh          no_output_timeout: 20mworkflows:  version: 2  web_app:    jobs:      - build_image:          filters:            branches:              only:                - /deploy-.*/

It also pushes static assets like JavaScript, CSS and images to an S3 bucketfrom where they are served directly through CDN. After building the dockerimage, another CircleCI job is run to do the following tasks.

• Create a new database in RDS and configure the required credentials.
• Scale up Review App's Auto Scaling Group by increasing the number of desiredinstances by 1.
• Run redis, database migration, seed-data population, unicorn and resqueinstances using nomad.

The ease of deploying a review app helped increase our productivity.

Changes to MongoDB instances

Depending on the requirements, modern web applications use different third-partyservices. For example, it's easy and cost effective to subscribe to a GeoIPlookup service than building and maintaining one. Some third-party services getvery expensive as the usage increases but people don't look for alternatives dueto legacy reasons.

In our case, our client had been paying more than $5,000/month for athird-party MongoDB service. This service charges based on the storage used andwe had years of data in it. This data is consumed by a machine learning systemto fight fraudulent purchases and users. We had a look at both the ML system andthe data in MongoDB and found we actually didn't need all the data in thedatabase. The system never read data older than 30-60 days in some of thebiggest mongo collections.

Since we were already using nomad as ourscheduler, we wrote a periodic nomad job that runs every week to deleteunnecessary data. The nomad job syncs both primary and secondary MongoDBinstances to release the free space back to OS. This helped reduce monthly billto $630/month.

Changes to MongoDB service provider

Then we looked at the MongoDB service provider. It was configured years backwhen the application was built. There are other vendors who provided the sameservice for a much cheaper price. We switched our MongoDB to mLab and now thedatabase runs in a $180/month dedicated cluster. With WiredTiger'scompressionenabled, we don't use as much storage we used to use before.

Making use of Auto Scaling

Auto Scaling can be a powerful tool when it comes to reducing costs. We had beenrunning around 15 large EC2 instances. This was inefficient due to following tworeasons.

1. It cannot cope up when the traffic increases beyond its limit.
2. Resources are underused when traffic is less.

Auto Scaling solves both the issues. For web servers, we switched to smallerinstances and usedTarget Tracking Scaling Policyto keep the average aggregate CPU utilization at 70%.

Background job workers made use of a nomad job we built. It periodicallycalculated the number of required instances based on the count of pending jobsand the job's queue priority. This number was pushed to CloudWatch as a metricand the Auto Scaling group scaled based on that. This approach was effective inboosting performance and reducing cost.

Buying reserved instances

AWS has a feature to reserve instances for services like EC2, RDS, etc.. It'soften preferable to buy reserved instances than running the application usingon-demand instances. We evaluated reserved instance utilization using thereporting tooland bought the required reserved instances.

Looking for cost-effective solutions

Sometimes, different solutions to the same problem can have different costs. Forexample, we had been facing small DDoS attack regularly and we had to rate-limitrequests based on IP and other parameters. Since we had been using Cloudflare,we could have used their rate-limiting feature. Performance wise, it was thebest solution but they charge based on the number of good requests. It would beexpensive for us since it's a high-traffic application. We looked for othersolutions and solved the problem using Rack::Attack. Wewrote a blogabout it sometime back. The solution presented in the blog was effective inmitigating the DDoS attack we faced and didn't cost us anything significant.

Requesting custom pricing

If you are a comparatively larger customer of a third-party service, it's morelikely that you don't have to pay the published price. Instead, we could requestfor custom pricing. Many companies will be happy to give 20% to 50% pricediscounts if we can commit to a minimum spending in the year. We triednegotiating a new contract for an expensive third-party service and got the dealwith 40% discount compared to their published minimum price.

Running an infrastructure can be both technically and economically challenging.But if we can look between the lines and if we are willing to update existingsystems, we would be amazed in terms of how much money we can save every month.