At neeto we are building 20+ applications, and most ofour applications are running in NeetoDeploy. Once we migrated from Heroku toNeetoDeploy, we started getting 520 response code for our applications. Thisissue was occurring randomly and rarely.

What is 520 response code?

A 520 response code happens when the connection is started on the origin webserver, but the request is not completed. This could be due to server crashes orthe inability to handle the incoming requests because of insufficient resources.

When we looked at our logs closely, we found that all the 520 response codesituations occurred when we restarted or deployed the app. From this, weconcluded that the new pods are failing to handle requests from the clientinitially and working fine after some time.

What is wrong with new pods?

Once our investigation narrowed down to the new pods, we quickly realized thatrequests are arriving at the server even when the server is not fully ready yetto take new requests.

When we create a new pod in Kubernetes, it is marked as "Ready", and requestsare sent to it as soon as its containers start. However, the servers initiatedwithin these containers may require additional time to boot up and to becomeready to accept the requests fully.

Let's try restarting the application

$ kubectl rollout restart deployment bling-staging-web

As we can see, a new container is getting created for the new pod. The READYstatus for the new pod is 0. It means it's not yet READY.

NAME                               READY  STATUS             RESTARTS  AGEbling-staging-web-656f74d9d-6kpzz  1/1    Running            0         2m8sbling-staging-web-79fc6f978-cdjf5  0/1    ContainerCreating  0         5s

Now we can see that the new pod is marked as READY (1 out of 1), and the old oneis terminating.

NAME                               READY  STATUS             RESTARTS  AGEbling-staging-web-656f74d9d-6kpzz  0/1    Terminating        0         2m9sbling-staging-web-79fc6f978-cdjf5  1/1    Running            0         6s

The new pod is shown as READY as soon as the container was created. But onchecking the logs, we could see that the server was still starting up and notready yet.

[1] Puma starting in cluster mode...[1] Installing dependencies...

From the above observation, we understood that the pod is marked as "READY"right after the container is created. Consequently, requests are received evenbefore the server is fully prepared to serve them, and they get a 520 responsecode.

Solution

To fix this issue, we must ensure that pods are marked as "Ready" only after theserver is up and ready to accept the requests. We can do this by usingKuberneteshealth probes.More than six years ago we wrotea blogon how we can leverage the readiness and liveness probes of Kubernetes.

Adding Startup probe

Initially, we only addedStartup probesince we had a problem with the boot-up phase. You can read more about theconfiguration settingshere.

The following configuration will add the Startup probe for the deployments:

startupProbe:  failureThreshold: 10  httpGet:    path: /health_check    port: 3000    scheme: HTTP  periodSeconds: 5  successThreshold: 1  timeoutSeconds: 60  initialDelaySeconds: 10

/health_check is a route in the application that is expected to return a 200response code if all is going well. Now, let's restart the application againafter adding the Startup probe.

Container is created for the new pod, but the pod is still not "Ready".

NAME                               READY  STATUS            RESTARTS  AGEbling-staging-web-656f74d9d-6kpzz  1/1    Running           0         2m8sbling-staging-web-79fc6f978-cdjf5  0/1    Running           0         5s

The new pod is marked as "Ready", and the old one is "Terminating".

NAME                               READY  STATUS            RESTARTS  AGEbling-staging-web-656f74d9d-6kpzz  0/1    Terminating       0         2m38sbling-staging-web-79fc6f978-cdjf5  1/1    Running           0         35s

If we check the logs, we can see the health check request:

  [1] Puma starting in cluster mode...  [1] Installing dependencies...  [1] * Puma version: 6.3.1 (ruby 3.2.2-p53) ("Mugi No Toki Itaru")  [1] *  Min threads: 5  [1] *  Max threads: 5  [1] *  Environment: heroku  [1] *   Master PID: 1  [1] *      Workers: 1  [1] *     Restarts: () hot () phased [1] * Listening on http://0.0.0.0:3000 [1] Use Ctrl-C to stop [2024-02-10T02:40:48.944785 #23]  INFO -- : [bb9e756a-51cc-4d6b-9a4a-96b0464f6740] Started GET "/health_check" for 192.168.120.195 at 2024-02-10 02:40:48 +0000 [2024-02-10T02:40:48.946148 #23]  INFO -- : [bb9e756a-51cc-4d6b-9a4a-96b0464f6740] Processing by HealthCheckController#healthy as */* [2024-02-10T02:40:48.949292 #23]  INFO -- : [bb9e756a-51cc-4d6b-9a4a-96b0464f6740] Completed 200 OK in 3ms (Allocations: 691)

Now, the pod is marked as "Ready" only after the health check succeeds, in otherwords, only when the server is prepared to accept the requests.

Fixing the Startup probe for production applications

Once we released the health check for our deployments, we found that healthchecks were failing for all production applications but working for staging andreview applications.

We were getting the following error in our production applications.

Startup probe failed: Get "https://192.168.43.231:3000/health_check": http: server gave HTTP response to HTTPS client2024-02-12 06:40:04 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>

From the above logs, it was clear that the issue was related to SSLconfiguration. On comparing the production environment configuration with theothers, we figured out that we had enabledforce_sslfor production applications. The force_ssl=true setting ensures that allincoming requests are SSL encrypted and will automatically redirect to their SSLcounterparts.

The following diagram broadly shows the path of an incoming request.

From the above diagram, we can infer the following things:

• SSL verification is happening in the ingress controller and not in the server.
• Client requests are going through the ingress controller before reaching theserver.
• Request from ingress controller to the pod is an HTTP request.
• The HTTP health check requests are directly sent from Kubelet to the pod anddo not go through the ingress controller.

Here is how our health check request works.

1. Kubeletsends an HTTP request to the server directly.
2. Since force_ssl is enabled,ActionDispatch::SSLmiddleware redirects the request to HTTPS.
3. When the HTTPS request reaches the server, Puma throwsAre you trying to open an SSL connection to a non-SSL Puma? error since noSSL certificates are configured with the server.

The solution to our problem lies in understanding why only the health checkrequest is rejected, whereas the request from the ingress controller is not,even though both are HTTP requests. This is because ingress controller sets someheaders before forwarding to the pod, and the header we are concerned about isX-FORWARDED-PROTO.The X-Forwarded-Proto header contains the HTTP/HTTPS scheme the client used toaccess the application. When a client makes an HTTPS request, the ingresscontroller terminates the SSL/TLS connection and forwards the request to thebackend service using plain HTTP after adding theX-Forwarded-Proto along withthe other headers.

Everything started working after adding the X-Forwarded-Proto header to ourstartup probe request.

startupProbe:  failureThreshold: 10  httpGet:    httpHeaders:      - name: X-FORWARDED-PROTO        value: https    path: <%= health_check_url %>    port: <%= port %>    scheme: HTTP  periodSeconds: 5  successThreshold: 1  timeoutSeconds: 60  initialDelaySeconds: 10

We also addedReadinessandLivenessprobes for our deployments.

If your application runs on Heroku, you can deploy it on NeetoDeploy without anychange. If you want to give NeetoDeploy a try, then please send us an email atinvite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, followNeetoDeploy on Twitter. You can also join ourSlack community to chat with us about any Neeto product.

What is sleep when idle feature

"Sleep when idle" is a feature of NeetoDeploy, which puts the deployedapplication to sleep when there is no hit to the server for 5 minutes. Thishelps reduce the cost of the server.

"Sleep when idle" feature can be enabled not only for the pull request reviewapplications, but for staging and production applications too. Many folks buildapplications to learn and for hobby. In such cases, there is no point in runningthe server when the server is not likely to get any traffic. Since NeetoDeploybilling is based on the usage "Sleep when idle" feature helps keep the bill lowfor the users.

Let's say you build something and you deployed to production. You shared it withyour friends. For a day or two you got a bit of traffic, and after that youmoved on to other things. If "sleep when idle" is enabled then you don't need toworry about anything. If the server is not getting any traffic then you will notbe billed.

How is Neeto using sleep when idle feature

At neeto, we are building 20+ applications at the sametime. It means lots of pull requests for all these products and thus lots of PRreview apps are created.

For a long time, we were using Heroku to build the review apps. However whenNeetoDeploy started to become stable, we movedto generating PR review apps fromHeroku to NeetoDeploy. This helped reduce cost.

How to make deployments sleep when idle?

This video describes how "sleep when idle" feature is implemented.

<iframewidth="560"height="315"src="https://youtube.com/embed/trn2DJyTjnw"frameborder="0"title="How we designed NeetoDeploy's 'sleep when idle' feature"allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"allowfullscreen

</iframe>

Keeping the apps running only when they're being used involves two steps:

1. Scaling the deployments down and bringing them back up again
2. Figuring out when to do the scaling

The deployments can be scaled easily using the kubectl scale command. Forexample, if we want to turn our deployment off, we can run the following toupdate our deployment to zero replicas, essentially destroying all the pods.

kubectl scale deployment/nginx --replicas=0

We can also delete our service, ingress or any other resource we might havecreated for our deployment. The configuration of the deployment itself wouldstill be present in the cluster even when we make it sleep, since the KubernetesDeployment is not deleted.

When we want to bring our app back up again, we can use the same command to spinup new pods:

kubectl scale deployment/nginx --replicas=1

The challenge was to figure out when to do this. We decided that we'd have athreshold based on the time the app is last accessed by users. If theapplication is not accessed for more than five minutes, we consider theapplication to be idle and we will scale it down. It'll be brought back up whena user tries to access it again.

Exploring existing solutions

There are existing CNCF projects like Knative andKeda, which can potentially be used to achieve what we wanthere. We spent some time exploring these but realized that these solutionsweren't exactly suitable for our requirements. Kubernetes also natively has aHPAScaleToZerofeature gatewhich enables theHorizontal Pod Autoscalerto scale down deployments to zero pods, but this is still in alpha and, hence isnot available in EKS yet.

Ultimately, we decided to write our own service for achieving this. The entirebackend of NeetoDeploy was designed asa collection of microservicesfrom day one. So it made sense to build our pod idling service as anothermicroservice that runs in our cluster.

Figuring out when to make applications sleep

To know when applications can be idled, we need to know when people areaccessing the applications from their browsers. Since all the requests toapplications deployed on NeetoDeploy would go through our load balancer, itwould contain the information of when every app was last accessed.

We use Traefik as our load balancer and we usedTraefik's middlewares toretrieve and process the information of when apps are being accessed. We wrote acustom middleware to send all the request information to the pod idling service,whenever an app is being accessed. The pod idling service would store all theURLs, along with the timestamp at which they were accessed, in a Redis cache.The following graphic shows how the request information would be collected andstored by the pod idling service into its Redis cache, both of which are runningwithin the cluster.

The pod idling service would then filter the apps that were last accessed morethan five minutes ago. It then sends a request to the cluster to scale all theseapps down. We'd also delete any related resources like the Services and theIngressRoutes used to configure networking for the deployments.

We first tested this by running the service manually, and sure enough, all theinactive deployments are filtered and scaled properly. We then added this as acron job in the pod idling service, which would run every five minutes. Thismeans that no app would run for more than five minutes if they're not beingused.

But wait! How would we bring the app back up after scaling it down?

Building the downtime service

As we discussed above, we use Traefik's IngressRoutes to route traffic to theapplication being accessed. We made use of thepriority parameterof IngressRoutes to boot up apps that are sleeping. Essentially, we created awildcard Traefik IngressRoute that points to a "downtime service" deployment,which is a React app that serves a message of There's nothing here, yet to letusers know that the app they're trying to access doesn't exist. You can see thisin action if you visit a random URL in NeetoDeploy, say something likenonexistent-appname.neetodeployapp.com

Wildcard IngressRoutes have the least priority by default. So if we create a"catch-all" wildcard IngressRoute, any invalid url without an IngressRoute ofits own, can be redirected to a single Service in Kubernetes. This is how we'reredirecting non-existent apps to the page shown above. In the following graphic,we can see how a request to a random URL is routed to the downtime service withthe wildcard IngressRoute.

This also means that if an app is scaled down by the pod idling service and getsits IngressRoute deleted, the next time a user tries to access the app, therequest would instead be routed to the downtime service. We need to handle thescale up logic from the downtime service.

Whenever a user requests a URL that doesn't have an IngressRoute, there are twopossibilities.

1. The app doesn't exist.
2. The app exists, but is currently scaled down.

The downtime service would first check the cluster if the requested app ispresent in the cluster in a sleeping state. If not then the user will be servedthe "There's nothing here, yet" page. If there is a sleeping deployment,however, we boot it back up. The downtime service sends the scale up request tothe cluster. We keep redirecting the user back to the url till the app is up andrunning. This redirection would keep happening until the app is scaled up sincewe create the Service and IngressRoute only after the pods of the app arerunning. At this point, the request will be routed to the correct pod by theapp's IngressRoute, since it has a higher priority than the wildcardIngressRoute of the downtime service. All of these steps are illustrated in theGIF below:

This design worked flawlessly and we were able to bring back scaled downapplications with as low as 20-30 seconds of wait time.

Conclusion

We've been running this setup for almost a year now, and it has been workingsmoothly so far. Pod idling service and the downtime service started as simplermicroservices and continue to evolve, adapting to the increasing demand as wegrow.

If your application runs on Heroku, you can deploy it on NeetoDeploy without anychange. If you want to give NeetoDeploy a try, then please send us an email atinvite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, followNeetoDeploy on X. You can also join ourcommunity Slack to chat with us about anyNeeto product.

One of the features that we wanted in our cloud deployment platform,NeetoDeploy was an application metrics. We decided to usePrometheus for building this feature. Prometheus is anopen source monitoring and alerting toolkit and is a CNCF graduated project.Venturing into the Cloud Native ecosystem of projects apart from Kubernetes wassomething we had never done before. We ended up learning a lot about Prometheusand how to use it during the course of building this feature.

Initial setup

We installed Prometheus in our Kubernetes cluster by writing a deploymentconfiguration YAML and applying it to our cluster. We also provisioned an AWSElastic Block Store volume using a PersistentVolumeClaim to store the metricsdata collected by Prometheus. Prometheus needed aconfiguration filewhere we defined what all targets it will be scraping metrics from. This is aYAML file which we stored in a ConfigMap in our cluster.

Targets in Prometheus can be anything that exposes metrics data in thePrometheus format at a /metrics endpoint. This can be your applicationservers, Kubernetes API servers or even Prometheus itself. Prometheus wouldscrape the data at the defined scrape_interval and store it in the volume astime series data. This can be queried and visualized in the Prometheus dashboardthat comes bundled in the Prometheus deployment.

We used kubectl port-forward command to test that Prometheus is workinglocally. Once everything was tested and we confirmed, we exposed Prometheus withan ingress so that we can hit its APIs with that url.

Initially we had configured the following targets:

1. node_exporter from Prometheus,which would scrape the metrics of the machine the deployment is running on.
2. kube-state-metrics whichwould listen to the Kubernetes API and store metrics of all the objects.
3. Traefik for all the network-related metrics(like the number of requests etc.) since we are using Traefik as our ingresscontroller.
4. kubernetes-nodes
5. kubernetes-pods
6. kubernetes-cadvisor
7. kubernetes-service-endpoints

The last 4 scrape jobs would be collecting metrics from the Kubernetes REST APIrelated to nodes, pods, containers and services respectively.

For scraping metrics from all of these targets, we had set a resource request of500 MB of RAM and 0.5 vCPU to our Prometheus deployment.

After setting up all of this, the Prometheus deployment was running fine, and wewere able to see the data from the Prometheus dashboard. Seeing this, we weresatisfied and happily started hacking with PromQL, Prometheus's query language.

The CrashLoopBackOff crime scene

CrashLoopBackOff is when a Kubernetes pod is going into a loop of crashing,restarting itself and then crashing again - and this was what was happening tothe Prometheus deployment we had created. From what we could see, the pod hadcrashed, and when it gets recreated, Prometheus would initialize itself and do areload of theWrite Ahead Log (WAL).

The WAL is there for adding additional durability to the database. Prometheusstores the metrics it scrapes in-memory before persisting them to the databaseas chunks, and the WAL makes sure that the in-memory data will not be lost inthe case of a crash. In our case, the Prometheus deployment was crashing and itwould get recreated. It would try to load the data from WAL into memory, andthen crash again before this was completed, leading to the CrashLoopBackOffstate.

We tried deleting the WAL blocks manually from the volume, even though thiswould incur some data loss. This was able to bring the deployment back up againsince WAL replay needn't be done. The deployment went into CrashLoopBackOffagain after a while.

Investigating the error

The first line of approach we took was to monitor the CPU, memory, and diskusage of the deployment. The disk usage seemed to be normal. We had provisioneda 100GB volume and it wasn't anywhere near getting used up. The CPU usage alsoseemed normal. The memory usage, however, was suspicious.

After the pods had crashed initially, we recreated the deployment and monitoredit using kubectl's --watch flag for following all the pod updates. While doingthis we were able to see that the pods were going into CrashLoopBackOff becausethey were getting OOMKilled first. The OOMKilled error in Kubernetes is when apod is terminated because it tries to use more memory than it is allotted in itsresource limits. We were consistently seeing the OOMKilled error so memorymust be the culprit here.

We added Prometheus itself as a target in Prometheus so that we could monitorthe memory usage of the Prometheus deployment. The following was the generaltrend of how Prometheus's memory was increasing over time. This would go onuntil the memory would cross the specified limit, and then the pod would go intoCrashLoopBackOff.

Now that we knew that memory was the issue, we started looking into what wascausing the memory leak. After talking with some folks from the Kubernetes Slackworkspace, we were asked to look at the TSDB status of the Prometheusdeployment. We monitored the stats in real time and saw that the number of timeseries data stored in the database was growing in tens of thousands by eachsecond! This lined up with the increase in the memory usage graph from earlier.

How we fixed it

We can calculate the memory requirement for Prometheus based on the number oftargets we are scraping metrics from and the frequency at which we are scrapingthe data. The memory requirement of the deployment is a function of both ofthese parameters. In our case ,this was definitely higher than what we couldafford to allocate (based on the nodegroup's machine type) since we werescraping a lot of data at a scrape interval of 15 seconds, which was set in thedefault configuration for Prometheus.

We increased the scrape interval to 60 seconds and removed all the targets fromthe Prometheus configuration whose metrics we didn't need for building thedashboard. Within the targets that we were scraping from, we used themetric_relabel_configs option to persist in the database only those metricswhich we needed and to drop everything else. We only needed thecontainer_cpu_usage_seconds_total, container_memory_usage_bytes and thetraefik_service_requests_total metrics - so we configured Prometheus so thatonly these three would be stored in our database, and by extension the WAL.

We redeployed Prometheus after making these changes and the memory showed greatstability afterwards. The following is the memory usage of Prometheus over thelast few days. It has not exceeded 1GB.

The aftermath

Once Prometheus was stable we were able to build the metrics dashboard with thePrometheus API in a straightforward manner. The metrics dashboard came to usewithin a couple of days, when the staging deployment ofNeetoCode had faced a downtime. You can see thechanges in the metrics from the time when the outage had occurred

The quintessential learning that we got from this experience is to always bewary of the resources that are being used up when it comes to tasks likescraping metrics over an extended period of time. We were scraping all themetrics initially in order to explore everything, even though all the metricswere not being used. But because of this, we were able to read a lot about howPrometheus works internally, and also learn some Prometheus best practices thehard way.

If your application runs on Heroku, you can deploy it on NeetoDeploy without anychange. If you want to give NeetoDeploy a try, then please send us an email atinvite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, followNeetoDeploy on Twitter. You can also join ourcommunity Slack to chat with us about any Neetoproduct.

Building the CLI tool for NeetoDeploy became our toppriority after we had built all the basic features in NeetoDeploy. Once westarted migrating our apps from Heroku to NeetoDeploy, the need for a CLI toolarose, since previously developers were using Heroku CLI for their scripts etc.

We started building the CLI as a Ruby gem using Thor.Installing the gem and using it would be as simple as runninggem install neetodeploy. We wanted the users to be authenticated via Neeto'sauthentication system before they could do anything else with the CLI. For this,we added a neetodeploy login command, which would create a session andredirect you to the browser where you can log in. The CLI would be polling thesession login status, and once you have logged in, it would store the sessiontoken and your email address inside the ~/.config/neetodeploy directory. Thiswill be used to authenticate your request when you run everything else.

The basic CLI functionality

The v1 release of the NeetoDeploy CLI shipped the following commands:

• config : For listing, creating, and deleting environment variables for yourapps.
• exec : For getting access to a shell running inside your deployed app.
• logs : For streaming the application logs right to your terminal.

Let's look at each of them in detail.

Setting environment vars

Building the config command was the most straightforward of the three. Thecentral source or truth for all the environment variables of the apps is storedin the web dashboard of NeetoDeploy. There are APIs already in place for doingCRUD operations on these from the dashboard app. This could be expanded for theuse of the CLI as well.

We first added API endpoints in the dashboard app, under a CLI namespace, tolist, create, and delete config variables. We then added different commands inthe CLI to send HTTP requests to the respective endpoints in the dashboard app.

You can pass the app name as an argument, and the CLI will send a request to thedashboard app along with the session token generated when you logged in. At thebackend, this session token would be used to check if you have access to the appyou're requesting, before doing anything else.

If you have used Heroku CLI, then you would be very comfortable sinceNeetoDeploy CLI follows the same style.

Architecting the neetodeploy exec command

The exec command is used to get access to a shell inside your deployed app.The NeetoDeploy web dashboard already has a functioning console feature withwhich you can get access to the shell inside your applications dynos. This wasbuilt using websockets. We would deploy a "console" pod with the same image andenvironment variables as the app, to which we'll connect with websockets. Wehave a dyno-console-manager app which spawns a new process inside the consoledyno and handles the websocket connections, etc. When we started building theexec feature for the CLI however, we explored a few options and learnt a lot ofthings before finally settling back on the websockets approach.

Weve been using the kubectl exec command on a daily basis to get access toshells inside pods running in the cluster. Since kubectl exec is cosmeticallysimilar to how we use SSH to access remote machines, the first approach we tookinvolved using SSH to expose containers outside the cluster.

We can expose our deployment outside the cluster using SSH in two ways. Thefirst approach would be running an SSH server as asidecar containerwithin the console pod. The second approach would be to bundlesshd within the image forthe console pod using a custom buildpack. With either of these two methods, youwould be able to SSH into your console deployment if you have the correct SSHkeys. We ran sshd inside a pod using acustom sshd buildpack. We thenconfigured the SSH public key inside the pod manually with kubectl exec. Withthis, we were able to SSH into the pod after port forwarding the sshd processsport to localhost using kubectl port-forward.

Great! Now that the SSH connection works, the part of the puzzle that remainednow was how we would actually expose the deployment outside the cluster inproduction. We tried doing this in a couple of different ways.

1. LoadBalancer

We were able to SSH into the pod after exposing it as aLoadBalancerservice. This was out of the question for production however, since AWS has ahard limit on the number of application load balancers (ALB) we can create perregion. Wed also have to pay for each load balancer we create. This approachwouldnt scale at all.

2. NodePort

The next option was to expose the pod as a NodePort service. A NodePortservice in Kubernetes means that the pod would be exposed through a certain portnumber through every node in the cluster. We could SSH into the pod with theexternal IP of any of the nodes. We tried exposing the test console pod wecreated with a NodePort service as well. We were able to SSH into the pod fromoutside the cluster without having to do the port forward. There was onelimitation for using NodePorts that we knew would render this method suboptimalfor building a Platform as a Service. The range of ports that can be allocatedfor NodePort services is from 30000 to 32768 by default. This means that wedonly be able to run 2768 instances of neetodeploy exec at any given time inour current cluster setup in EKS.

3. Bastion host, the SSH proxy server

We thought of deploying an SSH proxy server to circumvent the hard limits on theLoadBalancer and NodePort approaches. Such proxy servers are usually calledbastion hosts or jump servers. These are servers that are designed to provideaccess to a private network from a public network. The name "bastion" comes fromthe military structure that projects outwards from a castle or a fort. In asimilar sense, our bastion host will be a Kubernetes deployment that is exposedto the public network and acts as an interface between our private cluster andthe public internet. For our requirement, we can proxy SSH connections toconsole pods through this bastion host deployment. We only needed to expose thebastion host as a single LoadBalancer or a NodePort service, to which the CLIcan connect. This would solve the issue of having a hard limit on the number ofLoadBalancer or NodePort services we can create.

We quickly set up a bastion host and tested the whole idea. It was workingseamlessly! However, there were a lot of edge cases we'd have to handle if weare going to release this to the public. Ideally, we would want a new pair ofSSH keys for each console pod. We could generate this and store it as secrets inthe cluster. Apart from the hassle of handling all the SSH keys and updating thebastion host each time, we would have to make sure that users will not be ableto get SSH access to the bastion host deployment manually, outside the CLI. Thiswas difficult since the CLI would need access to the private key in order toconnect to the bastion host anyway. This would mean that users could SSH intothe bastion host after digging through the CLI gem's source, and possibly evenexec into any deployment in the cluster if they knew what they were looking for.Users with malicious intent could possibly bring down the bastion host itselfonce they're inside, rendering the exec command unusable.

We thought about installing kubectl inside the bastion host with a restrictiveRBAC so that wecan run kubectl exec from inside the bastion host, also while making sure thatusers could not run anything destructive from inside the bastion host. But thisjust adds more moving parts to the system. The SSH proxy approach makes sense ifyou have to expose your cluster among developers for internal use, but it is notideal when you are building a CLI for public use.

After figuring out how not to solve the problem, we came to the conclusionthat using websockets was the better approach for now. We already have thedyno-console-manager app, which we are using for the shell access from the webdashboard, we updated the same to handle connections from the CLI. From theCLI's end, we wrote a simple WebSocket client and handled the parsing of userinputs and printing the responses from the shell. All of the commands you enterwould be run on the console pod through the websocket connection.

Streaming logs to the terminal

Live logs were streamed to the web dashboard using websockets too. This was alsohandled by the above-mentioned dyno-console-manager. Since we had written a Rubywebsocket client for the exec command, we decided to use the same approachwith logs too.

The CLI would send a request to the dyno-console-manager with the apps name andthe dyno-console-manager would run kubectl logs for your app's deployment andstream it back to the CLI via a websocket connection it creates.

We took our time with architecting the NeetoDeploy CLI, but thanks to that welearnt a lot about designing robust systems that would scale in the long term.

If your application runs on Heroku, you can deploy it on NeetoDeploy without anychange. If you want to give NeetoDeploy a try, then please send us an email atinvite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, followNeetoDeploy on X. You can also join ourcommunity Slack to chat with us about anyNeeto product.

Challenges faced

The Cluster Autoscaler works by monitoring the pods and checking if they arewaiting to get scheduled. If no nodes are available to schedule the waitingpods, they will be in Pending state, until the kube-scheduler finds asuitable node.

With our default Cluster Autoscaler setup, if a pod is provisioned when thecluster is at maximum capacity, the pod will go into Pending state and remainthere until the cluster has scaled up. This entire flow consists of thefollowing steps:

• The Cluster Autoscaler scans the pods and identifies the un-schedulable pods.
• The Cluster Autoscaler provisions a new node. We have configured theautoscaler deployment with the necessary permissions for the autoscalinggroups in EC2, with which the autoscaler can provision new EC2 machines.
• Kubernetes adds the newly provisioned node to the clusters control plane, andthe cluster is scaled up.
• The pending pod gets scheduled to the newly added node.

From the perspective of a user who wants to get their application deployed, aconsiderable amount of time is taken to complete all these steps in addition tothe time taken to build and release their code. We can only optimize it so muchby decreasing the scan interval of the Cluster Autoscaler. We need a mechanismby which we can always have some buffer nodes ready so that the usersdeployments never go into Pending state and wait for the nodes to beprovisioned.

Pod Priority, Preemption and Overprovisioning

Kubernetes allows us to assign different priorities to pods. By default, all thepods have a priority of 0. When a pod is scheduled, it can preempt or evictother pods with lower priority than itself and take its place. We will use thisfeature of pod preemption to reserve resources for incoming deployments.

We created a new pod priority class with a priority of -1 and used this tooverprovisionpause pods. We allocateda fixed amount of CPU and memory resources to these pause pods. The pausecontainer is a placeholder container used by Kubernetes internally and doesntdo anything by itself.

Now if the cluster is at full capacity and a new deployment is created,technically the newly created pod doesn't have space to be scheduled in thecluster. In normal cases, this would warrant a scale-up to be triggered by theCluster Autoscaler and the pod would be Pending until the new node isprovisioned and attached to the cluster.

We can save a lot of time from this process since we have the overprovisionedpods with a lower pod priority. The newly created deployment would have a podpriority of 0 by default, and our placeholder pause pods with a priority of-1 would be evicted in favor of this application pod. This means that new podscan be scheduled without having to wait for the Cluster Autoscaler to do itsmagic. Some space would always be reserved in our cluster.

The pause pods would now move to Pending state after being evicted, which theCluster Autoscaler will pick up, and it will start provisioning a new node. Thisway, the autoscaler doesnt have to wait for a user to create a new deploymentto scale up the cluster. Instead, it would do it in advance to reserve somespace for potential deployments.

Trade-offs

Now that we have the overprovisioning deployments configured to reserve somespace in NeetoDeploy's cluster, we had to decide how much space to reserve. Ifwe increase the CPU and memory limit for the overprovisioning pods or theirnumber of replicas, we will have more space reserved in our cluster. This meansthat we can handle more user deployments concurrently, but we will incur thecost of keeping the extra buffer running. The trade-off here is between the costwe are willing to pay and the load we want to handle.

For running NeetoDeploy, we started with three copies of overprovisioning podswith 500 milli vCPUs, and later scaled it to 10 replicas after we moved thereview apps of all the Neeto products to NeetoDeploy. Wehave been running all our internal staging deployments and review apps for thepast eleven months, and this configuration has been working pretty well for usso far.

If your application runs on Heroku, you can deploy it on NeetoDeploy without anychange. If you want to give NeetoDeploy a try, then please send us an email atinvite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, followNeetoDeploy on Twitter. You can also join ourcommunity Slack to chat with us about anyNeeto product.

We started using NeetoDeploy internally for all our staging applications and the"pull request review" applications. Very soon, we ran into scaling challenges.Some days, we have too many pull requests, and some days, very few pullrequests. Some days, the staging sites have an extra load.

Scaling is a fundamental problem in container orchestration. We need to ensurethat we have enough computing resources to handle the load, and at the sametime, we need to make sure we are not spending money on resources that are notbeing utilized. If we need to run 10 applications, we should only have to payfor as much computing as is required for running 10 applications. But if thisnumber increases to 100 one day, our system should be able to provision newcomputing resources.

Understanding Kubernetes Autoscalers

When we need to scale, we can manually allocate resources. This istime-consuming and repetitive. Kubernetes excels in autoscaling. It providesmany different options to meet our scaling needs.

We can define autoscalers of different types to make our cluster and ourdeployments scale up or down based on various parameters and handle trafficgracefully. Kubernetes has three kinds of autoscalers operating at differentlevels.

Horizontal Pod Autoscaler (HPA)

The Horizontal Pod Autoscaler can scale up our Kubernetes deployments byincreasing the number of copies of our app's container, known asreplicas.We can configure our Horizontal Pod Autoscaler to scale up our apps based onmetrics like increased memory usage, network load, or even the time of the daywhen we expect more traffic. Incoming traffic would be redirected to any of thereplicas and Kubernetes takes care of this.

Vertical Pod Autoscaler (VPA)

The Vertical Pod Autoscaler takes a different approach to the same problemsolved by the Horizontal Pod Autoscaler. The Vertical Pod Autoscaler increasesthe CPU and memory resources allocated to our pod based on the load. The numberof containers running would remain the same, but they would have more resourcesto work with and, hence, would be able to handle increased loads.

Cluster Autoscaler

While horizontal and vertical pod autoscaling operate on the level of individualdeployments, the Cluster Autoscaler scales in the context of the entire cluster.It increases or decreases the number of nodes in the cluster, making space formore deployments.

The need for Cluster Autoscaler

Suppose our Kubernetes cluster has three nodes. When we create deployments, ourapplications will be deployed to any of these three nodes based on the resourceavailability in the nodes. Without the Cluster Autoscaler, the number ofmachines running in our cluster will be fixed at 3.

If there is a sudden surge in traffic and we have to deploy 100 apps instead of10, the resources needed would overflow the three nodes' combined CPU and memoryresources. Essentially, our cluster will not have enough resources toaccommodate all the deployments, and the pods for our deployments will be stuckin the Pending state.

How Cluster Autoscaler works

The Cluster Autoscaler increases and decreases the number of nodes in ourcluster based on the number of deployments created. So in the above example, ifwe had the Cluster Autoscaler running, it would check if any pods are stuck in aPending state, and if so, it'll spawn a new node in the cluster. If threenodes are not enough to accommodate all our deployments, the Cluster Autoscalerwould detect this and scale the cluster to four nodes.

Let us see step by step how cluster autoscaler would work:

1. If our Kubernetes cluster is running at full capacity, the pods of a newlycreated deployment would have no space to fit in the cluster. In this case,the pod would be stuck in the Pending state and wouldn't be "scheduled" toany node.

2. The Cluster Autoscaler keeps checking if there are unscheduled pods. If so,the Cluster Autoscaler triggers a scale-up by provisioning a new node andattaching it to the cluster.

3. Once the new node has been provisioned, the capacity of the cluster hasincreased, and the pod, which was previously in the Pending state, will bescheduled to the newly created node.

The same thing can happen in reverse as well. Consider the case wheredeployments are deleted or scaled down, and our nodes are not utilizedcompletely.

In such a scenario, the Cluster Autoscaler would scale down the nodes so thatonly the minimum required resources are running. Existing deployments fromdifferent nodes would be rescheduled and "packed" into a smaller number of nodesbefore the unwanted machines are terminated. The Cluster Autoscaler marks unusednodes as SchedulingDisabled so that no pods are scheduled or moved into thisnode.

The Cluster Autoscaler then unprovisions the unused node.

The Cluster Autoscaler ensures that the cluster is in a stable state where nopods are stuck in the Pending state, and no under-utilized resources are keptrunning. By default, the Cluster Autoscaler checks for pending pods and scalesdown candidates every 10 seconds, but we can configure this to increase ordecrease the speed at which the cluster autoscales.

Deploying the Cluster Autoscaler

The code for Cluster Autoscaler is available atkubernetes/autoscaler, and this canbe used to set up Cluster Autoscaler with any Kubernetes cluster.

All major cloud platforms like AWS, GCP, Azure, and DigitalOcean have ClusterAutoscaler support in their managed Kubernetes services. For example, if theCluster Autoscaler is deployed on Amazons Elastic Kubernetes Service (EKS). Itwould spawn new EC2 instances as needed and attach them to the EKS clusterscontrol plane. The complete list of all cloud providers ishere.

Since we are running NeetoDeploy on EKS, we referred to the documentation forsetting up autoscaler with EKS and created an OIDC identity provider in IAM. Wethen created an IAM policy for using the autoscaler with the necessarypermissions, such as DescribeAutoScalingGroups for describing autoscalinggroups in AWS and setDesiredCapacity to change the desired limit of nodes inthe cluster, etc.

We then created an IAM role with the IAM policy and OIDC provider createdearlier and used this role to set up RBAC in the cluster, along with the ClusterAutoscaler deployment. This ensures that the Cluster Autoscaler deployment hasall the permissions it needs for inspecting pending pods and working with nodes.

Seeing the autoscaler in action

Once all of this was done, we tried creating multiple deployments with moresignificant memory and CPU limits consecutively to see if the cluster couldhandle the increased load. Even though the autoscaler took some time to react tothe increased load, the cluster was getting scaled up according to therequirement, and none of the pods were stuck in the Pending state for long.Once we deleted our test deployments, the cluster automatically scaled backdown.

If your application runs on Heroku, you can deploy it on NeetoDeploy without anychange. If you want to give NeetoDeploy a try, then please send us an email atinvite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, followNeetoDeploy on Twitter. You can also join ourcommunity Slack to chat with us about anyNeeto product.

NeetoDeploy's PostgreSQL add-on has a continuous backup feature. This featureallows you to roll your database back to a specific time in the past. We cansplit the continuous backup process into data backup and recovery. Let's discussthese steps in detail.

Initial setup

Let us create a new PostgreSQL cluster and populate it with some data.

Creating parent database

initdb /usr/local/var/postgresql-parent

Creating some sample data

Start the PostgreSQL server and log in using psql. Provide the role and password ifneeded.

pg_ctl -D /usr/local/var/postgresql-parent startpsql -d postgres

Create a new table

postgres=#CREATE TABLE customers (  user_id SERIAL PRIMARY KEY,  username VARCHAR(50),  email VARCHAR(100),  registration_date DATE);

Seed sample data

postgres=#INSERT INTO customers  (username, email, registration_date)VALUES('johndoe', 'johndoe@example.com', '2021-01-01'),('janedoe', 'janedoe@example.com', '2021-02-15'),('bobsmith', 'bobsmith@example.com', '2021-03-10'),('sarahlee', 'sarahlee@example.com', '2021-04-05'),('maxwell', 'maxwell@example.com', '2021-05-20');

Data backup

To perform recovery, we need to have access to the existing data. Hence we muststore all the database changes in a separate location, allowing us to utilizethis data for recovery.

This is where theWrite Ahead Log (WAL)in PostgreSQL comes into play. PostgreSQL follows a mechanism that writes alltransactions to the WAL before committing them to the table data files. Bypreserving the WAL, we can reconstruct the data by traversing the loggedtransactions. Utilizing the information stored in the WAL, we can systematicallyreapply the recorded transactions to recreate the data and achieve the desiredrecovery. Your WAL files are in your PostgreSQL data directory's pg_walfolder.

WAL archiving

As data modifications occur in the databases, new WAL files are generated whileolder WAL files are eventually discarded. Therefore, it is crucial to store theWAL files before they are deleted. Here we will use anAWS S3 bucket to store our WAL files. Also, wewill use wal-e andenvdir to simplify the process.

Install helper modules

python3 -m pip install wal-e envdir

If you face any dependency issues, please referwal-e to fix them.

Store AWS S3 credentials

To use "envdir" with your AWS credentials, store them in a folder. The followingcode snippet demonstrates how to accomplish this:

mkdir ~/wal-e.envecho "YOUR_AWS_REGION" > ~/wal-e.env/AWS_REGIONecho "YOUR_AWS_SECRET_ACCESS_KEY" > ~/wal-e.env/AWS_SECRET_ACCESS_KEYecho "YOUR_AWS_ACCESS_KEY_ID" > ~/wal-e.env/AWS_ACCESS_KEY_IDecho "YOUR_AWS_STORAGE_PATH" > ~/wal-e.env/WALE_S3_PREFIX

Configure WAL archiving

Now, let's configure PostgreSQL to store the WAL files in your S3 bucket beforethey are deleted. Open your PostgreSQL configuration file "postgresql.conf",which is located in your data directory i.e.,"/usr/local/var/postgresql-parent", and make the following changes.

# To enable WAL archiving.archive_mode = on# Determines how much information is written to the WAL.wal_level = replica# Force PostgreSQL to switch WAL file in every 60 secondsarchive_timeout = 60# Command for pushing the wal files to the S3 bucket.archive_command = 'envdir ~/wal-e.env wal-e wal-push %p'

You can find more information about the above configurations by referring tothisdocumentation.

Now let's restart the server.

 data_directory="/usr/local/var/postgresql-parent" pg_ctl -D $data_directory restart -l $data_directory/postgresql.log

You can watch the PostgreSQL logs like this.

 tail -f /usr/local/var/postgresql-parent/postgresql.log

After starting your server, monitor the PostgreSQL logs. If you don't observeany WAL archive logs even after 60 seconds, it is because a WAL file switch willonly occur if there are any modifications. To generate the desired outcome, makeupdates in the database. Upon doing so, you will be able to see lines similar tothe following in your PostgreSQL logs:

Message: 'MSG: completed archiving to a file\nDETAIL: Archiving to "s3://neeto-deploy-backups/local/wal_005/000000020000000000001.lzo" complete at 158.209KiB/s.\nSTRUCTURED: time=2023-05-18T06:22:27.613242-00 pid=263 action=push-wal key=s3://neeto-deploy-backups/local/wal_005/000000020000000000001.lzo prefix=local/ rate=158.209 seg=000000020000000000001 state=complete'

Base backup

Since we have enabled WAL archiving, all new changes are stored in the bucket asWAL files. However, what about the data we created before configuring WALarchiving? To ensure a complete recovery, we also need that data. Therefore, weneed to take a base backup of our database. The following code will initiate thebase backup process and push it to the S3 bucket.

envdir ~/wal-e.env wal-e backup-push /usr/local/var/postgresql-parent

Running the above command yields the following output.

wal_e.main   INFO     MSG: starting WAL-EDETAIL: The subcommand is "backup-push".STRUCTURED: time=2023-05-18T06:24:27.613242-00 pid=41178wal_e.worker.upload INFO     MSG: begin uploading a base backup volumeDETAIL: Uploading to "s3://neeto-deploy-backups/local/basebackups_005/base_000000010000000000005_00000040/tar_partitions/part_00000000.tar.lzo".STRUCTURED: time=2023-05-18T06:24:29.527622-00 pid=41178wal_e.worker.upload INFO     MSG: finish uploading a base backup volumeDETAIL: Uploading to "s3://neeto-deploy-backups/local/basebackups_005/base_000000010000000000005_00000040/tar_partitions/part_00000000.tar.lzo"complete at 1459.68KiB/s.STRUCTURED: time=2023-05-18T06:24:33.121716-00 pid=41178NOTICE:  all required WAL segments have been archived

After completing the base backup, you can verify the presence of the base backupfolder in your S3 bucket. Now we have the backup and new WAL files are beingpushed to our bucket.

Data recovery

Let's consider a scenario where you need to perform a rollback. Suppose youaccidentally deleted a table from your database that contained multiple records,making it impractical to recall and reinsert them manually. As a first step,make a note of the time at which the table was deleted. For instance, let's sayit occurred on 2023-05-18 11:57:11 UTC. The objective is to roll back to a pointbefore that time. Let's roll back to 2023-05-18 at 11:57:00 UTC.

Let's make a mistake

We are going to drop a table.

postgres=#DROP TABLE customers;

Note the time the table got deleted. Here it is 2023-05-18 11:57:11 UTC. Nowlet's discuss how we can restore the deleted table.

Creating a new data directory

Let's create a new data directory where we can recover the data.

initdb /usr/local/var/postgresql-child

Fetch the base backup

The base backup serves as the starting point, while the remaining data will beregenerated by PostgreSQL using the WAL files. To fetch the base backup, you canuse the following code snippet.

envdir ~/wal-e.env wal-e backup-fetch /usr/local/var/postgresql-child LATEST

The "LATEST" tag will pull the latest base backup if multiple base backups arein the bucket. On running the above command you will receive an output similarto the following.

wal_e.main   INFO     MSG: starting WAL-EDETAIL: The subcommand is "backup-fetch".STRUCTURED: time=2023-05-18T06:29:22.826910-00 pid=41789wal_e.worker.s3.s3_worker INFO     MSG: beginning partition downloadDETAIL: The partition being downloaded is part_00000000.tar.lzo.HINT: The absolute S3 key is local/basebackups_005/base_000000010000000000000005_00000040/tar_partitions/part_00000000.tar.lzo.STRUCTURED: time=2023-05-18T06:29:24.674572-00 pid=41789

Configuring recovery

Now, we need to regenerate the remaining data from the WAL files. To do that,make the following changes in the PostgreSQL configuration file.

#postgresql.conf# Command to fetch each wal files from the bucket.restore_command = 'envdir ~/wal-e.env wal-e wal-fetch %f %p'# Time at which we want to stop recovery.recovery_target_time  = '2023-05-18 11:57:00'

We can stop the parent server since it is no longer needed.

pg_ctl -D /usr/local/var/postgresql-parent stop

To initiate the server in recovery mode and to allow PostgreSQL to regeneratethe data using the WAL files, create a file named "recovery.signal" in the newdata directory.

touch /usr/local/var/postgresql-child/recovery.signal

Start the child server pointing to the new data directory by running thefollowing command.

 data_directory="/usr/local/var/postgresql-child" pg_ctl -D $data_directory start -l $data_directory/postgresql.log

If you examine the PostgreSQL logs, you will notice that the recovery process ispaused at the specified time.

2023-05-18 12:07:41.688 IST [43159] LOG:  restored log file "000000010000000000006" from archive2023-05-18 12:07:41.692 IST [43159] LOG:  recovery stopping before commit of transaction 743, time 2023-05-18 11:57:11.135927+05:302023-05-18 12:07:41.692 IST [43159] LOG:  pausing at the end of recovery2023-05-18 12:07:41.692 IST [43159] HINT:  Execute pg_wal_replay_resume() to promote.

It's important to note that you won't be able to connect to the server usingpsql at this point because it is not ready to accept connections. By default,PostgreSQL pauses the recovery process after reaching the recovery target time.To change this behavior and to allow the server to accept connections afterreaching the target time, add the following configuration to your PostgreSQLconfiguration file.

recovery_target_action  = 'promote'

After making the above change, restart the server once again by running thefollowing command.

data_directory="/usr/local/var/postgresql-child"pg_ctl -D $data_directory restart -l $data_directory/postgresql.log

Now, you will observe that the server is ready to accept connections aftercompleting the recovery process. Once the recovery is successfully completed,PostgreSQL automatically deletes the "recovery.signal" file. You can verify thesuccessful deletion of the recovery file from your data directory.

Once the server is up and ready, connect to the server and check if the deletedtable is present. You will find that the deleted table is now restored andavailable in the database.

postgres=#select * from customers; user_id | username |        email         | registration_date---------+----------+----------------------+-------------------       1 | johndoe  | johndoe@example.com  | 2021-01-01       2 | janedoe  | janedoe@example.com  | 2021-02-15       3 | bobsmith | bobsmith@example.com | 2021-03-10       4 | sarahlee | sarahlee@example.com | 2021-04-05       5 | maxwell  | maxwell@example.com  | 2021-05-20(5 rows)

To learn more about PostgreSQL continuous backup, you can refer to theofficial PostgreSQL documentation.

Challenges faced

Restoration of PostgreSQL roles

When we restored the PostgreSQL database, the roles and passwords were alsorestored in the database. However, we create a new database with the desireddata in the continuous rollback feature. Therefore, we should use a new role andpassword to ensure security. We resolved this issue by changing the restoredrole and password to a new one.

Early recovery completion

Another issue we encountered was the early recovery of the database beforereaching the specified target time. Suppose no activity occurs in the database,and we attempt to roll back to the current time. PostgreSQL initiates therecovery process and recovers all the available WAL files. However, it fails tolocate the target time since no corresponding WAL file is available for thegiven timestamp. PostgreSQL raises the following error in this scenario andshuts down the server.

2023-05-18 02:12:15.846 UTC [76] LOG:  last completed transactionwas at log time 2023-05-18 02:04:02.755919+002023-05-18 02:12:15.846 UTC [76] FATAL:  recovery ended before configuredrecovery target was reached2023-05-18 02:12:15.850 UTC [75] LOG:  startup process (PID 76) exited withexit code 12023-05-18 02:12:15.850 UTC [75] LOG:  terminating any other active serverprocesses2023-05-18 02:12:15.853 UTC [75] LOG:  shutting down due to startupprocess failure2023-05-18 02:12:15.859 UTC [75] LOG:  database system is shut down

In this case, we can remove the specified target time and restart the server. Ifwe do not provide any target time, PostgreSQL will recover until the lastavailable WAL file, which will contain the latest transaction of our database.

Recovery is an asynchronous process

As mentioned above, we must update the restored role and password to new ones.

update-role-and-password(){# Create a new temporary user as we cannot update the current session user.psql -d <DB_NAME> \-c "CREATE USER tempuser WITH SUPERUSER LOGIN PASSWORD 'tempuser-pwd';"# Login as a temporary user and update the parent role and password.PGPASSWORD=tempuser  PGUSER=tempuser-password  psql -d <DB_NAME> \-c "ALTER ROLE <parent-role> RENAME TO <NEW_ROLE>;ALTER ROLE <NEW_ROLE> WITH PASSWORD <NEW_PASSWORD>;"# Login as the new user and remove the temporary user.PGPASSWORD=<NEW_ROLE> PGUSER=<NEW_PASSWORD> psql -d <DB_NAME> \-c "DROP ROLE tempuser;"}# Start the server in recovery mode.pg_ctl -D /usr/local/var/postgresql-child start# Call the method to update role and password.update-role-and-password

If you try the above code, you will encounter the following error.

ERROR:  cannot execute CREATE ROLE in a read-only transaction

Since the recovery process is asynchronous, the system may attempt to update therole before completing the recovery. It is important to note that the serverwill be in read-only mode during the recovery, which can lead to the errormentioned.

To solve this issue, it is necessary to determine when the recovery process wascompleted. This can be achieved by checking for the existence of the "recovery.signal" file in a loop.

wait_until_recovery_is_completed(){  recovery_signal_path="/usr/local/var/postgresql-child/recovery.signal"  while [ -f "$file_path" ]; do    sleep 1  # Wait for 1 second before checking again.  done}

pg_ctl -D /usr/local/var/postgresql-child startwait_until_recovery_is_completedupdate-role-and-password

How it all started

At Neeto, we are building20+ productssimultaneously. All of these products had their production and stagingenvironments hosted on Heroku. In addition, pull requests from these productshad review apps deployed on Heroku. With an average of 5 open PRs at any timeper product, we are talking about 100+ live review apps, 20+ staging apps, and15+ production apps. Each one of these apps needed PostgreSQL, Redis, andElasticSearch instances, besides other requirements.

In August 2022, Heroku made thisannouncement.

"Starting November 28, 2022, we plan to stop offering free product plans andplan to start shutting down free dynos and data services."

It meant that in 2 months, our monthly Heroku bill would skyrocket. At first, wetalked about switching to one of the Heroku alternatives.

"Every problem is an opportunity in disguise." - John Adams.

And we thought - "Why not build our own Heroku replacement?".

BigBinary has over 11 years of Ruby on Rails and Reactconsulting expertise. However, building a Heroku replacement, a complex TheDevOps-heavy project was way out of our comfort zone. With faith in ourengineering team and an exuberant spirit of youthful adventure, we embarked on ajourney in early September 2022 to buildNeetoDeploy.

The Journey

We put together a team of 4 developers. We discussed technologies ranging fromCapistrano, Ansible, and AWS to Kubernetes. The more we talked, the more itbecame clear that no single technology could solve this problem. We needed acomplex system with multiple independent components that would interact togetherto handle the following core responsibilities:

1. GitHub integration.
2. A build system that would identify the application's run-time dependencies,install them in the correct order, and build runnable machine images.
3. A scalable containerized environment that can run the pre-built machineimages.
4. A place to store the built machine images.
5. If the application needed a connection to external services (PostgreSQL,Redis, ElasticSearch, etc.), the ability to provision those on run-time.
6. A dashboard to view and manage the deployments.
7. Live logs.
8. Load balancing/routing.

The Architecture

Once we understood the core functionality, we devised an architecture for thesystem. We then drew an architecture diagram depicting the different componentsand how they would interact with each other.

This diagram helped facilitate the rest of the discussions and our questionsregarding the project's feasibility. We looked at the diagram and saw for thefirst time that we could break up this complex beast into manageable pieces thatwe could build, test and deploy independently.

As per the plan, NeetoDeploy would comprise the following independentcomponents:

1. Slug Compiler
2. Dyno Manager
3. Add-on Manager
4. The Compute Cluster - with the router and load balancer
5. Web Dashboard

The Slug Compiler

This component would bundle the application code, its run-time dependencies(Ruby, Node, Ruby gems, NPM packages, third-party tools like ffmpeg, etc.), andthe operating system into a single image. We could then run this image on acontainer.

Using Docker to build the image was the first approach that we tried. Afteranalyzing the problem, we understood that we needed to generate a Dockerfile onthe fly. While working on this problem, we stumbled uponCloud Native Buildpacks.

Their website said, "transform your application source code into images that canrun on any cloud". This is precisely what we needed.

CNB(Cloud Native Buildpacks) system has three components:

1. Stack
2. Builder
3. Buildpacks

CNB provides pack utility that wouldaccept stack, builder, and buildpacks as parameters and build an OCI-compliantmachine image that works with existing Docker tooling and any broader containerecosystem like Kubernetes. In addition, Heroku had open-sourced their olderstacks and builders.

Now we had everything we needed to build runnable images. But we needed theability to build multiple images in parallel since NeetoDeploy would havevarious projects, and each project could have numerous deployments (appcorresponding to the main branch for staging/production and review apps for eachcommit in a pull request).

We built the initial version of Slug Compiler as a Ruby on Rails application,exposing API endpoints to start and stop a build. It would need a GitHubrepository URL to fetch the source code, a list of buildpacks, and a callbackURL as inputs. Once the image is successfully built, it will make a POST requestto the callback URL.

The slug compiler would spin up an EC2 machine for each build request. We made acustom AMI from an Ubuntu image with pack and other packages required for theCNB build system to run. Once the pack command builds the image, it will pushit to our Dockerhub account (Docker registry) and return the image URL via acallback. It would then kill the EC2 instance.

We found uploading the image to Dockerhub very slow.So we set up a private Docker registry onAWS ECS.

The Add-on Manager

Applications typically need access to a database, search service, cache service,etc. The Neeto products use PostgreSQL, Redis, and ElasticSearch. We decided tobuild the first version of the add-on manager to support these add-ons.

We built the add-on infrastructure using three bigAWS EC2 boxes to keep things simple. Each machineran a PG, Redis, and ElasticSearch server, respectively. We made the add-onmanager as a Ruby on Rails API server, which would accept the add-on type asinput and return a connection URI with login credentials included. We wouldsupport only a fixed version of the add-ons in the initial version.

Upon receiving the request for a PG add-on, we will create a new database on thePG server with a unique username/password and return the connection string. ForRedis and ElasticSearch, we simulated separate connections using namespaces.

This add-on manager had some severe drawbacks since multiple applications sharedthe same add-on servers running on the same physical machine. Two of the biggestdrawbacks had to do with Security and Resource choking.

1. Security: With some effort, it was possible to tamper with the data of otherapplications.
2. Resource choking: If one application uses more CPU and Memory, it blocks allother applications.

Nevertheless, this throw-away version of the add-on manager was a good start.

The Dyno manager and the compute cluster

We needed a secure, scalable, containerized environment to run our pre-builtimages. Kubernetes was the obvious answer. Kubernetes,also known as K8s, is an open-source system for automating deployment, scaling,and managing containerized applications. We created our cluster onAWS EKS.

Dyno manager was once again a Ruby on Rails API server app, which takes apre-built OCI image URL and a callback URL as inputs. It would then interfacewith the Kubernetes cluster using thekubectl CLI to deploy the imageon the cluster.

Our first cluster was of fixed size. It had four nodes (EC2 machines).Kubernetes would deploy images to these machines and manage them. But thefixed-size cluster soon proved inadequate as we had more parallel deploymentsthan could be accommodated. So we set up cluster autoscaling.

Load balancing and routing

All applications hosted on NeetoDeploy are made publicly available on a_.neetoreviewapp.com subdomain. A public IP address is needed to set up aDNS CNAME recordfor _.neetoreviewapp.com. We provisioned anElastic Load Balancer, an AWScomponent with a public IP address. ELB is the component of our system that actsas the interface to the outside world.

We now needed to route the incoming web requests based on the subdomain to oneof the deployed applications, which required a programmable router. AnNginx ingress controllerwould meet this requirement. The Nginx ingress controller is essentially anNginx server set up as a reverse proxy with the redirect rules necessary.

The Web Dashboard

NeetoDeploy needed an admin user-facing web application that would perform thefollowing functions:

1. Admin user login.
2. Initiate Github account integration.
3. Select a repo and create a project.
4. View and manage the deployments for a project.
5. Interact with other components (Slug compiler, dyno manager, and add-onmanager) to coordinate and orchestrate the functioning of the NeetoDeploysystem.
6. Maintain state like store project-specific environment variables, add-onconnection URLs, pre-built image URLs, etc.

We built the Dashboard app using Ruby on Rails, React.js,neetoUI, andother integrations typical of any Neeto product.

Live logging

We needed to fetch and display two types of logs at run-time:

• Build and release logs.
• Application logs.

Fetching live logs from a remote machine and displaying them in the browser is acomplicated problem. We went withServer Side Events(SSE) for the first version and streamed the logs directly to the browser.

1. The dyno manager would tail kubectl logs.
2. STDOUT was continuously read and transmitted as server-side events.
3. The dyno manager sent the app-logs-url to the dashboard app via a callback.
4. Dashboard app had a log-display React component that streamed the SSE eventsusing the app-logs-url and displayed the live logs.

How it all came together

1. The Admin user signed up for a new account.
2. The admin logged in to the Dashboard app.
3. The GitHub account was connected by the admin.
4. The admin created a new project after selecting a GitHub repository.
5. The admin made a new Pull Request in the selected GitHub repository.
6. The dashboard app received the GitHub event for the new pull requestcreation.
7. The dashboard app requested the slug compiler to build an image for therepository using code corresponding to the pull request's latest commit. Italso sent the list of buildpacks to use.
8. The dashboard app also requested the add-on manager to create the necessaryadd-ons.
9. The dashboard app received the add-on URLs and saved them.
10. Slug compiler built the image, stored it on ECS and made a callback to thedashboard app with the image URL, and the dashboard app saved the image URL.
11. The dashboard app requested the dyno manager to deploy and start thenecessary web and worker processes after passing the image URL and add-onURLs as environment variables to set on the containers.
12. The Dyno manager talked to Kubernetes, completed the specified clusterdeployments and set up the necessary ingress rules.
13. The application is available for public access athttps://xyz.neetoreviewapp.com.

Roadmap

We improved each component to make them scalable, secure and robust. We addedfeatures like CLI, performance metrics, continuous database backup, enabledcaching, reduced build time, horizontal scaling to support the increased load,platform extensibility with custom build packs, stabilized live logging withFluentbit and more.

We migrated the review apps and staging environment of allneeto products to NeetoDeploy. We also migrated productioninstances of most of our static websites. We are now gearing up to migrate theproduction instances of all the Neeto products. Exciting features like customplans for dynos, support for more add-on types, autoscaling to supportunexpected surges in traffic and more, are in the pipeline.

In the coming weeks and months, we will be writing more about the challenges wefaced and how we are building NeetoDeploy. Stay tuned for more.

If your application runs on Heroku, you can deploy it on NeetoDeploy without anychange. If you want to give NeetoDeploy a try, then please send us an email atinvite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, followNeetoDeploy on Twitter. You can also join ourcommunity Slack to chat with us about anyNeeto product.