Building the CLI tool for NeetoDeploy became our top priority after we had built all the basic features in NeetoDeploy. Once we started migrating our apps from Heroku to NeetoDeploy, the need for a CLI tool arose, since previously developers were using Heroku CLI for their scripts etc.

We started building the CLI as a Ruby gem using Thor. Installing the gem and using it would be as simple as running gem install neetodeploy. We wanted the users to be authenticated via Neeto's authentication system before they could do anything else with the CLI. For this, we added a neetodeploy login command, which would create a session and redirect you to the browser where you can log in. The CLI would be polling the session login status, and once you have logged in, it would store the session token and your email address inside the ~/.config/neetodeploy directory. This will be used to authenticate your request when you run everything else.

The basic CLI functionality

The v1 release of the NeetoDeploy CLI shipped the following commands:

• config : For listing, creating, and deleting environment variables for your apps.
• exec : For getting access to a shell running inside your deployed app.
• logs : For streaming the application logs right to your terminal.

Let's look at each of them in detail.

Setting environment vars

Building the config command was the most straightforward of the three. The central source or truth for all the environment variables of the apps is stored in the web dashboard of NeetoDeploy. There are APIs already in place for doing CRUD operations on these from the dashboard app. This could be expanded for the use of the CLI as well.

We first added API endpoints in the dashboard app, under a CLI namespace, to list, create, and delete config variables. We then added different commands in the CLI to send HTTP requests to the respective endpoints in the dashboard app.

You can pass the app name as an argument, and the CLI will send a request to the dashboard app along with the session token generated when you logged in. At the backend, this session token would be used to check if you have access to the app you're requesting, before doing anything else.

If you have used Heroku CLI, then you would be very comfortable since NeetoDeploy CLI follows the same style.

Architecting the neetodeploy exec command

The exec command is used to get access to a shell inside your deployed app. The NeetoDeploy web dashboard already has a functioning console feature with which you can get access to the shell inside your application’s dynos. This was built using websockets. We would deploy a "console" pod with the same image and environment variables as the app, to which we'll connect with websockets. We have a dyno-console-manager app which spawns a new process inside the console dyno and handles the websocket connections, etc. When we started building the exec feature for the CLI however, we explored a few options and learnt a lot of things before finally settling back on the websockets approach.

We’ve been using the kubectl exec command on a daily basis to get access to shells inside pods running in the cluster. Since kubectl exec is cosmetically similar to how we use SSH to access remote machines, the first approach we took involved using SSH to expose containers outside the cluster.

We can expose our deployment outside the cluster using SSH in two ways. The first approach would be running an SSH server as a sidecar container within the console pod. The second approach would be to bundle sshd within the image for the console pod using a custom buildpack. With either of these two methods, you would be able to SSH into your console deployment if you have the correct SSH keys. We ran sshd inside a pod using a custom sshd buildpack. We then configured the SSH public key inside the pod manually with kubectl exec. With this, we were able to SSH into the pod after port forwarding the sshd process’s port to localhost using kubectl port-forward.

Great! Now that the SSH connection works, the part of the puzzle that remained now was how we would actually expose the deployment outside the cluster in production. We tried doing this in a couple of different ways.

1. LoadBalancer

We were able to SSH into the pod after exposing it as a LoadBalancer service. This was out of the question for production however, since AWS has a hard limit on the number of application load balancers (ALB) we can create per region. We’d also have to pay for each load balancer we create. This approach wouldn’t scale at all.

2. NodePort

The next option was to expose the pod as a NodePort service. A NodePort service in Kubernetes means that the pod would be exposed through a certain port number through every node in the cluster. We could SSH into the pod with the external IP of any of the nodes. We tried exposing the test console pod we created with a NodePort service as well. We were able to SSH into the pod from outside the cluster without having to do the port forward. There was one limitation for using NodePorts that we knew would render this method suboptimal for building a Platform as a Service. The range of ports that can be allocated for NodePort services is from 30000 to 32768 by default. This means that we’d only be able to run 2768 instances of neetodeploy exec at any given time in our current cluster setup in EKS.

3. Bastion host, the SSH proxy server

We thought of deploying an SSH proxy server to circumvent the hard limits on the LoadBalancer and NodePort approaches. Such proxy servers are usually called bastion hosts or jump servers. These are servers that are designed to provide access to a private network from a public network. The name "bastion" comes from the military structure that projects outwards from a castle or a fort. In a similar sense, our bastion host will be a Kubernetes deployment that is exposed to the public network and acts as an interface between our private cluster and the public internet. For our requirement, we can proxy SSH connections to console pods through this bastion host deployment. We only needed to expose the bastion host as a single LoadBalancer or a NodePort service, to which the CLI can connect. This would solve the issue of having a hard limit on the number of LoadBalancer or NodePort services we can create.

We quickly set up a bastion host and tested the whole idea. It was working seamlessly! However, there were a lot of edge cases we'd have to handle if we are going to release this to the public. Ideally, we would want a new pair of SSH keys for each console pod. We could generate this and store it as secrets in the cluster. Apart from the hassle of handling all the SSH keys and updating the bastion host each time, we would have to make sure that users will not be able to get SSH access to the bastion host deployment manually, outside the CLI. This was difficult since the CLI would need access to the private key in order to connect to the bastion host anyway. This would mean that users could SSH into the bastion host after digging through the CLI gem's source, and possibly even exec into any deployment in the cluster if they knew what they were looking for. Users with malicious intent could possibly bring down the bastion host itself once they're inside, rendering the exec command unusable.

We thought about installing kubectl inside the bastion host with a restrictive RBAC so that we can run kubectl exec from inside the bastion host, also while making sure that users could not run anything destructive from inside the bastion host. But this just adds more moving parts to the system. The SSH proxy approach makes sense if you have to expose your cluster among developers for internal use, but it is not ideal when you are building a CLI for public use.

After figuring out how not to solve the problem, we came to the conclusion that using websockets was the better approach for now. We already have the dyno-console-manager app, which we are using for the shell access from the web dashboard, we updated the same to handle connections from the CLI. From the CLI's end, we wrote a simple WebSocket client and handled the parsing of user inputs and printing the responses from the shell. All of the commands you enter would be run on the console pod through the websocket connection.

Streaming logs to the terminal

Live logs were streamed to the web dashboard using websockets too. This was also handled by the above-mentioned dyno-console-manager. Since we had written a Ruby websocket client for the exec command, we decided to use the same approach with logs too.

The CLI would send a request to the dyno-console-manager with the app’s name and the dyno-console-manager would run kubectl logs for your app's deployment and stream it back to the CLI via a websocket connection it creates.

We took our time with architecting the NeetoDeploy CLI, but thanks to that we learnt a lot about designing robust systems that would scale in the long term.

If your application runs on Heroku, you can deploy it on NeetoDeploy without any change. If you want to give NeetoDeploy a try, then please send us an email at invite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, follow NeetoDeploy on X. You can also join our community Slack to chat with us about any Neeto product.

Challenges faced

The Cluster Autoscaler works by monitoring the pods and checking if they are waiting to get scheduled. If no nodes are available to schedule the waiting pods, they will be in Pending state, until the kube-scheduler finds a suitable node.

With our default Cluster Autoscaler setup, if a pod is provisioned when the cluster is at maximum capacity, the pod will go into Pending state and remain there until the cluster has scaled up. This entire flow consists of the following steps:

• The Cluster Autoscaler scans the pods and identifies the un-schedulable pods.
• The Cluster Autoscaler provisions a new node. We have configured the autoscaler deployment with the necessary permissions for the autoscaling groups in EC2, with which the autoscaler can provision new EC2 machines.
• Kubernetes adds the newly provisioned node to the cluster’s control plane, and the cluster is “scaled up.”
• The pending pod gets scheduled to the newly added node.

From the perspective of a user who wants to get their application deployed, a considerable amount of time is taken to complete all these steps in addition to the time taken to build and release their code. We can only optimize it so much by decreasing the scan interval of the Cluster Autoscaler. We need a mechanism by which we can always have some buffer nodes ready so that the users’ deployments never go into Pending state and wait for the nodes to be provisioned.

Pod Priority, Preemption and Overprovisioning

Kubernetes allows us to assign different priorities to pods. By default, all the pods have a priority of 0. When a pod is scheduled, it can preempt or evict other pods with lower priority than itself and take its place. We will use this feature of pod preemption to reserve resources for incoming deployments.

We created a new pod priority class with a priority of -1 and used this to “overprovision” pause pods. We allocated a fixed amount of CPU and memory resources to these pause pods. The pause container is a placeholder container used by Kubernetes internally and doesn’t do anything by itself.

Now if the cluster is at full capacity and a new deployment is created, technically the newly created pod doesn't have space to be scheduled in the cluster. In normal cases, this would warrant a scale-up to be triggered by the Cluster Autoscaler and the pod would be Pending until the new node is provisioned and attached to the cluster.

We can save a lot of time from this process since we have the overprovisioned pods with a lower pod priority. The newly created deployment would have a pod priority of 0 by default, and our placeholder pause pods with a priority of -1 would be evicted in favor of this application pod. This means that new pods can be scheduled without having to wait for the Cluster Autoscaler to do its magic. Some space would always be reserved in our cluster.

The pause pods would now move to Pending state after being evicted, which the Cluster Autoscaler will pick up, and it will start provisioning a new node. This way, the autoscaler doesn’t have to wait for a user to create a new deployment to scale up the cluster. Instead, it would do it in advance to reserve some space for potential deployments.

Trade-offs

Now that we have the overprovisioning deployments configured to reserve some space in NeetoDeploy's cluster, we had to decide how much space to reserve. If we increase the CPU and memory limit for the overprovisioning pods or their number of replicas, we will have more space reserved in our cluster. This means that we can handle more user deployments concurrently, but we will incur the cost of keeping the extra buffer running. The trade-off here is between the cost we are willing to pay and the load we want to handle.

For running NeetoDeploy, we started with three copies of overprovisioning pods with 500 milli vCPUs, and later scaled it to 10 replicas after we moved the review apps of all the Neeto products to NeetoDeploy. We have been running all our internal staging deployments and review apps for the past eleven months, and this configuration has been working pretty well for us so far.

If your application runs on Heroku, you can deploy it on NeetoDeploy without any change. If you want to give NeetoDeploy a try, then please send us an email at invite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, follow NeetoDeploy on Twitter. You can also join our community Slack to chat with us about any Neeto product.

We started using NeetoDeploy internally for all our staging applications and the "pull request review" applications. Very soon, we ran into scaling challenges. Some days, we have too many pull requests, and some days, very few pull requests. Some days, the staging sites have an extra load.

Scaling is a fundamental problem in container orchestration. We need to ensure that we have enough computing resources to handle the load, and at the same time, we need to make sure we are not spending money on resources that are not being utilized. If we need to run 10 applications, we should only have to pay for as much computing as is required for running 10 applications. But if this number increases to 100 one day, our system should be able to provision new computing resources.

Understanding Kubernetes Autoscalers

When we need to scale, we can manually allocate resources. This is time-consuming and repetitive. Kubernetes excels in autoscaling. It provides many different options to meet our scaling needs.

We can define autoscalers of different types to make our cluster and our deployments scale up or down based on various parameters and handle traffic gracefully. Kubernetes has three kinds of autoscalers operating at different levels.

Horizontal Pod Autoscaler (HPA)

The Horizontal Pod Autoscaler can scale up our Kubernetes deployments by increasing the number of copies of our app's container, known as replicas. We can configure our Horizontal Pod Autoscaler to scale up our apps based on metrics like increased memory usage, network load, or even the time of the day when we expect more traffic. Incoming traffic would be redirected to any of the replicas and Kubernetes takes care of this.

Vertical Pod Autoscaler (VPA)

The Vertical Pod Autoscaler takes a different approach to the same problem solved by the Horizontal Pod Autoscaler. The Vertical Pod Autoscaler increases the CPU and memory resources allocated to our pod based on the load. The number of containers running would remain the same, but they would have more resources to work with and, hence, would be able to handle increased loads.

Cluster Autoscaler

While horizontal and vertical pod autoscaling operate on the level of individual deployments, the Cluster Autoscaler scales in the context of the entire cluster. It increases or decreases the number of nodes in the cluster, making space for more deployments.

The need for Cluster Autoscaler

Suppose our Kubernetes cluster has three nodes. When we create deployments, our applications will be deployed to any of these three nodes based on the resource availability in the nodes. Without the Cluster Autoscaler, the number of machines running in our cluster will be fixed at 3.

If there is a sudden surge in traffic and we have to deploy 100 apps instead of 10, the resources needed would overflow the three nodes' combined CPU and memory resources. Essentially, our cluster will not have enough resources to accommodate all the deployments, and the pods for our deployments will be stuck in the Pending state.

How Cluster Autoscaler works

The Cluster Autoscaler increases and decreases the number of nodes in our cluster based on the number of deployments created. So in the above example, if we had the Cluster Autoscaler running, it would check if any pods are stuck in a Pending state, and if so, it'll spawn a new node in the cluster. If three nodes are not enough to accommodate all our deployments, the Cluster Autoscaler would detect this and scale the cluster to four nodes.

Let us see step by step how cluster autoscaler would work:

1. If our Kubernetes cluster is running at full capacity, the pods of a newly created deployment would have no space to fit in the cluster. In this case, the pod would be stuck in the Pending state and wouldn't be "scheduled" to any node.

2. The Cluster Autoscaler keeps checking if there are unscheduled pods. If so, the Cluster Autoscaler triggers a scale-up by provisioning a new node and attaching it to the cluster.

3. Once the new node has been provisioned, the capacity of the cluster has increased, and the pod, which was previously in the Pending state, will be scheduled to the newly created node.

The same thing can happen in reverse as well. Consider the case where deployments are deleted or scaled down, and our nodes are not utilized completely.

In such a scenario, the Cluster Autoscaler would scale down the nodes so that only the minimum required resources are running. Existing deployments from different nodes would be rescheduled and "packed" into a smaller number of nodes before the unwanted machines are terminated. The Cluster Autoscaler marks unused nodes as SchedulingDisabled so that no pods are scheduled or moved into this node.

The Cluster Autoscaler then unprovisions the unused node.

The Cluster Autoscaler ensures that the cluster is in a stable state where no pods are stuck in the Pending state, and no under-utilized resources are kept running. By default, the Cluster Autoscaler checks for pending pods and scales down candidates every 10 seconds, but we can configure this to increase or decrease the speed at which the cluster autoscales.

Deploying the Cluster Autoscaler

The code for Cluster Autoscaler is available at kubernetes/autoscaler, and this can be used to set up Cluster Autoscaler with any Kubernetes cluster.

All major cloud platforms like AWS, GCP, Azure, and DigitalOcean have Cluster Autoscaler support in their managed Kubernetes services. For example, if the Cluster Autoscaler is deployed on Amazon’s Elastic Kubernetes Service (EKS). It would spawn new EC2 instances as needed and attach them to the EKS cluster’s control plane. The complete list of all cloud providers is here.

Since we are running NeetoDeploy on EKS, we referred to the documentation for setting up autoscaler with EKS and created an OIDC identity provider in IAM. We then created an IAM policy for using the autoscaler with the necessary permissions, such as DescribeAutoScalingGroups for describing autoscaling groups in AWS and setDesiredCapacity to change the desired limit of nodes in the cluster, etc.

We then created an IAM role with the IAM policy and OIDC provider created earlier and used this role to set up RBAC in the cluster, along with the Cluster Autoscaler deployment. This ensures that the Cluster Autoscaler deployment has all the permissions it needs for inspecting pending pods and working with nodes.

Seeing the autoscaler in action

Once all of this was done, we tried creating multiple deployments with more significant memory and CPU limits consecutively to see if the cluster could handle the increased load. Even though the autoscaler took some time to react to the increased load, the cluster was getting scaled up according to the requirement, and none of the pods were stuck in the Pending state for long. Once we deleted our test deployments, the cluster automatically scaled back down.

If your application runs on Heroku, you can deploy it on NeetoDeploy without any change. If you want to give NeetoDeploy a try, then please send us an email at invite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, follow NeetoDeploy on Twitter. You can also join our community Slack to chat with us about any Neeto product.

NeetoDeploy's PostgreSQL add-on has a continuous backup feature. This feature allows you to roll your database back to a specific time in the past. We can split the continuous backup process into data backup and recovery. Let's discuss these steps in detail.

Initial setup

Let us create a new PostgreSQL cluster and populate it with some data.

Creating parent database

initdb /usr/local/var/postgresql-parent

Creating some sample data

Start the PostgreSQL server and log in using psql. Provide the role and password if needed.

pg_ctl -D /usr/local/var/postgresql-parent start
psql -d postgres

Create a new table

postgres=#
CREATE TABLE customers (
  user_id SERIAL PRIMARY KEY,
  username VARCHAR(50),
  email VARCHAR(100),
  registration_date DATE
);

Seed sample data

postgres=#
INSERT INTO customers  (username, email, registration_date)
VALUES
('johndoe', 'johndoe@example.com', '2021-01-01'),
('janedoe', 'janedoe@example.com', '2021-02-15'),
('bobsmith', 'bobsmith@example.com', '2021-03-10'),
('sarahlee', 'sarahlee@example.com', '2021-04-05'),
('maxwell', 'maxwell@example.com', '2021-05-20');

Data backup

To perform recovery, we need to have access to the existing data. Hence we must store all the database changes in a separate location, allowing us to utilize this data for recovery.

This is where the Write Ahead Log (WAL) in PostgreSQL comes into play. PostgreSQL follows a mechanism that writes all transactions to the WAL before committing them to the table data files. By preserving the WAL, we can reconstruct the data by traversing the logged transactions. Utilizing the information stored in the WAL, we can systematically reapply the recorded transactions to recreate the data and achieve the desired recovery. Your WAL files are in your PostgreSQL data directory's pg_wal folder.

WAL archiving

As data modifications occur in the databases, new WAL files are generated while older WAL files are eventually discarded. Therefore, it is crucial to store the WAL files before they are deleted. Here we will use an AWS S3 bucket to store our WAL files. Also, we will use wal-e and envdir to simplify the process.

Install helper modules

python3 -m pip install wal-e envdir

If you face any dependency issues, please refer wal-e to fix them.

Store AWS S3 credentials

To use "envdir" with your AWS credentials, store them in a folder. The following code snippet demonstrates how to accomplish this:

mkdir ~/wal-e.env
echo "YOUR_AWS_REGION" > ~/wal-e.env/AWS_REGION
echo "YOUR_AWS_SECRET_ACCESS_KEY" > ~/wal-e.env/AWS_SECRET_ACCESS_KEY
echo "YOUR_AWS_ACCESS_KEY_ID" > ~/wal-e.env/AWS_ACCESS_KEY_ID
echo "YOUR_AWS_STORAGE_PATH" > ~/wal-e.env/WALE_S3_PREFIX

Configure WAL archiving

Now, let's configure PostgreSQL to store the WAL files in your S3 bucket before they are deleted. Open your PostgreSQL configuration file "postgresql.conf", which is located in your data directory i.e., "/usr/local/var/postgresql-parent", and make the following changes.

# To enable WAL archiving.
archive_mode = on

# Determines how much information is written to the WAL.
wal_level = replica

# Force PostgreSQL to switch WAL file in every 60 seconds
archive_timeout = 60

# Command for pushing the wal files to the S3 bucket.
archive_command = 'envdir ~/wal-e.env wal-e wal-push %p'

You can find more information about the above configurations by referring to this documentation.

Now let's restart the server.

 data_directory="/usr/local/var/postgresql-parent"
 pg_ctl -D $data_directory restart -l $data_directory/postgresql.log

You can watch the PostgreSQL logs like this.

 tail -f /usr/local/var/postgresql-parent/postgresql.log

After starting your server, monitor the PostgreSQL logs. If you don't observe any WAL archive logs even after 60 seconds, it is because a WAL file switch will only occur if there are any modifications. To generate the desired outcome, make updates in the database. Upon doing so, you will be able to see lines similar to the following in your PostgreSQL logs:

Message: 'MSG: completed archiving to a file\nDETAIL: Archiving to "s3://neeto-deploy-backups/local/wal_005/000000020000000000001.lzo" complete at 158.209KiB/s.\nSTRUCTURED: time=2023-05-18T06:22:27.613242-00 pid=263 action=push-wal key=s3://neeto-deploy-backups/local/wal_005/000000020000000000001.lzo prefix=local/ rate=158.209 seg=000000020000000000001 state=complete'

Base backup

Since we have enabled WAL archiving, all new changes are stored in the bucket as WAL files. However, what about the data we created before configuring WAL archiving? To ensure a complete recovery, we also need that data. Therefore, we need to take a base backup of our database. The following code will initiate the base backup process and push it to the S3 bucket.

envdir ~/wal-e.env wal-e backup-push /usr/local/var/postgresql-parent

Running the above command yields the following output.

wal_e.main   INFO     MSG: starting WAL-E
DETAIL: The subcommand is "backup-push".
STRUCTURED: time=2023-05-18T06:24:27.613242-00 pid=41178
wal_e.worker.upload INFO     MSG: begin uploading a base backup volume

DETAIL: Uploading to "s3://neeto-deploy-backups/local/basebackups_005/
base_000000010000000000005_00000040/tar_partitions/part_00000000.tar.lzo".
STRUCTURED: time=2023-05-18T06:24:29.527622-00 pid=41178
wal_e.worker.upload INFO     MSG: finish uploading a base backup volume

DETAIL: Uploading to "s3://neeto-deploy-backups/local/basebackups_005/
base_000000010000000000005_00000040/tar_partitions/part_00000000.tar.lzo"
complete at 1459.68KiB/s.
STRUCTURED: time=2023-05-18T06:24:33.121716-00 pid=41178
NOTICE:  all required WAL segments have been archived

After completing the base backup, you can verify the presence of the base backup folder in your S3 bucket. Now we have the backup and new WAL files are being pushed to our bucket.

Data recovery

Let's consider a scenario where you need to perform a rollback. Suppose you accidentally deleted a table from your database that contained multiple records, making it impractical to recall and reinsert them manually. As a first step, make a note of the time at which the table was deleted. For instance, let's say it occurred on 2023-05-18 11:57:11 UTC. The objective is to roll back to a point before that time. Let's roll back to 2023-05-18 at 11:57:00 UTC.

Let's make a mistake

We are going to drop a table.

postgres=#
DROP TABLE customers;

Note the time the table got deleted. Here it is 2023-05-18 11:57:11 UTC. Now let's discuss how we can restore the deleted table.

Creating a new data directory

Let's create a new data directory where we can recover the data.

initdb /usr/local/var/postgresql-child

Fetch the base backup

The base backup serves as the starting point, while the remaining data will be regenerated by PostgreSQL using the WAL files. To fetch the base backup, you can use the following code snippet.

envdir ~/wal-e.env wal-e backup-fetch /usr/local/var/postgresql-child LATEST

The "LATEST" tag will pull the latest base backup if multiple base backups are in the bucket. On running the above command you will receive an output similar to the following.

wal_e.main   INFO     MSG: starting WAL-E
DETAIL: The subcommand is "backup-fetch".
STRUCTURED: time=2023-05-18T06:29:22.826910-00 pid=41789
wal_e.worker.s3.s3_worker INFO     MSG: beginning partition download
DETAIL: The partition being downloaded is part_00000000.tar.lzo.
HINT: The absolute S3 key is local/basebackups_005/
base_000000010000000000000005_00000040/tar_partitions/part_00000000.tar.lzo.
STRUCTURED: time=2023-05-18T06:29:24.674572-00 pid=41789

Configuring recovery

Now, we need to regenerate the remaining data from the WAL files. To do that, make the following changes in the PostgreSQL configuration file.

#postgresql.conf

# Command to fetch each wal files from the bucket.
restore_command = 'envdir ~/wal-e.env wal-e wal-fetch %f %p'

# Time at which we want to stop recovery.
recovery_target_time  = '2023-05-18 11:57:00'

We can stop the parent server since it is no longer needed.

pg_ctl -D /usr/local/var/postgresql-parent stop

To initiate the server in recovery mode and to allow PostgreSQL to regenerate the data using the WAL files, create a file named "recovery.signal" in the new data directory.

touch /usr/local/var/postgresql-child/recovery.signal

Start the child server pointing to the new data directory by running the following command.

 data_directory="/usr/local/var/postgresql-child"
 pg_ctl -D $data_directory start -l $data_directory/postgresql.log

If you examine the PostgreSQL logs, you will notice that the recovery process is paused at the specified time.

2023-05-18 12:07:41.688 IST [43159] LOG:  restored log file "000000010000000000006" from archive
2023-05-18 12:07:41.692 IST [43159] LOG:  recovery stopping before commit of transaction 743, time 2023-05-18 11:57:11.135927+05:30
2023-05-18 12:07:41.692 IST [43159] LOG:  pausing at the end of recovery
2023-05-18 12:07:41.692 IST [43159] HINT:  Execute pg_wal_replay_resume() to promote.

It's important to note that you won't be able to connect to the server using psql at this point because it is not ready to accept connections. By default, PostgreSQL pauses the recovery process after reaching the recovery target time. To change this behavior and to allow the server to accept connections after reaching the target time, add the following configuration to your PostgreSQL configuration file.

recovery_target_action  = 'promote'

After making the above change, restart the server once again by running the following command.

data_directory="/usr/local/var/postgresql-child"
pg_ctl -D $data_directory restart -l $data_directory/postgresql.log

Now, you will observe that the server is ready to accept connections after completing the recovery process. Once the recovery is successfully completed, PostgreSQL automatically deletes the "recovery.signal" file. You can verify the successful deletion of the recovery file from your data directory.

Once the server is up and ready, connect to the server and check if the deleted table is present. You will find that the deleted table is now restored and available in the database.

postgres=#
select * from customers;

 user_id | username |        email         | registration_date
---------+----------+----------------------+-------------------
       1 | johndoe  | johndoe@example.com  | 2021-01-01
       2 | janedoe  | janedoe@example.com  | 2021-02-15
       3 | bobsmith | bobsmith@example.com | 2021-03-10
       4 | sarahlee | sarahlee@example.com | 2021-04-05
       5 | maxwell  | maxwell@example.com  | 2021-05-20
(5 rows)

To learn more about PostgreSQL continuous backup, you can refer to the official PostgreSQL documentation.

Challenges faced

Restoration of PostgreSQL roles

When we restored the PostgreSQL database, the roles and passwords were also restored in the database. However, we create a new database with the desired data in the continuous rollback feature. Therefore, we should use a new role and password to ensure security. We resolved this issue by changing the restored role and password to a new one.

Early recovery completion

Another issue we encountered was the early recovery of the database before reaching the specified target time. Suppose no activity occurs in the database, and we attempt to roll back to the current time. PostgreSQL initiates the recovery process and recovers all the available WAL files. However, it fails to locate the target time since no corresponding WAL file is available for the given timestamp. PostgreSQL raises the following error in this scenario and shuts down the server.

2023-05-18 02:12:15.846 UTC [76] LOG:  last completed transaction
was at log time 2023-05-18 02:04:02.755919+00
2023-05-18 02:12:15.846 UTC [76] FATAL:  recovery ended before configured
recovery target was reached
2023-05-18 02:12:15.850 UTC [75] LOG:  startup process (PID 76) exited with
exit code 1
2023-05-18 02:12:15.850 UTC [75] LOG:  terminating any other active server
processes
2023-05-18 02:12:15.853 UTC [75] LOG:  shutting down due to startup
process failure
2023-05-18 02:12:15.859 UTC [75] LOG:  database system is shut down

In this case, we can remove the specified target time and restart the server. If we do not provide any target time, PostgreSQL will recover until the last available WAL file, which will contain the latest transaction of our database.

Recovery is an asynchronous process

As mentioned above, we must update the restored role and password to new ones.

update-role-and-password(){
# Create a new temporary user as we cannot update the current session user.
psql -d <DB_NAME> \
-c "CREATE USER tempuser WITH SUPERUSER LOGIN PASSWORD 'tempuser-pwd';"

# Login as a temporary user and update the parent role and password.
PGPASSWORD=tempuser  PGUSER=tempuser-password  psql -d <DB_NAME> \
-c "ALTER ROLE <parent-role> RENAME TO <NEW_ROLE>;
ALTER ROLE <NEW_ROLE> WITH PASSWORD <NEW_PASSWORD>;"

# Login as the new user and remove the temporary user.
PGPASSWORD=<NEW_ROLE> PGUSER=<NEW_PASSWORD> psql -d <DB_NAME> \
-c "DROP ROLE tempuser;"
}


# Start the server in recovery mode.
pg_ctl -D /usr/local/var/postgresql-child start

# Call the method to update role and password.
update-role-and-password

If you try the above code, you will encounter the following error.

ERROR:  cannot execute CREATE ROLE in a read-only transaction

Since the recovery process is asynchronous, the system may attempt to update the role before completing the recovery. It is important to note that the server will be in read-only mode during the recovery, which can lead to the error mentioned.

To solve this issue, it is necessary to determine when the recovery process was completed. This can be achieved by checking for the existence of the "recovery. signal" file in a loop.

wait_until_recovery_is_completed()
{
  recovery_signal_path="/usr/local/var/postgresql-child/recovery.signal"
  while [ -f "$file_path" ]; do
    sleep 1  # Wait for 1 second before checking again.
  done
}

pg_ctl -D /usr/local/var/postgresql-child start
wait_until_recovery_is_completed
update-role-and-password

How it all started

At Neeto, we are building 20+ products simultaneously. All of these products had their production and staging environments hosted on Heroku. In addition, pull requests from these products had review apps deployed on Heroku. With an average of 5 open PRs at any time per product, we are talking about 100+ live review apps, 20+ staging apps, and 15+ production apps. Each one of these apps needed PostgreSQL, Redis, and ElasticSearch instances, besides other requirements.

In August 2022, Heroku made this announcement.

"Starting November 28, 2022, we plan to stop offering free product plans and plan to start shutting down free dynos and data services."

It meant that in 2 months, our monthly Heroku bill would skyrocket. At first, we talked about switching to one of the Heroku alternatives.

"Every problem is an opportunity in disguise." - John Adams.

And we thought - "Why not build our own Heroku replacement?".

BigBinary has over 11 years of Ruby on Rails and React consulting expertise. However, building a Heroku replacement, a complex The DevOps-heavy project was way out of our comfort zone. With faith in our engineering team and an exuberant spirit of youthful adventure, we embarked on a journey in early September 2022 to build NeetoDeploy.

The Journey

We put together a team of 4 developers. We discussed technologies ranging from Capistrano, Ansible, and AWS to Kubernetes. The more we talked, the more it became clear that no single technology could solve this problem. We needed a complex system with multiple independent components that would interact together to handle the following core responsibilities:

1. GitHub integration.
2. A build system that would identify the application's run-time dependencies, install them in the correct order, and build runnable machine images.
3. A scalable containerized environment that can run the pre-built machine images.
4. A place to store the built machine images.
5. If the application needed a connection to external services (PostgreSQL, Redis, ElasticSearch, etc.), the ability to provision those on run-time.
6. A dashboard to view and manage the deployments.
7. Live logs.
8. Load balancing/routing.

The Architecture

Once we understood the core functionality, we devised an architecture for the system. We then drew an architecture diagram depicting the different components and how they would interact with each other.

This diagram helped facilitate the rest of the discussions and our questions regarding the project's feasibility. We looked at the diagram and saw for the first time that we could break up this complex beast into manageable pieces that we could build, test and deploy independently.

As per the plan, NeetoDeploy would comprise the following independent components:

1. Slug Compiler
2. Dyno Manager
3. Add-on Manager
4. The Compute Cluster - with the router and load balancer
5. Web Dashboard

The Slug Compiler

This component would bundle the application code, its run-time dependencies (Ruby, Node, Ruby gems, NPM packages, third-party tools like ffmpeg, etc.), and the operating system into a single image. We could then run this image on a container.

Using Docker to build the image was the first approach that we tried. After analyzing the problem, we understood that we needed to generate a Dockerfile on the fly. While working on this problem, we stumbled upon Cloud Native Buildpacks.

Their website said, "transform your application source code into images that can run on any cloud". This is precisely what we needed.

CNB(Cloud Native Buildpacks) system has three components:

1. Stack
2. Builder
3. Buildpacks

CNB provides pack utility that would accept stack, builder, and buildpacks as parameters and build an OCI-compliant machine image that works with existing Docker tooling and any broader container ecosystem like Kubernetes. In addition, Heroku had open-sourced their older stacks and builders.

Now we had everything we needed to build runnable images. But we needed the ability to build multiple images in parallel since NeetoDeploy would have various projects, and each project could have numerous deployments (app corresponding to the main branch for staging/production and review apps for each commit in a pull request).

We built the initial version of Slug Compiler as a Ruby on Rails application, exposing API endpoints to start and stop a build. It would need a GitHub repository URL to fetch the source code, a list of buildpacks, and a callback URL as inputs. Once the image is successfully built, it will make a POST request to the callback URL.

The slug compiler would spin up an EC2 machine for each build request. We made a custom AMI from an Ubuntu image with pack and other packages required for the CNB build system to run. Once the pack command builds the image, it will push it to our Dockerhub account (Docker registry) and return the image URL via a callback. It would then kill the EC2 instance.

We found uploading the image to Dockerhub very slow. So we set up a private Docker registry on AWS ECS.

The Add-on Manager

Applications typically need access to a database, search service, cache service, etc. The Neeto products use PostgreSQL, Redis, and ElasticSearch. We decided to build the first version of the add-on manager to support these add-ons.

We built the add-on infrastructure using three big AWS EC2 boxes to keep things simple. Each machine ran a PG, Redis, and ElasticSearch server, respectively. We made the add-on manager as a Ruby on Rails API server, which would accept the add-on type as input and return a connection URI with login credentials included. We would support only a fixed version of the add-ons in the initial version.

Upon receiving the request for a PG add-on, we will create a new database on the PG server with a unique username/password and return the connection string. For Redis and ElasticSearch, we simulated separate connections using namespaces.

This add-on manager had some severe drawbacks since multiple applications shared the same add-on servers running on the same physical machine. Two of the biggest drawbacks had to do with Security and Resource choking.

1. Security: With some effort, it was possible to tamper with the data of other applications.
2. Resource choking: If one application uses more CPU and Memory, it blocks all other applications.

Nevertheless, this throw-away version of the add-on manager was a good start.

The Dyno manager and the compute cluster

We needed a secure, scalable, containerized environment to run our pre-built images. Kubernetes was the obvious answer. Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and managing containerized applications. We created our cluster on AWS EKS.

Dyno manager was once again a Ruby on Rails API server app, which takes a pre-built OCI image URL and a callback URL as inputs. It would then interface with the Kubernetes cluster using the kubectl CLI to deploy the image on the cluster.

Our first cluster was of fixed size. It had four nodes (EC2 machines). Kubernetes would deploy images to these machines and manage them. But the fixed-size cluster soon proved inadequate as we had more parallel deployments than could be accommodated. So we set up cluster autoscaling.

Load balancing and routing

All applications hosted on NeetoDeploy are made publicly available on a _.neetoreviewapp.com subdomain. A public IP address is needed to set up a DNS CNAME record for _.neetoreviewapp.com. We provisioned an Elastic Load Balancer, an AWS component with a public IP address. ELB is the component of our system that acts as the interface to the outside world.

We now needed to route the incoming web requests based on the subdomain to one of the deployed applications, which required a programmable router. An Nginx ingress controller would meet this requirement. The Nginx ingress controller is essentially an Nginx server set up as a reverse proxy with the redirect rules necessary.

The Web Dashboard

NeetoDeploy needed an admin user-facing web application that would perform the following functions:

1. Admin user login.
2. Initiate Github account integration.
3. Select a repo and create a project.
4. View and manage the deployments for a project.
5. Interact with other components (Slug compiler, dyno manager, and add-on manager) to coordinate and orchestrate the functioning of the NeetoDeploy system.
6. Maintain state like store project-specific environment variables, add-on connection URLs, pre-built image URLs, etc.

We built the Dashboard app using Ruby on Rails, React.js, neetoUI, and other integrations typical of any Neeto product.

Live logging

We needed to fetch and display two types of logs at run-time:

• Build and release logs.
• Application logs.

Fetching live logs from a remote machine and displaying them in the browser is a complicated problem. We went with Server Side Events (SSE) for the first version and streamed the logs directly to the browser.

1. The dyno manager would tail kubectl logs.
2. STDOUT was continuously read and transmitted as server-side events.
3. The dyno manager sent the app-logs-url to the dashboard app via a callback.
4. Dashboard app had a log-display React component that streamed the SSE events using the app-logs-url and displayed the live logs.

How it all came together

1. The Admin user signed up for a new account.
2. The admin logged in to the Dashboard app.
3. The GitHub account was connected by the admin.
4. The admin created a new project after selecting a GitHub repository.
5. The admin made a new Pull Request in the selected GitHub repository.
6. The dashboard app received the GitHub event for the new pull request creation.
7. The dashboard app requested the slug compiler to build an image for the repository using code corresponding to the pull request's latest commit. It also sent the list of buildpacks to use.
8. The dashboard app also requested the add-on manager to create the necessary add-ons.
9. The dashboard app received the add-on URLs and saved them.
10. Slug compiler built the image, stored it on ECS and made a callback to the dashboard app with the image URL, and the dashboard app saved the image URL.
11. The dashboard app requested the dyno manager to deploy and start the necessary web and worker processes after passing the image URL and add-on URLs as environment variables to set on the containers.
12. The Dyno manager talked to Kubernetes, completed the specified cluster deployments and set up the necessary ingress rules.
13. The application is available for public access at https://xyz.neetoreviewapp.com.

Roadmap

We improved each component to make them scalable, secure and robust. We added features like CLI, performance metrics, continuous database backup, enabled caching, reduced build time, horizontal scaling to support the increased load, platform extensibility with custom build packs, stabilized live logging with Fluentbit and more.

We migrated the review apps and staging environment of all neeto products to NeetoDeploy. We also migrated production instances of most of our static websites. We are now gearing up to migrate the production instances of all the Neeto products. Exciting features like custom plans for dynos, support for more add-on types, autoscaling to support unexpected surges in traffic and more, are in the pipeline.

In the coming weeks and months, we will be writing more about the challenges we faced and how we are building NeetoDeploy. Stay tuned for more.

If your application runs on Heroku, you can deploy it on NeetoDeploy without any change. If you want to give NeetoDeploy a try, then please send us an email at invite@neeto.com.

If you have questions about NeetoDeploy or want to see the journey, follow NeetoDeploy on Twitter. You can also join our community Slack to chat with us about any Neeto product.