Database Connection Pooling

When a Rails application needs to interact with a database, it establishes aconnection, which is a dedicated communication channel between the applicationand the database server. When a new request comes to Rails, the operation can behandled like this.

1. Create a connection
2. Do database operation
3. Close the connection

When the next request comes, then repeat the above process.

Creating new database connections is an expensive operation - it takes time toestablish the connection, authenticate, and set up the communication channel. Itmeans that every single time a request comes, we are spending time in setting upthe connection.

Wouldn't it be better to store the established connection to somewhere and whena new request comes, then get the pre-established connection from this pool.This expedites the process since we don't need to create a connection and closea connection in every single request.

The new process might look like this.

1. Create a connection
2. Do database operation
3. Put the connection in a pool

Now, when a new request comes, the operation will look like this.

1. Get the connection from the pool
2. Do database operation
3. Return the connection to the pool

Database connection pooling is a performance optimization technique thatmaintains a set of reusable database connections.

Active Record Connection Pool Implementation

Active Record manages a pool of database connections for each web and backgroundprocess. Each process will have a connection pool of its own, which means aRails application running with multiple processes (like Puma processes, Sidekiqprocesses) will have multiple independent connection pools. The pool is a set ofdatabase connections that are shared among threads in the same process.

Note that the pooling happens at the process level. A thread from process Acan't get a connection from process B.

When a connection is needed, the thread checks out a connection from the pool,perform operations, and then returns the connection to the pool. This is done ata query level now. For each individual query, a connection is leased, used, andthen returned back to the pool.

Pre Rails 7.2, the connection used to be leased and held till the end of therequest if it is a web request and till the job is done if it is a backgroundjob. This was a problem for applications that spent a lot of time doing I/O. Thethread will hog the connection for the entire duration of the I/O operationlimiting the number of queries that can be executed concurrently. To facilitatethis change and make query caching work, the query cache has beenupdated to be owned by the pool.

This means that the query cache is now shared across all connections in thepool. Previously, each connection had its own query cache. As the whole requestused the same connection, this was fine. But now, as the connection is leasedfor each query, the query cache needs to be shared across all connections in thepool.

Connection Pool Configuration Options

Active Record's connection pool behavior can be customized through severalconfiguration options in the database.yml file:

• pool:Sets the maximum number of connections the pool will maintain. The default istied to RAILS_MAX_THREADS, but you can set it to any value. There is a smallproblem when you set it to RAILS_MAX_THREADS which we'll discuss later.
• checkout timeout:Determines how long a thread will wait to get a connection before timing out.The default is 5 seconds. If all connections are in use and a thread waitslonger than this value, an ActiveRecord::ConnectionTimeoutError exceptionwill be raised.
• idle timeout:Specifies how long a connection can remain idle before it's removed from thepool. The default is 300 seconds. This helps reclaim resources fromconnections that aren't being used.
• reaping frequency:Controls how often the Reaper(which we'll discuss shortly) process runs toremove dead or idle connections. The default is 60 seconds.

Active Record Connection Pool Reaper

Database connections can sometimes become "dead" due to issues like databaserestarts, network problems etc. Active Record provides Reaper to handle this.

The Reaper periodically checks connections in the pool and removes deadconnections as well as idle connections that have been checked out for a longtime.

It acts somewhat like a garbage collector for database connections. The Reaperuses the idle_timeout setting to determine how long a connection can remainidle before being removed, tracking idle time based on when connections werelast used.

There is another configuration option called reaping_frequency that controlshow often the Reaper runs to remove dead or idle connections from the pool. Bydefault, this is set to 60 seconds. It means the Reaper will wake up once everyminute to perform its maintenance tasks.

If your application is spiky and receives a lot of traffic in surges, then setthe reaping frequency and idle timeout to a lower value. This will ensure thatthe reaper runs more frequently and removes idle connections more quickly,helping to keep the connection pool healthy and responsive.

Why are idle connections bad?

Idle database connections can significantly impact database performance forseveral interconnected reasons:

Memory Consumption: Each database connection, even when idle, maintains itsown memory allocation. The database must reserve memory for session state,buffers, user context, and transaction workspace. This memory remains allocatedeven when the connection isn't doing any work. For example, if each connectionuses 10 MB of memory, 100 idle connections would unnecessarily consume 1 GB ofyour database's memory that could otherwise be used for active queries, caching,or other productive work.

CPU overhead: While "idle" suggests no activity, the database still performsregular maintenance work for each connection. It must monitor connection healthvia keepalive checks, manage process tables etc.

The crucial issue is that the overhead of having idle connections scalesnon-linearly. As we add more idle connections, the database spends an increasingproportion of its CPU time just managing these connections rather thanprocessing actual queries. Thankfully, the reaper handles this for us.

How many database connections will the web and background processes utilize at maximum?

As we learned, the connection pool is managed at the process level. Each Railsprocess maintains its own pool.

1. In web processes (Puma):

Each Puma process is a separate process with its own connection pool. In aprocess, each thread can check out one connection. Therefore, maximumconnections needed per process equals max_threads setting in Puma.

2. In background processes (Sidekiq):

Sidekiq runs as a separate process with its own connection pool. The Sidekiqconcurrency setting determines the number of threads and therefore the maximumconnections needed equals the concurrency value.

Note: If you're using Sidekiq swarm and running multiple Sidekiq processes,then take that it into account.

We can calculate the total potential connections for a typical application asshown below.

Web connections = Number of web dynos * Number of Puma processes                  * `max_threads` valueBackground connections = Number of worker dynos * Number of Sidekiq processes                         * Threads per processTotal number of connections = Web connections + Background connections

The key thing to note here is that the database needs to support at least thesemany simultaneous connections.

Note that if preboot is enabled, then the maximum number of connections will bedouble the above value. This is because during the release phase, there is asmall window in which both the old dynos and new dynos are running.

In Rails 7,load_asyncwas introduced which allows us to run database queries asynchronously in abackground thread. When load_async is in used, the calculation for maximumnumber of connections needed changes a bit. First, let's understand howload_async works.

How load_async works

load_async allows Rails to execute database queries asynchronously inbackground threads. Unlike regular ActiveRecord queries which are lazily loaded,load_async queries are always executed immediately in background threads andjoined to the main thread when results are needed.

The async executor is configured through theconfig.active_record.async_query_executor setting. There are three possibleconfigurations:

1. nil (default): Async queries are disabled, and load_async will executequeries synchronously.
2. :global_thread_pool: Uses a single thread pool for all databaseconnections.
3. :multi_thread_pool: Uses separate thread pools for each databaseconnection.

Rails provides a configuration option namedglobal_executor_concurrency(default: 4) that controls how many concurrent async queries can run perprocess. So, the maximum number of connections per process when load_async isused.

Maximum connections per process = Process level concurrency                                  + global_executor_concurrency + 1

Here Process level concurrency means max_threads for the Puma process andsidekiq_conurrency for the Sidekiq process.

The "+1" accounts for the main control thread, which may occasionally need aconnection (e.g., during model introspection at class load time).

There is a nicecalculatorcreated by the folks at Judoscale which can be used tocalculate the maximum number of connections needed for your application.

Setting Database Pool Size Configuration

Our database.yml file has the following line.

pool: <%%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>

We know that a thread doesn't take more than one DB connection. So the maximumnumber of connections needed per pool is equal to the total number of threads.So the above configuration looks fine.

However this doesn't take into account whether we use load_async or not. If weuse load_async, then the number of connections needed per process will beRAILS_MAX_THREADS + global_executor_concurrency + 1.

Do we really need to go into this much detail to determine the pool size? Turnsout there is a much easier answer.

Almost all database hosting providers mention the maximum number of connectionsallowed in their plan. We can just set the pool config to the maximum number ofconnections supported by our database plan. Let us say we have a Standard-0database on Heroku. It supports up to 120 connections. So we can set the poolconfig to 120.

pool: 120

We can do this because the database connections are lazily initialized in thepool. The application doesn't create more database connections than it needs. Sowe needn't be conservative here.

The only thing we need to ensure is that the maximum connection utilizationdoesn't exceed the database plan limit. If that happens, then we have anothersolution - PgBouncer.

PgBouncer

PgBouncer is a lightweight connection pooler for PostgreSQL. It sits between ourapplication(s) and our PostgreSQL database and manages a pool of databaseconnections.

While both PgBouncer and Active Record provide connection pooling, they operateat different levels and serve different purposes.

Active Record connection pool operates within a single Ruby process and managesconnections for threads within the process, whereas PgBouncer is an externalconnection pooler that sits between the application and the database and managesconnections across all the application processes.

The dreaded ActiveRecord::ConnectionTimeoutError

This error comes up when a thread waits more than checkout_timeout seconds toacquire a connection. This usually happens when the pool size is set to avalue less than the concurrency.

For example, lets say we have set the Sidekiq concurrency to 10 and pool sizeto 5. If we have more than 5 threads wanting a connection at any point of time,the threads will have to wait.

What's the solution? As we discussed earlier setting the pool to a really highvalue should fix the error in most cases.

Even after setting the config correctly, ActiveRecord::ConnectionTimeoutErrorcan still happen and it could be puzzling. Let is discuss a few scenarios wherethis can happen.

Custom code spinning up new threads and taking up connections

class SomeService  def process    threads = []    5.times do |index|      threads << Thread.new do        ActiveRecord::Base.connection.execute("select pg_sleep(5);")      end    end    threads.each(&:join)  endend

Here 5 threads are spun up. Note that these threads also take up connectionsfrom the same pool allotted to the process.

Active Storage proxy mode

Even if our application code is not spinning up new threads, Rails itself cansometimes spin up additional threads. For example Active Storage configured inproxy mode.

Active Storages proxy controllers1,2generate responses as streams, which require dedicated threads for processing.

This means that when serving an Active Storage file through one of these proxycontrollers, Rails actually utilizes two separate threads - one for the mainrequest and another for the streaming process. Each of these threads requiresits own separate database connection from the ActiveRecord connection pool.

Rack timeouts

rack-timeout is commonly used acrossRails applications to automatically terminate long-running requests. While ithelps prevent server resources from being tied up by slow requests, it can alsocause a few issues.

Rack timeout uses Ruby'sThread#raise API to terminaterequests that exceed the configured timeout. When a timeout occurs, rack-timeoutraises a Rack::Timeout::RequestTimeoutException from another thread. If thisexception is raised while a thread is in the middle of database operations, itcan prevent proper cleanup of database connections.

Tracking down ActiveRecord::ConnectionTimeoutErrors

If we still frequently see ActiveRecord::ConnectionTimeoutError exceptions inour application, we can get additional context by logging the connection poolinfo to our error monitoring service. This can help identify which all threadswere holding onto the connections when the error occurred.

config.before_notify do |notice|  if notice.error_class == "ActiveRecord::ConnectionTimeoutError"    notice.context = { connection_pool_info: detailed_connection_pool_info }  endenddef detailed_connection_pool_info  connection_info = {}  ActiveRecord::Base.connection_pool.connections.each_with_index do |conn, index|    connection_info["connection_#{index + 1}"] = conn.owner ? conn.owner.inspect : "[UNUSED]"  end  connection_info["current_thread"] = Thread.current.inspect  connection_infoend

<thread_obj>.inspect gives us the name, id and status of the thread. Forexample, if one entry in the hash looks like#<Thread:0x00006a42eca73ba0@puma srv tp 002 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 sleep_forever>then it means that the connection is taken up by a Puma thread.

Monitoring Active Record Connection Pool Stats

If we want to monitor Active Record Connection Pool stats, then periodically weneed to send the stats to a service provider which can display the datagraphically. For periodically checking the stat, we are usingrufus-scheduler gem. Forcollecting the data and showing the data we are using NewRelic but you can useany APM of your choice. We have configured to send the pool stat every 15seconds.

Here is thegist which collects and sends data.

This was Part 5 of our blog series onscaling Rails applications. If any part of theblog is not clear to you then please write to us atLinkedIn,Twitter orBigBinary website.

Inpart 1we saw how to find ideal number of processes for our Rails application.

Inpart 3,we learned about Amdahl's law, which helps us find the ideal number of threadstheoretically.

In this blog, we'll run a bunch of tests on our real production application tosee what the actual number of threads should be for each process.

Inpart 1we discussed the presence of GVL and the concept of thread switching. Based onthe GVL's interaction, a thread can be in one of these three states.

1. Running: The thread has the GVL and is executing Ruby code.
2. Idle: The thread doesn't want the GVL because it is performing I/Ooperations.
3. Stalled: The thread wants the GVL and is waiting for it in the GVL waitqueue.

Based on the above diagram, we can approximately equate idle time toI/O time.

GVL instrumentation using perfm

Thanks to Jean Boussier's work on theGVL instrumentation API and JohnHawthorn's work on gvl_timing, we cannow measure the time a thread spends in each of these states for apps running onRuby 3.2 or higher.

Using the great work done by these folks, we have createdperfm, to help us figure out the idealnumber of Puma threads based on the application's workload.

Perfm inserts a Rack middleware to our Rails application. This middlewareinstruments the GVL, collects the required metrics and stores them in a table.It also has a Perfm::GvlMetricsAnalyzer class which can be used to generate areport on the data collected.

Using perfm to measure the application's I/O percentage

To use perfm, we need to add the following line to our Gemfile.

gem 'perfm'

We'll run bin/rails generate perfm:install. This will generate the migrationto create perfm_gvl_metrics which will be used to store request-level metrics.

Now we'll create an initializer config/initializers/perfm.rb.

Perfm.configure do |config|  config.enabled = true  config.monitor_gvl = true  config.storage = :localendPerfm.setup!

After deploying the code to production, we need to collect around 20K requestsas that will give us a fair number of data points to analyze. The GVL monitoringcan be disabled after that by setting config.monitor_gvl to false so thatthe table doesn't keep growing.

After collecting the request data, now it's time to analyze it.

Run the following code in the Rails console.

irb(main):001* gvl_metrics_analyzer = Perfm::GvlMetricsAnalyzer.new(irb(main):002*   start_time: 2.days.ago, # configure thisirb(main):003*   end_time: Time.currentirb(main):004> )irb(main):005>irb(main):006> results = gvl_metrics_analyzer.analyzeirb(main):007> io_percentage = results[:summary][:total_io_percentage]=> 45.09

This will give us the percentage of time spent doing I/O. We ran it in ourNeetoCal production application and we got a value of45%.

As we discussed inpart 3,Amdahl's law gives us a theoretical maximum speedup based on the parallelizableportion of our workload. The formula is given below.

Where:

• p is the portion that can be parallelized (in our case, it's 0.45)
• N is the number of threads
• (1 - p) is the portion that must run sequentially (in our case, it's 0.55)

Let's calculate the theoretical speedup for different numbers of threads with p= 0.45:

We can see that after 4 threads, the percentage improvement drops below 5%. Thismeans that 4 is a reasonable value for max_threads. We can set the value ofRAILS_MAX_THREADS to 4.

Looking at the table, adding a 5th thread would only give us a 3% performanceimprovement, which may not justify the additional memory usage and potential GVLcontention.

We have also created a small application to help visualize and find the idealnumber of threads when the I/O percentage is provided as input.Here is the link to theapp.

Validate thread count using stall time

This value of 4 we got theoretically by using Amdahl's law. Now it's time toput this law to test. Let's see in the real world if the value of 4 is thecorrect value or not.

What we need to do is start with RAILS_MAX_THREADS env variable (Pumamax_threads) set to 4 and then check if this value provides minimal GVLcontention. By GVL contention, we mean the amount of time a thread spendswaiting for the GVL i.e the stall time.

If the stall time is high, that means the set thread count is high. We don'twant our request threads to spend their time doing nothing causing latencyspikes.75ms is an acceptable value for stall time. The lesser the better ofcourse.

The average stall time can be found in the perfm analyzer results. As wementioned earlier, we had collected data for NeetoCal.Now let's find the average stall time.

irb(main):001* gvl_metrics_analyzer = Perfm::GvlMetricsAnalyzer.new(irb(main):002*   start_time: 2.days.ago,irb(main):003*   end_time: Time.current,irb(main):004*   puma_max_threads: 4irb(main):005> )irb(main):006> results = gvl_metrics_analyzer.analyzeirb(main):007> avg_stall_ms = results[:summary][:average_stall_ms]=> 110.24

The stall time seems a bit high. Let us decrease the RAILS_MAX_THREADS valueby 1 and collect a few data points(i.e around 20K requests). Now the value ofRAILS_MAX_THREADS will be 3. This process has to be repeated until we findthe value for which the average stall time is less than 75ms.

irb(main):001* gvl_metrics_analyzer = Perfm::GvlMetricsAnalyzer.new(irb(main):002*   start_time: 2.days.ago,irb(main):003*   end_time: Time.current,irb(main):004*   puma_max_threads: 3irb(main):005> )irb(main):006> results = gvl_metrics_analyzer.analyzeirb(main):007> avg_stall_ms = results[:summary][:average_stall_ms]=> 79.38

Now the output is closer to 75 ms.

Hence we can finalize on the value 3 as the value for RAILS_MAX_THREADS. If wedecrease the value again by one i.e set it to 2, the stall time will decreasebut we're limiting the concurrency of our application. It is a trade-off.

Remember that our goal is to maximize concurrency while minimizing GVLcontention. But if our app spends a lot of time doing I/O - for instance, if wehave a proxy application that makes a lot of external API calls directly fromthe controller, then we can switch the app server toFalcon. Falcon is tailor-made for such usecases.

Broadly speaking, one should take care of the following items to ensure that thetime spent by the request doing I/O is minimal.

• Remove N+1 queries
• Remove long-running queries
• Move inline third party API calls to background job processor
• Move heavy computational stuff to background job processor

For a finely optimized Rails application, the max_threads value will bearound 3. That's why the default value of max_threads for Rails applicationsis 3 now. This has been decided after a lot of discussionhere. We recommend you read thewhole discussion. It is very interesting.

This was Part 4 of our blog series onscaling Rails applications. If any part of theblog is not clear to you then please write to us atLinkedIn,Twitter orBigBinary website.

We have only two parameters to work with if we want to fine-tune our Pumaconfiguration.

1. The number of processes.
2. The number of threads each process can have.

Inpart 1of Scaling Rails series,we saw what the number of processes should be. Now let's look at what the numberof threads in each process should be.

Amdahl's law

Each application has a few things which must be performed in "serial order" anda few things which can be "parallelized". If we draw a diagram, then this iswhat it will look like.

Let's say that T1' and T2' are the enhanced times. These are the times theapplication would take after the enhancement has been applied. In this case theenhancement will come in the form of increasing the threads in a process.

T1' will be same as T1 since it's the serial part. T2' will be lower thanT2 since we will parallelize some of the code. After the parallelization isdone, the enhanced version would look something like this.

It's clear that the serial part (T1) will limit how much speedup we can get nomatter how much we parallelize T2.

Computer scientist Gene Amdahl cameup with Amdahl's law which givesthe mathematical value for the overall speedup that can be achieved.

I made a video explaining how this formula came about.

<iframewidth="966"height="604"src="https://www.youtube.com/embed/2hYs2X6Fb1M?si=f-P_-pcwnotnyUKT"title="Amdahl's Law"frameborder="0"allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"allowfullscreen

</iframe>

Amdahl's law states that the theoretical speedup gained from parallelization isdirectly determined by the fraction of sequential code in the program.

Now, let's see how we can use Amdahl's law to determine the ideal number ofthreads.

The parallelizable portion in this case is the portion of the application thatspends time doing I/O. The non-parallelizable portion is the time spent by theapplication executing Ruby code. Remember that because of GVL, within a process,only one thread has access to CPU at any point of time.

Now we need to know what percentage of time our app spends doing I/O. This willbe the value p as per the video.

Later in this series, we'll show you how to calculate what percentage of thetime the Rails application is spending doing I/O. For this blog let's assumethat our application spends 37% of the time doing I/O i.e the value of p is0.37.

Let's calculate how much speedup we will get if we use one thread(n is 1). Nowlet's change n to 2 and get the speedup value. Similarly, we bump up n all theway to 15 and we record the speedup.

Now let's draw a graph between the overall speedup and the number of threads.

From the graph, it can be seen that the speedup increases as the number ofthreads increases, but the rate of increase diminishes as more threads areadded. This is because the serial portion remains constant and is unaffected bythe increase in threads.

By examining the graph we can observe that the speedup gain from increasingthreads seem significant up to 4 threads, after which the incremental gain inspeedup starts to plateau.

Remember that these are theoretical maximums based on Amdahl's law. In practice,we need to use fewer threads as adding more threads can cause an increase inmemory usage and GVL contention, thereby causing latency spikes.

It's obvious that if we add more threads then more requests can be handled byPuma concurrently. What it means is that requests will be waiting for lessertime at the load balancer layer as there are more Puma threads waiting to pickup the request for processing. But in part 1, we saw that just because we havemore threads, it doesn't mean things will move faster. More threads might causeother threads to wait for the GVL.

There is no point in accepting requests if our web server can't respond to itpromptly. Whereas, if the max_threads value is set to a lower value, requestswill queue up at the Load Balancer layer which is better than overwhelming theapplication server.

If more and more requests are waiting at the load balancer level, then therequest queue time will shoot up. The right way to solve this problem is to addmore Puma processes. It is advised to increase the capacity of the Puma serverby adding more processes rather than increasing the number of threads.

Hereis a middleware that can be used to track the request queue time. This code istaken fromjudoscale.

Note that Request Queue Time is the time spent waiting before the request ispicked up for processing.

This middleware will only work if the load balancer is adding theHTTP_REQUEST_START header. Heroku automatically adds this header.

Now we need to use this middleware and for that open config/application.rbfile and we need to add the following line.

config.middleware.use RequestQueueTimeMiddleware

This was Part 3 of our blog series onscaling Rails applications. If any part of theblog is not clear to you then please write to us atLinkedIn,Twitter orBigBinary website.

Let's start from the basics. Let's see how a standard web application mostlybehaves.

Web applications and CPU usage

Code in a web application typically works like this.

• Do some data manipulation.
• Make a few database calls.
• Do more calculations.
• Make some network calls.
• Do more calculations.

Visually, it'll look something like this.

CPU work includes operations like view rendering, string manipulation, any kindof business logic processing etc. In short, anything that involves Ruby codeexecution can be considered CPU work. For the rest of the work, like databasecalls, network call etc. CPU is idle. Another way of looking at when the CPU isworking and when it's idle is this picture.

When a program is using CPU, then that portion of the code is called CPUbound and when program is not using CPU, then that portion of the code iscalled IO bound.

CPU bound or IO bound

Let us understand what CPU bound truly means. Consider the following pieceof code.

10.times do  Net::HTTP.get(URI.parse("https://bigbinary.com"))end

In the above code, we are hitting the BigBinary website 10 times sequentially.Running the above code takes time because making a network connection is atime-consuming process.

Let's assume the code above takes 10 seconds to finish. We want the code to runfaster. So we bought a better CPU for the server. Do you think now the code willrun faster?

It will not. That's because the above code is not CPU bound. CPU is not thelimiting factor in this case. This code is I/O bound.

A program is CPU bound if the program will run faster if the CPU werefaster.

A program is I/O bound if the program will run faster if the I/O operationswere faster.

Some of the examples of I/O bound operations are:

• making database calls: reading data from tables, creating new tables etc.
• making network calls: reading data from a website, sending emails etc.
• dealing with file systems: reading files from the file system.

Previously, we saw that our CPU was idle sometimes. Now we know that thetechnical term for that idleness is IO bound let's update the picture.

When a program is I/O bound, the CPU is not doing anything. We don't wantprecious CPU cycles to be wasted. So what can we do so that the CPU is fullyutilized?

So far we have been dealing with only one thread. We can increase the number ofthreads in the process. In this way, whenever the CPU is done executing CPUbound code of one thread and that thread is doing an I/O bound operation, thenthe CPU can switch and handle the work from another thread. This will ensurethat the CPU is efficiently utilized. We will look at how the switching betweenthreads works a bit later in the article.

Concurrency vs Parallelism

Concurrency and parallelism sound similar, and in your daily life, you cansubstitute one for another, and you will be fine. However, from the computerengineering point of view, there is a difference between work happeningconcurrently and work happening in parallel.

Imagine a person who has to respond to 100 emails and 100 Twitter messages. Theperson can reply to an email and then reply to a Twitter message, then do thesame all over again: reply to an email and reply to a Twitter message.

The boss will see the count of pending emails and Twitter messages go down from100 to 99 to 98. The boss might think that the work is happening in "parallel."But that's not true.

Technically, the work is happeningconcurrently. For a system to be parallel, itshould have two or more actions executed simultaneously. In this case, at anygiven moment, the person was either responding to email or responding toTwitter.

Another way to look at it is that Concurrency is about dealing with lots ofthings at the same time. Parallelism is about doing lots of things at thesame time.

If you find it hard to remember which one is which then remember thatconcurrency starts with the word con. Concurrency is the conman. It'spretending to be doing things "in parallel," but it's only doing thingsconcurrently.

Understanding GVL in Ruby

GVL (Global VM Lock) in Ruby is a mechanism that prevents multiple threads fromexecuting Ruby code simultaneously. The GVL acts like a traffic light in aone-lane bridge. Even if multiple cars (threads) want to cross the bridge at thesame time, the traffic light (GVL) allows only one car to pass at a time. Onlywhen one car has made it safely to the other end, the second car is allowed bythe traffic light(GVL) to start.

Ruby's memory management(like garbage collection) and some other parts of Rubyare not thread-safe. Hence, GVL ensures that only one thread runs Ruby code at atime to avoid any data corruption.

When a thread "holds the GVL", it has exclusive access to modify the VMstructures.

It's important to note that GVL is there to protect how Ruby works and managesRuby's internal VM state. GVL is not there to protect our application code. It'sworth repeating. The presence of GVL doesn't mean that we can write our code ina thread unsafe manner and expect Ruby to take care of all threading issues inour code.

Ruby offers tools like Mutex andconcurrent-ruby gem tomanage concurrent code. For example, the followingcode(source) is not threadsafe and the GVL will not protect our code from race conditions.

from = 100_000_000to = 050.times.map do  Thread.new do    while from > 0      from -= 1      to += 1    end  endend.map(&:join)puts "to = #{to}"

When we run this code, we might expect the result to always equal 100,000,000since we're just moving numbers from from to to. However, if we run itmultiple times, we'll get different results.

This happens because multiple threads are trying to modify the same variables(from and to) simultaneously without any synchronization. This is called racecondition and it happens because the operation to += 1 and from -= 1 arenon-atomic at the CPU-level. In simpler terms operation to += 1 can be writtenas three CPU-level operations.

1. Read current value of to.
2. Add 1 to it.
3. Store the result back to to.

To fix this race condition the above code can be re-written using aMutex.

from = 100_000_000to = 0lock = Mutex.new50.times.map do  Thread.new do    while from > 0      lock.synchronize do        if from > 0          from -= 1          to += 1        end      end    end  endend.map(&:join)puts "to = #{to}"

It's worth nothing that Ruby implementations like JRuby and TruffleRuby don'thave a GVL.

GVL dictates how many processes we will need

Let's say that we deploy the production app to AWS's EC2's t2.medium machine.This machine has 2 vCPU as we can see from this chart.

Without going into CPU vs vCPU discussion, let's keep things simple and assumethat the AWS machine has two cores. So we have deployed our code on a machinewith two cores but we have only one process running in production. No worries.We have three threads. So three threads can share two cores. You would thinkthat something like this should be possible.

But it's not possible. Ruby doesn't allow it.

Currently, it's not possible because Thread 1 and Thread 2 belong to the sameprocess. This is because of the Global VM Lock (GVL).

The GVL ensures that only one thread can execute CPU bound code at a time withina single Ruby process. The important thing to note here is that this lock isonly for the CPU bound code and only for the same process.

In the above case, all three threads can do DB operations in parallel. But twothreads of the same process can't be doing CPU operations in parallel.

We can see that "Thread 1" is using Core 1. Core 2 is available but "Thread 2"can't use Core 2. GVL won't allow it.

Again, let's revisit what GVL does. For the CPU bound code GVL will ensure thatonly one thread from a process can access CPU.

So now the question is how do we utilize Core 2. Well, the GVL is applied at aprocess level. Threads of the same process are not allowed to do CPU operationsin parallel. Hence, the solution is to have more processes.

To have two Puma processes we need to set the value of env variableWEB_CONCURRENCY to 2 and reboot Puma.

WEB_CONCURRENCY=2 bundle exec rails s

Now we have two processes. Now both Core 1 and Core 2 are being utilized.

What if the machine has 5 cores. Do we need 5 processes?

Yes. In that case, we will need to have 5 processes to utilize all the cores.

Therefore for achieving maximum utilization, the rule of thumb is that thenumber of processes i.e WEB_CONCURRENCY should be set to the number of coresavailable in the machine.

Thread switching

Now let's see how switching between threads happens in a multi-threadedenvironment. Note that the number of threads is 2 in this case.

As we can see, the CPU switches between Thread 1 and Thread 2 whenever it'sidle. This is great. We don't waste CPU cycles now, as we saw in thesingle-threaded case. But the switching logic is much more nuanced than what isshown in the picture.

Ruby manages multiple threads at two levels: the operating system level and theRuby level. When we create threads in Ruby, they are "native threads" - meaningthey are real threads that the operating system (OS) can see and manage.

All operating systems have a component called the scheduler. In Linux, it'scalled theCompletely Fair Scheduleror CFS. This scheduler decides which thread gets to use the CPU and for howlong. However, Ruby adds its own layer of control through the Global VM Lock(GVL).

In Ruby a thread can execute CPU bound code only if it holds the GVL. The RubyVM makes sure that a thread can hold the GVL for up to 100 milliseconds. Afterthat the thread will be forced to release GVL to another thread if there isanother thread waiting to execute CPU bound code. This ensures that the waitingRuby threads are notstarved.

When a thread is executing CPU-bound code, it will continue until either:

1. It completes its CPU-bound work.
2. It hits an I/O operation (which automatically releases the GVL).
3. It reaches the limit of 100ms.

When a thread starts running, the Ruby VM uses a background timer thread at theVM level that checks every 10ms how long the current Ruby thread has beenrunning. If the thread has been running longer than the thread quantum (100ms bydefault), the Ruby VM takes back the GVL from the active thread and gives it tothe next thread waiting in the queue. When a thread gives up the GVL (eithervoluntarily or is forced to give up), the thread goes to the back of the queue.

The default thread quantum is 100ms and starting from Ruby 3.3, it can beconfigured using the RUBY_THREAD_TIMESLICE environment variable.Here is the link to the discussion.This environment variable allows fine-tuning of thread scheduling behavior - asmaller quantum means more frequent thread switches, while a larger quantummeans fewer switches.

Let's see what happens when we have two threads.

1. T1 completes quantum limit of 100ms and gives up the GVL to T2.
2. T2 completes 50ms of CPU work and voluntarily gives up the GVL to do I/O.
3. T1 completes 75 ms of CPU work and voluntarily gives the GVL to do I/O.
4. Both T1 and T2 are doing I/O and doesn't want the GVL.

It means that Thread 2 would be a lot faster if it had more access to CPU. Tomake CPU instantly available, we can have lesser number of threads CPU has tohandle. But we need to play a balancing game. If the CPU is idle then we arepaying for the processing cost for no reason. If the CPU is extremely busy thenthat means requests will take longer to process.

Thread switching can lead to misleading data

Let's take a look at a simple code given below. This code is taken froma blogby Jean Boussier.

start = Time.nowdatabase_connection.execute("SELECT ...")query_duration = (Time.now - start) * 1000.0puts "Query took: #{query_duration.round(2)}ms"

The code looks simple. If the result is say Query took: 80ms then you wouldthink that the query actually took 80ms. But now we know two things

• Executing database query is an IO operation (IO bound)
• Once the IO bound operation is done then the thread might not immediately gethold of the GVL to execute CPU bound code.

Think about it. What if the query took only 10ms and the rest of the 70msThe thread was waiting for the CPU because of the GVL. The only way to knowwhich portion took how much time is by instrumenting the GVL.

Visualizing the effect of the GVL

To better understand the effect of multiple threads when it comes to Ruby'sperformance, let's do a quick test. We'll start with a cpu_intensive methodthat performs pure arithmetic operations in nested loops, creating a workloadthat is heavily CPU dependent.

Hereis the code.

Running this script produced the following output:

Running demonstrations with GVL tracing...Starting demo with 1 threads doing CPU-bound workTime elapsed: 7.4921 secondsStarting demo with 3 threads doing CPU-bound workTime elapsed: 7.8146 seconds

From the output, we can see that for CPU-bound work, a single thread performedbetter. Why? Let's visualize the result with the help of the traces generated inthe above script using the gvl-tracinggem. The trace files can be visualized usingPerfetto, which provides a timeline view showing howthreads interact with the GVL.

We can see above that in the case of CPU-bound work if we have a single threadthen it's not waiting for GVL. However, if we have three threads then eachthread is waiting for GVL multiple times.

Understanding the advantage of multiple threads in mixed workloads

Now let's look at mixed workloads in single-threaded and multi-threadedenvironment. We'll use a separate script with a mixed_workload method thatcombines CPU-bound work with I/O operations. We use IO.select with blockingbehavior to simulate I/O operations. This creates actual I/O blocking thatreleases the GVL and shows as "waiting" in the GVL trace, accuratelyrepresenting real-world I/O operations like database queries.

Hereis the code for the mixed workload test.

Running this script with 1 thread and 3 threads produced the following output:

Running demonstrations with GVL tracing...Starting demo with 1 thread doing Mixed I/O and CPU workTime elapsed: 9.32 secondsStarting demo with 3 threads doing Mixed I/O and CPU workTime elapsed: 6.1344 seconds

The key advantage of multiple threads in mixed workloads lies in how the GVL ismanaged during I/O operations. When a thread encounters an I/O operation (like adatabase query, network call, or file read), it voluntarily releases the GVL.This is fundamentally different from CPU-bound work, where threads compete forthe GVL and one thread must wait for another to finish or reach the 100msquantum limit.

During I/O operations, the thread is essentially blocked waiting for an externalresource (database, network, disk). While waiting, the thread doesn't need theGVL because it's not executing Ruby code. This creates an opportunity for otherthreads to acquire the GVL and do useful CPU work. The result is that CPU cyclesthat would otherwise be wasted during I/O waits are now being utilizedproductively by other threads.

Let's visualize this with the single-threaded case first:

In the single-threaded case, the threads wait for I/O operations to complete.During these I/O waits, the CPU sits idle. The thread performs some CPU work,then waits for I/O, then does more CPU work, then waits for I/O again. Duringall the I/O wait periods, no productive work is happening. The CPU is availablebut there's no other thread to utilize it.

Now let's look at the multi-threaded case with three threads:

When there are three threads, the situation changes a bit. Threads nowoccasionally spend time waiting for the GVL, but the overall throughput issignificantly better.

When Thread 1 releases the GVL to perform I/O, Thread 2 can immediately acquireit and start executing CPU-bound work. While Thread 2 is working, Thread 1 mightstill be waiting for its I/O operation to complete. Then when Thread 2 releasesthe GVL for its own I/O operation, Thread 3 can acquire it. This creates apipeline effect where threads are constantly handing off the GVL to each otherensuring that the CPU is almost always doing useful work.

The small amount of GVL contention we see in the multi-threaded case (threadswaiting for GVL) is more than compensated for by the elimination of idle CPUtime. Instead of the CPU sitting idle during I/O operations, other threads keepit busy.

This is why Rails applications with typical workloads (lots of database queries,API calls, and other I/O operations) benefit significantly from having multiplethreads.

Why can't we increase the thread count to a really high value?

In the previous section, we saw that increasing the number of threads can helpin utilizing the CPU better. So why can't we increase the number of threads to areally high value? Let us visualize it.

In the hope of increasing performance, let us bump up the number of threads inthe previous code snippet to 20 and see the gvl-tracing result.

As we can see in the above picture, the amount of GVL contention is massivehere. Threads are waiting to get a hold of the GVL. Same will happen inside aPuma process if we increase the number of threads to a very high value. As weknow, each request is handled by a thread. GVL contention therefore, means thatthe requests keep waiting, thereby increasing latency.

What's next

In the coming blogs, we'll see how we can figure out the ideal value formax_threads, both theoretically and empirically, based on our application'sworkload.

This was Part 2 of our blog series onscaling Rails applications. If any part of theblog is not clear to you then please write to us atLinkedIn,Twitter orBigBinary website.

If we dorails newto create a new Railsapplication,Pumawill be the default web server. Let's startby explaining how Puma handles requests.

How does Puma handle requests?

Puma listens to incoming requests on a TCP socket. When a request comes in, thenthat request is queued up in the socket. The request is then picked up by a Pumaprocess. In Puma, a process is a separate OS process that runs an instance ofthe Rails application.

Note that Puma official documentation calls a Puma process a Puma worker. Sincethe term "worker" might confuse people with background workers like Sidekiq orSolidQueue, in this article, we have used the word Puma process at a few placesto remove any ambiguity.

Now, let's look at how a request is processed by Puma step-by-step.

1. All the incoming connections are added to the socket backlog, which is an OSlevel queue that holds pending connections.

2. A separate thread (created by theReactor class) reads theconnection from the socket backlog. As the name suggests, this Reactor classimplements thereactor pattern. The reactorcan manage multiple connections at a time thanks to non-blocking I/O and anevent-driven architecture.

3. Once the incoming request is fully buffered in memory, the request is passedto the thread pool where the request resides in the @todo array.

4. A thread in the thread pool pulls a request from the @todo array andprocesses it. The thread calls the Rack application, which, in our case is aRails application, and generates a response.

5. The response is then sent back to the client via the same connection. Oncethis is complete, the thread is released back to the thread pool to handlethe next item from the @todo array.

Modes in Puma

1. Single Mode: In single mode, only a single Puma process boots and doesnot have any additional child processes. It is suitable only for applicationswith low traffic.

   

2. Cluster Mode: In cluster mode, Puma boots up a master process, whichprepares the application and then invokes thefork() system call tocreate one or more child processes. These processes are the ones that areresponsible for handling requests. The master process monitors and managesthese child processes.

Default Puma configuration in a new Rails application

When we create a new Rails 8 or higher application, the default Pumaconfig/puma.rb will have the following code.

Please note that we are mentioning Rails 8 here because the Puma configurationis different in prior versions of Rails.

threads_count = ENV.fetch("RAILS_MAX_THREADS", 3)threads threads_count, threads_countrails_env = ENV.fetch("RAILS_ENV", "development")environment rails_envcase rails_envwhen "production"  workers_count = Integer(ENV.fetch("WEB_CONCURRENCY", 1))  workers workers_count if workers_count > 1  preload_app!when "development"  worker_timeout 3600end

For a brand new Rails application, the env variables RAILS_MAX_THREADS andWEB_CONCURRENCY won't be set. This means threads_count will be set to 3 andworkers_count will be 1.

Now let's look at the second line from the above mentioned code.

threads threads_count, threads_count

In the above code, threads is a method to which we are passing two arguments.The default value of threads_count is 3. So effectively, we are calling methodthreads like this.

threads(3, 3)

The threads method in Puma takes two arguments: min and max. These argumentsspecify the minimum and maximum number of threads that each Puma process willuse to handle requests. In this case Puma will initialize 3 threads in thethread pool.

Now let's look at the following line from the above mentioned code.

workers workers_count if workers_count > 1

The value of workers_count in this case is 1, so Puma will run in singlemode. As mentioned earlier in Puma a worker is basically a process. It's notbackground job worker.

What we have seen is that if we don't specify RAILS_MAX_THREADS orWEB_CONCURRENCY then, by default, Puma will boot a single process and thatprocess will have three threads. In other words Rails will boot with the abilityto handle 3 requests concurrently.

This is the default value for Puma for Rails booting in development or inproduction mode - a single process with three threads.

Configuring Puma's concurrency and parallelism

When it comes to concurrency and parallelism in Puma, there are two primaryparameters we can configure: the number of threads each process will have andthe number of processes we need.

To figure out the right value for each of these parameters, we need to know howRuby works. Specifically, we need to know how GVL in Ruby works and how itimpacts the performance of Rails applications.

We also need to know what kind of Rails application it is. Is it a CPU intensiveapplication, IO intensive or somewhere in between.

Don't worry in the next blog, we will start from the basics and will discuss allthis and much more.

This was Part 1 of our blog series onscaling Rails applications. If any part of theblog is not clear to you then please write to us atLinkedIn,Twitter orBigBinary website.

Same with deploying to production. You need to change your database. Other thanthat it comes with sane defaults. You don't really need to know what'sRAILS_MAX_THREADS and what's WEB_CONCURRENCY. However, as your applicationstarts getting more traffic, you want to scale your application.

Over the last 13 years of consultancy at BigBinary, we have seen all types ofapplications.

We have seen Rails applications which are IO heavy like scraping websites. Thereare applications which are heavy on background jobs. Then there are flash salesites where there is no traffic one minute and the next minute there are tons oftraffic. Then there are ticketing sites.

Each application has its own challenges.

If an application is not properly tuned, then we can run into all kinds ofissues. You can run out of memory or database connections. In some cases, becauseof the wrong configuration, Sidekiq jobs were failing and these jobs continued toget enqueued which made the situation worse.

To solve all these types of issues, we need to know what's actuallyhappening. And that's what we will do. We will look at under the hood to see howRails works, how Puma works, how connection pooling works, how to tune Sidekiqand how to measure what we need to know to make decisions.

It's a journey to understand from the ground up how to scale Rails applications.This page will have links to all the future blogs. You can follow the ScalingRails series by joining the newsletter or following us ontwitter orLinkedIn. We even have anRSS feed.

Part 1 - Understanding how Puma handles requests

• How Puma handles requests
• Default Puma configuration in a new Rails application

Part 2 - GVL in Ruby and the impact of GVL in scaling Rails applications

• Web applications and CPU usage
• CPU bound or IO bound
• Concurrency vs Parallelism
• Understanding the GVL
• GVL dictates how many processes you will need
• Thread switching
• Visualizing the effect of the GVL

Part 3 - Amdahl's Law: The Theoretical Relationship Between Speedup and Concurrency

• Amdahl's law
• Relationship between speedup gained and the number of threads
• Ideal number of threads in a process
• Request queue time

Part 4 - Finding ideal number of threads per process using GVL Instrumentation

• GVL instrumentation using perfm
• Determining the I/O workload of an application using the GVL data
• Empirically determining the ideal number of Puma threads for an application

Part 5 - Understanding Active Record Connection Pooling

• Database connection pooling
• Active Record connection pool implementation
• Connection pool configuration options
• Active Record connection pool reaper
• How many database connections will the web and background processes utilize atmaximum?
• How does using load_async affect the connection usage?
• Setting database pool size configuration
• PgBouncer
• Tracking down ActiveRecord::ConnectionTimeoutError
• Monitoring Active Record connection pool stats

Part 6 - Understanding Queueing Theory

• Queueing systems
• Basic terminology in queueing theory
• Little's law
• The knee curve
• Theoretical parallelism
• Concurrency and effective parallelism