Identifying the problem

The crash happened inside Slots::SyncAllCalendarsService, a service that syncs multiple calendars concurrently using the Async gem. What made it particularly puzzling was that switching from Async::Barrier to Thread.new made the error disappear. This led us down a rabbit hole thinking it was an Async-specific issue.

The stack trace pointed to a strange location:

activerecord (7.2.1.1) lib/arel/visitors/to_sql.rb:898:in `each'
  from lib/arel/visitors/to_sql.rb:898:in `each_with_index'
  from lib/arel/visitors/to_sql.rb:898:in `inject_join'
  from lib/arel/visitors/to_sql.rb:627:in `visit_Arel_Nodes_Or'
  ... 1132 levels...

Over a thousand levels deep into visit_Arel_Nodes_Or. That's unusual. Why would visiting OR nodes in a SQL query cause such deep recursion?

The stack trace mentioned Slots::SyncAllCalendarsService, but that service itself had nothing obviously wrong. It was just iterating through calendars and syncing them. We traced deeper into the code and found the actual culprit in Neeto's Integrations::Icloud::SyncEventsService:

def abandoned_events
  pairs = event_params.map { |e| [e[:url], e[:recurrence_id]] }

  @_abandoned_events = calendar.events
    .where("start_time BETWEEN ? AND ?", start_time, end_time)
    .where.not([:url, :recurrence_id] => pairs)
end

This code finds "abandoned" events by excluding events that match specific URL and recurrence ID combinations. In production, pairs contained 233 entries. Nothing immediately suspicious, right?

To understand what was happening, we needed to see what Rails was actually doing with this query. The where.not([:url, :recurrence_id] => pairs) syntax is Rails' composite key feature. With 233 pairs, Rails generates SQL like this:

WHERE NOT (
  (url = 'url1' AND recurrence_id = 'rec1') OR
  (url = 'url2' AND recurrence_id = 'rec2') OR
  (url = 'url3' AND recurrence_id = 'rec3') OR
  -- ... 230 more OR clauses
)

233 OR clauses. That's a lot, but databases can handle it. The problem wasn't the SQL itself - it was how Rails built the internal representation of that query.

Going into Rails Internals

To really understand what changed, we used bundle open activerecord to look at the Rails source code. We compared the same code in Rails 7.1.5.2 and 7.2.1.1.

What We Found in Rails 7.1.5.2

In Rails 7.1.5.2, the code that visits OR nodes looked like this:

def visit_Arel_Nodes_Or(o, collector)
  stack = [o.right, o.left]

  while o = stack.pop
    if o.is_a?(Arel::Nodes::Or)
      stack.push o.right, o.left
    else
      visit o, collector
      collector << " OR " unless stack.empty?
    end
  end

  collector
end

This is an iterative approach using a manual stack. No matter how deeply nested the OR conditions are, this code only uses a single stack frame for the visit_Arel_Nodes_Or method. It's stack-safe.

What Changed in Rails 7.2.1.1

In Rails 7.2.1.1, the same method became much simpler:

def visit_Arel_Nodes_Or(o, collector)
  inject_join o.children, collector, " OR "
end

Where inject_join does:

def inject_join(list, collector, join_str)
  list.each_with_index do |x, i|
    collector << join_str unless i == 0
    collector = visit(x, collector)  # Recursive call
  end
  collector
end

This is a recursive approach. If an OR node contains another OR node, it calls visit, which calls visit_Arel_Nodes_Or, which calls inject_join again. The recursion depth grows with the nesting depth of the OR tree.

The Tree Structure Problem

But why is the OR tree so deeply nested? We found the answer in Active Record's PredicateBuilder#grouping_queries:

def grouping_queries(queries)
  if queries.one?
    queries.first
  else
    queries.map! { |query| query.reduce(&:and) }
    queries = queries.reduce { |result, query| Arel::Nodes::Or.new([result, query]) }
    Arel::Nodes::Grouping.new(queries)
  end
end

That reduce call is the key. It builds a left-deep nested tree:

Level 1:  Query1 OR Query2 = OR_Node_1
Level 2:  OR_Node_1 OR Query3 = OR_Node_2
Level 3:  OR_Node_2 OR Query4 = OR_Node_3
...
Level 232: OR_Node_231 OR Query233 = OR_Node_232

With 233 pairs, this creates 232 levels of nesting. Each level adds approximately 5 stack frames when Rails traverses it recursively. That's 1,160 stack frames just for the query building.

The Async Connection

Why did Thread.new work but Async::Barrier didn't?

We wrote a test script to measure the actual stack depth limits in different contexts:

def test_recursion_depth(context_name)
  depth = 0

  recurse = lambda do |n|
    depth = n
    recurse.call(n + 1)
  end

  begin
    recurse.call(0)
  rescue SystemStackError
    return depth
  end
end

thread_depth = nil
Thread.new do
  thread_depth = test_recursion_depth("thread")
end.join
puts "Thread recursion limit: ~#{thread_depth} calls"

async_depth = nil
Async do
  async_depth = test_recursion_depth("async")
end
puts "Async fiber recursion limit: ~#{async_depth} calls"

When we ran this, the results were revealing:

Thread recursion limit:      ~11910 calls
Async fiber recursion limit: ~1482 calls

Threads have roughly 8x more stack space than Async fibers. This is because threads use larger stacks (1MB-8MB) while fibers are designed to be lightweight with smaller stacks (512KB-1MB). This difference in stack allocation is why threads can handle deeper recursion before hitting the overflow limit.

With 233 pairs requiring ~1,160 stack frames, we were right at the edge of the Async fiber limit. But we were still well within the thread limit, which is why switching to Thread.new seemed to fix it.

When we tested with 500 pairs (which some of our larger calendars had), even Thread.new failed with a stack overflow. So it wasn't really a fix, it just pushed the problem further down the road.

The GitHub Trail

After we understood the problem, we searched the Rails repository and found the exact history:

1. April 4, 2024 - PR #51492: This PR changed OR nodes from binary to n-ary to handle multiple children in a single node.

2. September 24, 2024 - Issue #53031: Someone reported the exact issue we were seeing. Queries with 800+ OR conditions that worked in Rails 7.1 now crashed in Rails 7.2.

3. September 25, 2024 - PR #53032: fatkodima fixed it with a one-line change in PredicateBuilder#grouping_queries:

Before:

queries = queries.reduce { |result, query| Arel::Nodes::Or.new([result, query]) }

After:

queries = Arel::Nodes::Or.new(queries)

The difference is profound. The old code using reduce created a deeply nested structure like Russian nesting dolls:

Or([Or([Or([Query1, Query2]), Query3]), Query4])  # 232 levels with 233 pairs

The new code creates a flat structure:

Or([Query1, Query2, Query3, Query4, ...Query233])  # Just 1 level

With a flat structure, the recursive visitor only traverses one level instead of 232, eliminating the stack overflow. The fix is available in Rails 7.2.2+.

Our Solution

We're currently on Rails 7.2.1.1 and upgrading immediately would require extensive testing. So we implemented a workaround in our code:

def abandoned_events
  return @_abandoned_events if defined?(@_abandoned_events)

  pairs = event_params.map { |e| [e[:url], e[:recurrence_id]] }
  return if pairs.empty?

  events_in_time_range = calendar.events
    .where("start_time BETWEEN ? AND ?", start_time, end_time)

  # Get candidate events using simple IN clauses
  urls = pairs.map(&:first).uniq
  recurrence_ids = pairs.map(&:last).uniq

  candidate_events = events_in_time_range
    .where(url: urls)
    .where(recurrence_id: recurrence_ids)

  # Filter to exact pairs in memory
  pairs_set = pairs.to_set
  ids_to_exclude = candidate_events.select { |event|
    pairs_set.include?([event.url, event.recurrence_id])
  }.map(&:id)

  @_abandoned_events = if ids_to_exclude.empty?
    events_in_time_range
  else
    events_in_time_range.where.not(id: ids_to_exclude)
  end
end

This approach:

• Uses simple IN clauses instead of complex OR conditions
• Filters the exact pairs in memory (fast with a Set)
• Excludes by ID, which is a flat list
• Never builds deeply nested Arel nodes

The trade-off is we run 2 queries instead of 1, but the queries are simpler and more efficient. The in-memory filtering is negligible since we're already constrained by time range.

Measuring and Verifying

To prove our hypothesis, we wrote verification scripts that patched Rails internals to measure what was actually happening.

Counting Arel Visitor Calls

We created a script that patched visit_Arel_Nodes_Or to count how many times it was called and how deep the recursion went:

module Arel
  module Visitors
    class ToSql
      alias_method :original_visit_Arel_Nodes_Or, :visit_Arel_Nodes_Or

      def visit_Arel_Nodes_Or(o, collector)
        @or_call_count ||= 0
        @or_call_count += 1
        @max_depth ||= 0
        @current_depth ||= 0
        @current_depth += 1
        @max_depth = [@max_depth, @current_depth].max

        result = original_visit_Arel_Nodes_Or(o, collector)

        @current_depth -= 1
        result
      end
    end
  end
end

Results with 50 test pairs:

• Rails 7.1.5.2: visit_Arel_Nodes_Or was called 1 time and max recursion depth was 0.
• Rails 7.2.1.1: visit_Arel_Nodes_Or was called 49 times and max recursion depth was 100.

With 233 pairs in production:

• Rails 7.1.5.2: Still called 1 time, depth 0 (iterative approach)
• Rails 7.2.1.1: Called 232 times, depth ~466 (exceeds Async fiber limit)

Testing the Actual Breaking Points

We ran the problematic query with different pair counts to find exactly where it breaks:

# Test with Async
Async do
  pairs = calendar.events.limit(count).pluck(:url, :recurrence_id)
  calendar.events.where.not([:url, :recurrence_id] => pairs).count
end.wait

# Test with Thread
Thread.new do
  pairs = calendar.events.limit(count).pluck(:url, :recurrence_id)
  calendar.events.where.not([:url, :recurrence_id] => pairs).count
end.join

Results:

• Async Fiber: Breaks at approximately 233 pairs
• Thread.new: Breaks at approximately 500 pairs

This confirmed that Thread.new wasn't a real solution - it just had more headroom before hitting the same problem.

For now, we're sticking with our workaround. It works reliably, performs well, and doesn't require upgrading Rails immediately. When we do upgrade to Rails 7.2.2+, we can revert to the original clean syntax, knowing that the Rails team has fixed the underlying issue.

If you're on Rails 7.2.0 through 7.2.1.x and use composite key queries with large datasets, watch out for this issue. The fix is in Rails 7.2.2+, or you can work around it like we did.

References

• PR #51492 - Bug introduced
• Issue #53031 - Bug reported
• PR #53032 - Bug fixed

Marking an association as deprecated

Simply pass the deprecated: true option when declaring an association.

class User < ApplicationRecord
  has_many :meetings, deprecated: true
end

Now, every time the meeting association is invoked, we’ll get a deprecation warning in our logs.

> user.meetings
The association User#meetings is deprecated, the method meetings was invoked ((calendar):4:in '<main>')

> User.includes(:meetings).where(id: 1)
The association User#meetings is deprecated, referenced in query to preload records ()

> User.joins(:meetings).where(id: 1)
The association User#meetings is deprecated, referenced in query to join its table ()

Working with different association types

We can deprecate any association type.

class Order < ApplicationRecord
  # has_many
  has_many :line_items, deprecated: true

  # belongs_to
  belongs_to :customer, deprecated: true

  # has_one
  has_one :profile, deprecated: true

  # has_many through
  has_many :archived_comments, through: :comments, deprecated: true
end

Reporting modes and backtrace support

This feature supports three deprecation modes:

• :warn (default) Logs a warning to our Active Record logger.
• :raise Raises an exception when the deprecated association is used.
• :notify Emits an Active Support notification event with the key deprecated_association.active_record. This can be used to send notifications to external services like Honeybadger etc. We can check the details about its payload here.

Backtraces are disabled by default. If :backtrace is true, :warn mode will include a clean backtrace in the message, and :notify mode will have a backtrace key in the payload. Exceptions raised via :raise mode will always have a clean stack trace.

We can change the global default mode in an initializer.

# config/initializers/deprecated_associations.rb
ActiveRecord.deprecated_associations_options = {
  mode: :warn,      # :warn | :raise | :notify
  backtrace: true   # whether to include a cleaned backtrace
}

It can also be set at an environment level.

# config/environments/development.rb
Rails.application.configure do
  config.active_record.deprecated_associations_options = { mode: :raise, backtrace: true }
end

# config/environments/production.rb
Rails.application.configure do
  config.active_record.deprecated_associations_options = { mode: :warn, backtrace: true }
end

Why deprecate rather than remove?

In large applications, it’s often hard to guarantee complete test coverage. Some association usages may only surface in production.

Deprecating an association first, lets us:

• Identify every code path (tests, console, background jobs) that relies on it.
• Gradually refactor those references before removal.
• Ensure confidence that deleting the association won’t break anything unexpectedly.

This new feature in Rails provides a clean and intuitive way to phase out our associations with deprecation warnings, making it easier to maintain and refactor large codebases.

This feature was merged recently, and will be released in the next Rails minor/patch version.

Same with deploying to production. You need to change your database. Other than that it comes with sane defaults. You don't really need to know what's RAILS_MAX_THREADS and what's WEB_CONCURRENCY. However, as your application starts getting more traffic, you want to scale your application.

Over the last decade and a half of consultancy, we have seen all types of applications.

We have seen Rails applications which are IO heavy like scraping websites. There are applications which are heavy on background jobs. Then there are flash sale sites where there is no traffic one minute and the next minute there are tons of traffic. Then there are ticketing sites.

Each application has its own challenges.

If an application is not properly tuned, then we can run into all kinds of issues. You can run out of memory or database connections. In some cases, because of the wrong configuration, Sidekiq jobs were failing and these jobs continued to get enqueued which made the situation worse.

To solve all these types of issues, we need to know what's actually happening. And that's what we will do. We will look at under the hood to see how Rails works, how Puma works, how connection pooling works, how to tune Sidekiq and how to measure what we need to know to make decisions.

It's a journey to understand from the ground up how to scale Rails applications. This page will have links to all the future blogs. You can follow the Scaling Rails series by joining the newsletter or following us on twitter or LinkedIn. We even have an RSS feed.

Part 1 - Understanding how Puma handles requests

• How Puma handles requests
• Default Puma configuration in a new Rails application

Part 2 - GVL in Ruby and the impact of GVL in scaling Rails applications

• Web applications and CPU usage
• CPU bound or IO bound
• Concurrency vs Parallelism
• Understanding the GVL
• GVL dictates how many processes you will need
• Thread switching
• Visualizing the effect of the GVL

Part 3 - Amdahl's Law: The Theoretical Relationship Between Speedup and Concurrency

• Amdahl's law
• Relationship between speedup gained and the number of threads
• Ideal number of threads in a process
• Request queue time

Part 4 - Finding ideal number of threads per process using GVL Instrumentation

• GVL instrumentation using perfm
• Determining the I/O workload of an application using the GVL data
• Empirically determining the ideal number of Puma threads for an application

Part 5 - Understanding Active Record Connection Pooling

• Database connection pooling
• Active Record connection pool implementation
• Connection pool configuration options
• Active Record connection pool reaper
• How many database connections will the web and background processes utilize at maximum?
• How does using load_async affect the connection usage?
• Setting database pool size configuration
• PgBouncer
• Tracking down ActiveRecord::ConnectionTimeoutError
• Monitoring Active Record connection pool stats

Part 6 - Understanding Queueing Theory

• Queueing systems
• Basic terminology in queueing theory
• Little's law
• The knee curve
• Theoretical parallelism
• Concurrency and effective parallelism

Dual-booting is a process that allows you to run your application with different sets of dependencies, making it easy to switch between them. This approach enables you to quickly test your code with both the current and the newer version of what you are upgrading, ensuring that everything works smoothly before fully committing to the upgrade.

How to dual-boot Ruby?

The dual-boot technique involves maintaining two separate Gemfile.lock files within your Rails project: one for the current version of Ruby and another for the next version you're upgrading to.

To get started, first create a symbolic link for Gemfile.next with the following command:

ln -s Gemfile Gemfile.next

This command creates Gemfile.next file, which you'll use for the new Ruby version. This file isn't a separate copy but rather a pointer to your original Gemfile in the Rails project directory.

Next, add the following snippet at the top of your Gemfile:

def next?
  File.basename(__FILE__) == "Gemfile.next"
end

This code snippet helps determine which Gemfile is in use, whether it's the standard one or the Gemfile.next for the new Ruby version.

Now, let's utilize the next? method to dynamically set the Ruby version in the Gemfile using a conditional operator:

ruby_version = next? ? "3.3.0" : "3.2.2"
ruby ruby_version

This code snippet helps determine the Ruby version based on the next? method. If next? returns true, meaning you're operating within Gemfile.next, it sets the Ruby version to "3.3.0". Otherwise, if the standard Gemfile is being processed, it defaults to "3.2.2".

How to install dependencies?

With the dual-boot setup in place, you can effectively manage dependencies for both your current and next versions of the application.

Installing current dependencies: To install dependencies for your current version, simply run:

bundle install

This command uses the standard Gemfile to resolve and install dependencies for your current application.

Installing next dependencies: To install dependencies for the next version of your application, use the following command:

BUNDLE_GEMFILE=Gemfile.next bundle install

This command specifies Gemfile.next as the Gemfile, allowing you to install dependencies specifically for the next version.

Managing commands for the next version: To perform various operations for the next version, you can use the syntax BUNDLE_GEMFILE=Gemfile.next <command>. For example, to start the Rails server with the next version's dependencies, you would use:

BUNDLE_GEMFILE=Gemfile.next rails s

This approach ensures that Gemfile.next is explicitly used for the specified command, allowing you to work with the next version of your application while keeping dependencies and configurations correctly managed.

Considerations for dual-boot setup

When implementing a dual-boot setup, it's crucial to consider where both Ruby and gem versions are defined. While gem versions are usually specified only in the Gemfile, the Ruby version can be defined in several locations. The most common are:

• .ruby-version File: Used by version managers like rbenv to specify the Ruby version.
• Gemfile: This is where Bundler, RVM, Heroku, and similar tools reference the Ruby version to manage the Ruby environment.
• Gemfile.lock: Although the Ruby version in this file is mostly informative, it's worth noting.
• Dockerfile: Defines the Ruby version for Docker containers if you're using Docker.
• CI setup steps and configurations.
• Other less common locations within applications.

We need to make sure that dual-boot setup is compatible with all of them to ensure a smooth transition and consistency across different environments.

Sorting and pagination

If you have worked with tables, you may already be familiar with the following code. In this example, we are using the Table component from NeetoUI. It is the developer's responsibility to handle the sorting and pagination logic, make the API calls with an updated payload, and ensure that the URL is updated accordingly.

import React, { useState } from "react";
import { Table } from "@bigbinary/neetoui";

const Teams = () => {
  const [page, setPage] = useState(1);
  const [sortBy, setSortBy] = useState(null);
  const [orderBy, setOrderBy] = useState(null);

  const handleSort = ({ sortBy, orderBy }) => {
    setSortBy(sortBy);
    setOrderBy(orderBy);
    // Custom sort logic
  };

  const handlePageChange = page => {
    setPage(page);
    // Custom pagination logic.
  };

  const { data: teams, isLoading } = useFetchTeams({ page, sortBy, orderBy });

  return (
    <Table
      currentPageNumber={page}
      handlePageChange={handlePageChange}
      onChange={(_, __, sorter) => handleSort(sorter)}
      {...{ totalCount, rowData, columnData, ...otherProps }}
    />
  );
};

export default Teams;

To make both the frontend and backend more standardized and reusable, we established some conventions. The NeetoUI Table will handle both sorting and pagination internally. When the user navigates to a different page, NeetoUI will update the page and page_size query parameters in the URL.

Similarly, when a column is sorted, the sort_by and order_by query parameters are updated. We no longer need a dedicated state to store these values. When the query parameter changes, the API will be called with the modified payload.

Here is what the simplified code looks like:

import React from "react";
import { Table } from "@bigbinary/neetoui";
import { useQueryParams } = "@bigbinary/neeto-commons-frontend/react-utils"

const Teams = () => {
  const { page, sortBy, orderBy } = useQueryParams();

  const { data: teams, isLoading } = useFetchTeams({ page,
                                                     sortBy,
                                                     orderBy });

  return (
    <Table {...{ totalCount, rowData, columnData, ...otherProps }} />
  );
};
export default Teams;

Here, we utilize a useQueryParams utility helper. It parses the URL and returns the query parameters after converting them to camel case.

We noticed that a similar cleanup can be done on the backend side. After establishing a consistent variable naming convention for page, page_size, order_by, and sort_by in every listing page, it became easier to create common utility functions for the backend. Here's what it looks like:

module Filterable
  extend ActiveSupport::Concern
  include Pagy::Backend

  def sort_and_paginate(records)
    sorted_records = apply_sort(records)
    apply_pagination(sorted_records)
  end

  def apply_sort(records)
    records.order(sort_by => order_by)
  end

  def sort_by
    params[:sort_by].presence || "created_at"
  end

  def order_by
    case params[:order_by]&.downcase
    when "asc", "ascend"
      "ASC"
    when "desc", "descend"
      "DESC"
    else
      "ASC"
    end
  end

  def apply_pagination(records)
    pagination_method = records.is_a?(ActiveRecord::Relation) ? :pagy : :pagy_array
    pagy, paginated_records = send(pagination_method, records, page: params[:page], items: params[:page_size])

    [pagy, paginated_records]
  end
end

The apply_pagination helper method can be used to paginate the results. It uses the Pagy gem internally to perform the pagination. It provides a generic method pagy that works with ActiveRecord out of the box. It also provides a pagy_array method which paginates an array of records. Both these methods returns a Pagy instance along with the paginated records.

The apply_sort method sorts the given records based on a specified column name and direction. By default, records are sorted in ascending order of created_at timestamps, but this can be customized by specifying values for sort_by and order_by params. As sorting and pagination are common requirements of the listing pages, we also added a sort_and_paginate method which performs both these operations on provided records.

Searching

Searching is a common functionality in any web application. Developers need to ensure that the results are refetched whenever the search term changes. The debouncing logic is added to minimize API requests and improve user experience. As you can see in the below snippet, we had to maintain a dedicated state for search term as well.

import React, { useState } from "react";
import { useDebounce } from "@bigbinary/neeto-commons-frontend/react-utils";
import { useFetchTeams } from "hooks/reactQuery/useFetchTeamsApi";

const Teams = () => {
  const [searchString, setSearchString] = useState("");
  const debouncedSearchString = useDebounce(searchString.trim());

  const { data: teams, isLoading } = useFetchTeams(debouncedSearchString);

  return (
    <Input
      type="search"
      value={searchString}
      onChange={({ target: { value } }) => setSearchString(value)}
    />
  );
};

export default Teams;

As we standardized the naming pattern for the search term, we were able to directly incorporate the search term value into the URL query parameters. This helped us to retrieve the search term, eliminating the need for a separate state.

We also introduced a new component in NeetoMolecules, to handle the search functionality in all products. This Search component will internally handle the debounced updates when the search term changes. It will also update the search_term query param in URL. Here is what the simplified code looks like:

import React from "react";
import Search from "@bigbinary/neeto-molecules/Search";
import { useFetchTeams } from "hooks/reactQuery/useFetchTeamsApi";

const Teams = () => {
  const { searchTerm = "" } = useQueryParams();

  const { data: teams, isLoading } = useFetchTeams(searchTerm);

  return <Search />;
};

export default Teams;

We also updated the Filterable concern mentioned earlier to simplify the logic in the backend. The search_term method retrieves the keyword specified in the URL query parameters, while search? checks whether the results should be filtered based on any keyword.

module Filterable
  extend ActiveSupport::Concern

  # ... previous code

  def search_term
    filter_params[:search_term]
  end

  def search?
    search_term.present?
  end
end

Let's consolidate everything in the TeamsController. In the following code, we need to fetch all the teams belonging to an organization, filter them based on the search term and return the results after sorting and pagination.

def index
  if params[:search_string].present?
    @teams = @organization.teams.filter_by_name(params[:search_string])
  end

  @teams = @teams
    .order(params[:column] => params[:direction])
    .page(params[:current_page])
    .per(params[:limit])
end

We were able to simplify this logic with the help of the methods provided by the Filterable concern. As you can see below, this helped us in removing a lot of boilerplate code and improving the readability.

def index
  @teams = @organization.teams.filter_by_name(search_term) if search?
  @teams = sort_and_paginate(@teams)
end

This standardized approach helped us to extract and centralize the common logic. It also accelerated the development and maintenance cycle as the code structure is now clear and consistent across all the products.

Traditionally, this involved manually inspecting the logs and counting the number of queries. Needless to say, this becomes tedious and error-prone for actions generating a significant number of queries in the order of tens or hundreds.

Thankfully, Rails 7.2 introduces a helpful improvement by enhancing the log output to include the query count alongside existing information.

The improved log output now includes the query count within the ActiveRecord section. It shows 3 queries, 1 cached, indicating that three database queries were executed, with one being served from the cache.

This seemingly small addition allows for quick identification of query volume and potential optimization areas. You can easily see if caching is working effectively and if the number of queries aligns with expectations, saving developers valuable time and effort.

Check this pull request for more details:

Rails now include all the key building blocks needed to do basic authentication, but many new developers are still uncertain of how to put them together, so they end up leaning on all-in-one gems that hide the mechanics.

To address this, Rails 8 has introduced a generator that simplifies the addition of basic authentication to Rails applications. In this blog post, we'll explore the components included in this authentication scaffold.

Adding authentication to your Rails application

To set up a basic authentication system in your Rails application, execute the following code.

bin/rails generate authentication

This command generates a set of essential files that provide a foundational start for implementing authentication, including support for database-tracked sessions and password reset functionality. Here is the gist of the generated scaffold.

Let’s examine the components created by the authentication generator.

Models & migrations

To lay the groundwork for handling user accounts and session management, the models User, Current and Session are set up along with their corresponding migrations. This setup includes:

• CreateUsers migration: It creates a users table, with a uniquely indexed email_address field and password_digest for storing the securely hashed passwords using has_secure_password.

• CreateSessions migration: It sets up the sessions table, which includes a unique token field, along with ip_address and user_agent fields to record the user's device and network details. The Session model uses has_secure_token to generate unique session tokens.

• The Current model manages per-request state and provides access to the current user's information through a delegated user method.

• Additionally, the bcrypt gem is added to the Gemfile. If this gem is not present, it is added; if it is commented, it is uncommented, and then bundle install is run to install the gem.

Authentication concern

The core authentication logic and session management are encapsulated in the Authentication concern.

• require_authentication: This is a before_action callback which attempts to restore an existing session with resume_session. If no session is found, it redirects the user to the login page using request_authentication.

• resume_session: Retrieves a session using a signed token from the cookie via the find_session_by_cookie method. Sets it as the current session, and saves the token in a permanent, HTTP-only cookie using the set_current_session method.

• authenticated?: A helper method that checks if the current user has an active session.

• allow_unauthenticated_access: This class method allows specific actions to bypass the require_authentication callback.

• after_authentication_url: Returns the URL to redirect to after authentication.

• start_new_session_for(user): Creates a new session for the given user with details of the user's device and IP address and then sets the current session.

• terminate_session: Destroys the current session and removes the session token from cookies.

Managing sessions

The SessionsController handles user session management and includes the following actions:

• new: Renders a login form for the user credentials. The new.html.erb file includes fields for email and password, displays flash messages for success and errors, and provides a link for password recovery.

• create: Authenticates the user with the provided credentials. On success, it creates a new session and redirects to the after_authentication_url; on failure, it redirects to the login form with an error message.

• destroy: Terminates the user’s session and redirects to the login form.

Combining all of this, a basic authentication flow would look like this.

Password reset functionality

The basic password reset functionality includes initiating a password reset request, dispatching reset instructions via email, and updating the password. The PasswordsController manages these actions as follows:

• new: Displays the form to request a password reset using the new.html.erb template.

• create: Handles the reset request, sends a reset email via the PasswordMailer if the user exists, and redirects with a notice. This email contains a link to the password reset page, which includes a password_reset_token as a parameter. The password_reset_token is part of the newly added configuration to has_secure_password. It has a default expiration period of 15 minutes.

• edit: Renders the edit.html.erb template to set a new password.

• update: Updates the user's password, redirects on success, or shows an alert on failure.

• set_user_by_token: It is the before_action callback for the edit and update actions which retrieve the user based on the password_reset_token provided in the request parameters, ensuring proper identification of the user.

  Putting these together, the flow of resetting the password would be as follows:

Current limitations and considerations

The current authentication generator supports email-password login for existing users but does not handle new account creation. Future updates may incorporate user account creation and other customizations.

Please check out the following pull requests for more details:

• https://github.com/rails/rails/pull/52328
• https://github.com/rails/rails/pull/52472
• https://github.com/rails/rails/pull/52483

neeto-templates-nano serves as an incredibly flexible and user-friendly template builder within the Neeto ecosystem. In this blog post, we will explore the evolution from previous approaches to the development of neeto-templates-nano and how it effectively addresses the challenges encountered.

Evolution of templates at NeetoSite

NeetoSite is a website builder.

Before the advent of neeto-templates-nano, NeetoSite relied on three tables - templates_sites, templates_pages, and template_blocks to store site, page, and blocks data for templates. This structure duplicated the schema already employed in NeetoSite for sites, pages, and blocks, resulting in redundancy.

Another challenge arose with the storage of each template in a YAML file, necessitating a custom rake task for maintenance. Tasks like updating templates, such as changing font families, required laborious manual edits to every YAML file, affecting scalability and maintainability.

This YAML snippet portrays the complexity of maintaining templates. The need for manual edits in each YAML file hampers scalability and maintainability.

## Example of a template in YAML format
name: Document Signer
keywords: "E-signature, AI, Cost effective, Paperless"
pages:
  - name: "Home"
    url: "/"
    blocks:
      - name: "header_with_logo_title"
        category: "header"
        kind: "header_with_logo_title"
        identifier: "Header"
        configurations:
          design:
            body:
              border:
                borderColor: "#FFFFFF"
                borderStyle: none
                borderWidth: 0
              backgroundColor: "#FFFFFF"
              paddingVertical: 8
              paddingHorizontal: 48
            logo:
              height: 52
            links:
              color: "#1F2433"
              fontSize: "1em"
              fontFamily: Lato
              fontWeight: 500
              letterSpacing: 0
            buttons:
              color: "#ffffff"
              border:
                borderColor: "#f4620c"
                borderStyle: solid
                borderWidth: 1
              fontSize: "0.875em"
              fontFamily: Open Sans
              fontWeight: 500
              borderRadius: 9999
              letterSpacing: 0
              backgroundColor: "#f4620c"
            logoTitle:
              color: "#1F2433"
              fontSize: "0.875em"
              fontFamily: Inter
              fontWeight: 500
              letterSpacing: 0
            hamburgerMenu:
              color: "#000000"
          properties:
            logo:
              alt: Max Chat
              url: "#!"
              title: ""
            links:
              - to: "#Clients"
                label: Clients
                action: internal
              - to: "#Insights"
                label: Insights
                action: internal
            position: sticky
            enableAnimation: true

Recognizing these challenges, we opted for a more scalable and maintainable solution.

The neeto-templates-nano solution

Eliminating redundancy

To streamline template creation and management, neeto-templates-nano introduces a comprehensive solution. It includes a frontend package, @bigbinary/neeto-templates-frontend, and a ruby gem, neeto-templates-engine (private gem).

Architecture

The architecture addresses redundancy by whitelisting the templates organization for template creation. Once a site is constructed from the templates organization and published, it becomes a template accessible from other organizations.

In this illustration, we've published 5 sites at templates organization of NeetoSite. After publication, the sites become accessible to users across all organizations. Choosing any template will clone it from the templates organization to the user's organization. This straightforward mechanism simplifies the management and sharing of templates across different organizations.

The CreateTemplateModal was exported from @bigbinary/neeto-templates-frontend, allows users to see and select templates when creating a new site.

This GIF shows how simple it is to browse through available templates using the CreateTemplateModal. Users can preview templates, pick the one that suits their needs, and quickly start their projects without reinventing the wheel.

Advantages of porting to neeto-templates-nano

1. Effortless Template Management: With neeto-templates-nano, managing templates is a breeze. Administrators can easily add, update, or delete templates by accessing the templates organization and simply publishing the changes. This streamlines the template lifecycle, ensuring that users always have access to the latest versions.

2. Enhanced Template Customization: neeto-templates-nano introduces new customizations that enable administrators to add tags and cover images for templates directly within the templates organization. This functionality enhances the visual appeal of templates and facilitates better organization and searchability. Administrators can tailor templates to specific project requirements, improving overall user experience and productivity.

Implementation across Neeto products

neeto-templates-nano has seamlessly integrated into NeetoForm and NeetoSite, offering a standardized approach to template management across all Neeto products.

Counter cache integration challenges

When introducing counter caches to large datasets, developers often encounter two primary challenges:

• Backfilling data efficiently: Adding a counter cache column to an existing table with a substantial amount of data can be problematic. Backfilling the counter cache values separately from the column addition is necessary to prevent prolonged table locks, which can significantly impact application performance. This process requires careful consideration to ensure data integrity while minimizing downtime and avoiding disruptions to user experience.
• Ensuring data consistency: Once the counter cache is in place, maintaining data consistency becomes paramount. Methods such as size, any?, and others that utilize counter caches internally must return accurate results. However, during the backfilling process, relying solely on the counter cache may produce incorrect counts until all records are appropriately updated.

Safer counter cache implementation in Rails 7.2

The new update in Rails 7.2 introduces a feature that allows developers to manage counter caches more effectively, especially in scenarios involving existing large datasets. By introducing the active option in the counter cache configuration, developers can control when the counter cache is actively utilized. This enables them to backfill counter cache columns separately from their addition, minimizing table locks and potential performance issues. Once the backfilling process is complete, developers can activate the counter cache, ensuring accurate association counts without compromising application performance.

Let's illustrate the implementation of this update in Rails using a straightforward example involving a blog application with articles and comments. In our blog application, each article can have multiple comments. We want to add a counter cache to track the number of comments associated with each article. However, our database already contains a significant amount of data, making traditional counter cache implementation challenging.

Implementation Steps

1. Define the association: Initially, we define the association between the Article model and the Comment model, specifying the counter cache with the active: false option to keep it inactive during the initial setup.

class Comment < ApplicationRecord
  belongs_to :article, counter_cache: { active: false }
end

2. Backfill the Counter Cache: With the association configured, we proceed to backfill the counter cache column in the articles table. During this phase, the counter cache remains inactive, and methods like size, any?, etc., retrieve results directly from the database. This prevents incorrect values from getting displayed during backfilling.

3. Activate the Counter Cache: Once the backfilling process is complete, we activate the counter cache by removing the active: false option from the counter cache definition.

class Comment < ApplicationRecord
  belongs_to :article, counter_cache: true
end

Upon activation, the counter cache integrates into the association, efficiently tracking the number of comments associated with each article.

This PR introduced active option. Checkout full feature discussion here.

This blog is about how we automated handling and displaying error pages in neeto applications.

Before we jump in, let's look at Axios and Zustand, two npm packages we heavily rely on.

Axios

We use Axios to make API calls. Axios allows us to intercept API calls and modify them if necessary.

axios.interceptors.request.use(request => {
  // intercept requests
});
axios.interceptors.response.use(request => {
  // intercept responses
});

This feature is useful for tasks such as setting authentication tokens, handling case conversion for requests and responses, cleaning up sensitive headers when sending requests to third-party domains, etc. This is also a good place to universally handle any errors on API calls.

Zustand

Zustand is a small, fast, scalable, barebones state management solution using simplified flux principles. We use Zustand for state management instead of Redux/React context. Here is a simple example of how to create a store using Zustand.

import { create } from "zustand";

const useBearStore = create(set => ({
  bears: 0,
  increasePopulation: () => set(state => ({ bears: state.bears + 1 })),
  removeAllBears: () => set({ bears: 0 }),
}));

To learn more about Zustand, check out there documentation.

Detecting errors

Let's look into how we handle errors in our applications. As a first step, let's examine the universal Axios response interceptor we wrote to detect any error.

import axios from "axios";

axios.interceptors.response.use(null, error => {
  if (error.response?.status === 401) {
    resetAuthTokens();
    window.location.href = `/login?redirect_uri=${encodeURIComponent(
      window.location.href
    )}`;
  } else {
    const fullUrl = error.request?.responseURL || error.config.url;
    const status = error.response?.status;
    //TODO: Notify the user that an error is occurred.
  }
  return Promise.reject(error);
});

For each response, if it detects a 401 HTTP error, the code resets the authentication tokens and redirects the user to the login page. For any other error, we need to notify the user with an appropriate message.

Storing errors

When an error happens, the user could be on any page, so we should be able to display an error message no matter which page the user is on. For that, we need to store the error in a way that it can be retrieved from anywhere. This is where Zustand comes into play. It is very easy to create a store in Zustand and can be used as a hook in React applications.

import { prop } from "ramda";
import { create } from "zustand";

const useDisplayErrorPage = () => useErrorDisplayStore(prop("showErrorPage"));

export const useErrorDisplayStore = create(() => ({
  showErrorPage: false,
  statusCode: 404,
  failedApiUrl: "",
}));

export default useDisplayErrorPage;

The code snippet above creates a Zustand store for storing error-related data. Additionally, it provides a hook that conveniently checks whether an error has occurred anywhere within the application.

Back to our Axios interceptor, we can store the error using the useDisplayErrorPage hook.

import axios from "axios";
import { useErrorDisplayStore } from "./useDisplayErrorPage";

axios.interceptors.response.use(null, error => {
  if (error.response?.status === 401) {
    resetAuthTokens();
    window.location.href = `/login?redirect_uri=${encodeURIComponent(
      window.location.href
    )}`;
  } else {
    const fullUrl = error.request?.responseURL || error.config.url;
    const status = error.response?.status;
    useErrorDisplayStore.setState({
      showErrorPage: true,
      statusCode: status,
      failedApiUrl: fullUrl,
    });
  }
  return Promise.reject(error);
});

Display errors

Now, we can use the useDisplayErrorPage hook at the root of our React application. When an error happens, showErrorPage will become true; we can use that to show an error page.

import useDisplayErrorPage from "./useDisplayErrorPage";
import ErrorPage from "./ErrorPage";
const Main = () => {
  const showErrorPage = useDisplayErrorPage();
  if (showErrorPage) {
    return <ErrorPage />;
  }

  return <>Our App</>;
};

Let's look at the ErrorPage component. The component reads the error data from the Zustand stores and displays the appropriate error message and the picture.

import { useErrorDisplayStore } from "./useDisplayErrorPage";
import { shallow } from "zustand/shallow";

const ERRORS = {
  404: {
    imageSrc: "not-found.png",
    errorMsg: "The page you're looking for can't be found.",
    title: "Page not found",
  },
  403: {
    imageSrc: "unauthorized.png",
    errorMsg: "You don't have permission to access this page.",
    title: "Unauthorized",
  },
  500: {
    imageSrc: "server-error.png",
    errorMsg:
      "The server encountered an error and could not complete your request.",
    title: "Internal server error",
  },
};

const ErrorPage = ({ statusCode }) => {
  const { storeStatusCode, showErrorPage } = useErrorDisplayStore(
    pick(["statusCode", "showErrorPage"]),
    shallow
  );
  const status = statusCode || storeStatusCode;
  const { imageSrc, errorMsg, title } = ERRORS[status] || ERRORS[404];

  return (
    <div className="flex flex-col items-center justify-center h-screen">
      <title>{title}</title>
      <img src={imageSrc} className="mb-4" alt="Error Image" />
      <div className="text-lg font-medium">{errorMsg}</div>
    </div>
  );
};

Custom error pages in Rails

Ruby on Rails comes with default error pages for commonly encountered requests such as 404, 500, and 422 errors. Each request has an associated static HTML page located within the public directory. Even though we can customize them to look like our ErrorPage component, maintaining error pages in different places will be difficult, and in the long run, they can go out of sync. So, we decided to use the ErrorPage component for these scenarios as well.

To accomplish this, we'll start by creating a controller named ErrorsController. This controller will contain a single show action, where we'll extract the error code from the raised exception and render the appropriate view.

class ErrorsController < ApplicationController
  before_action :set_default_format

  def show
    @status_code = params[:status_code] || "404"
    error = @status_code == "404" ?  "Not Found" : "Something went wrong!"

    if params[:url]
      Rails.logger.warn "ActionController::RoutingError (No route matches [#{request.method}] /#{params[:url]})"
    end

    respond_to do |format|
      format.json { render json: { error: }, status: @status_code }
      format.any { render status: @status_code }
    end
  end

  private

    def set_default_format
      request.format = :html unless request.format == :json
    end
end

Next, we'll create a view file for this show action. In this view file, we will render the Error component and pass error_status_code to it as props.

<%= react_component("Error", { error_status_code: @status_code }, { class: "root-container" }) %>

In the Error component, we will render the same ErrorPage component created above and pass the error status code received.

import React from "react";
import ErrorPage from "./ErrorPage";

const reactProps = JSON.parse(
  document.getElementsByClassName("root-container")[0]?.dataset?.reactProps ||
    "{}"
);

const Error = () => <ErrorPage status={reactProps.error_status_code} />;

export default Error;

Now add the following route in the config/routes.rb file to point 404 and 500 requests to the show action in the ErrorsController.

Rails.application.routes.draw do
  match "/:status_code", constraints: { status_code: /404|500/, format: :html }, to: "errors#show", via: :all
end

Finally, we need to tell the Rails application to use our new controller and routes setup instead of those HTML templates in the public directory. For this, add the following to the class Application < Rails::Application block in config/application.rb file.

config.exceptions_app = self.routes

Handling unmatched routes

Sometimes, we get random requests from bots and crawlers to access the paths that don't exist. When this occurs, our server raises an ActionController::RoutingError exception. This also happens when a user mistypes a URL. However, in such cases, instead of just throwing the exception, we should inform the user that what they are requesting does not exist by showing a 404 page.

To handle this, we implemented a catch_all route, as the name suggests, which catches any routes that don't match already defined routes, which will be placed at the end of our routes configuration. This placement ensures that it is only called if a request doesn't match any other defined route.

Rails.application.routes.draw do
  #Define other routes here...

  #Catch-all route for handling errors
  match ":url", to: "errors#show", via: :all, url: /.*/, constraints: -> (request) do
    request.path.exclude?("/rails/")
  end
end

This route also uses the same ErrorsController mentioned above for handling unmatched requests and will show a 404 page.

Please note that Solid Queue currently doesn't support cron-style or recurring jobs. There is a PR open regarding this issue. We have only partially migrated to Solid Queue. For recurring jobs, we are still using Sidekiq. Once the PR is merged, we will migrate completely to Solid Queue.

Migrating to Solid Queue from Sidekiq

Here is a step-by-step migration guide you can use to migrate your Rails application from Sidekiq to Solid Queue.

1. Installation

• Add gem "solid_queue" to your Rails application's Gemfile and run bundle install.
• Run bin/rails generate solid_queue:install which copies the config file and the required migrations.
• Run the migrations using bin/rails db:migrate.

2. Configuration

The installation step should have created a config/solid_queue.yml file. Uncomment the file and modify it as per your needs. Here is how the file looks for our application.

default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "auth"
      threads: 3
      processes: 1
      polling_interval: 0.1
    - queues: "urgent"
      threads: 3
      processes: 1
      polling_interval: 0.1
    - queues: "low"
      threads: 3
      processes: 1
      polling_interval: 2
    - queues: "*"
      threads: 3
      processes: 1
      polling_interval: 1

development:
  <<: *default

staging:
  <<: *default

heroku:
  <<: *default

test:
  <<: *default

production:
  <<: *default

3. Starting Solid Queue

On your development machine, you can start Solid Queue by running the following command.

bundle exec rake solid_queue:start

This will start Solid Queue's supervisor process and will start processing any enqueued jobs. The supervisor process forks workers and dispatchers according to the configuration provided in the config/solid_queue.yml file. The supervisor process also controls the heartbeats of workers and dispatchers, and sends signals to stop and start them when needed.

Since we use foreman, we added the above command to our Procfile.

# Procfile
web:  bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -C config/sidekiq.yml
solidqueueworker: bundle exec rake solid_queue:start
release: bundle exec rake db:migrate

4. Setting the Active Job queue adapter

You can set the Active Job queue adapter to :solid_queue by adding the following line in your application.rb file.

# application.rb
config.active_job.queue_adapter = :solid_queue

The above change sets the queue adapter at the application level for all the jobs. However, since we wanted to use Solid Queue for our regular jobs and continue using Sidekiq for cron jobs, we didn't make the above change in application.rb.

Instead, we created a new base class that inherited from ApplicationJob and set the queue adapter to :solid_queue inside that.

# sq_base_job.rb
class SqBaseJob < ApplicationJob
  self.queue_adapter = :solid_queue
end

Then we made all the classes implementing regular jobs inherit from this new class SqBaseJob instead of ApplicationJob.

# send_email_job.rb
- class SendEmailJob < ApplicationJob
+ class SendEmailJob < SqBaseJob
  # ...
end

By making the above change, all our regular jobs got enqueued via Solid Queue instead of Sidekiq.

But we realized later that emails were still being sent via Sidekiq. On debugging and looking into Rails internals, we found that ActionMailer uses ActionMailer::MailDeliveryJob for enqueuing or sending emails.

ActionMailer::MailDeliveryJob inherits from ActiveJob::Base rather than the application's ApplicationJob. So even if we set the queue_adapter in application_job.rb, it won't work. ActionMailer::MailDeliveryJob fallbacks to using the adapter defined in application.rb or environment-specific (production.rb / staging.rb / development.rb) config files. But we can't do that because we still want to use Sidekiq for cron jobs.

To use Solid Queue for mailers, we needed to override the queue_adapter for mailers. We can do that in application_mailer.rb.

# application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  # ...
  ActionMailer::MailDeliveryJob.queue_adapter = :solid_queue
end

This change is only until we use both Sidekiq and Solid Queue. Once cron style jobs feature lands in Solid Queue, we can remove this override and set the queue_adapter directly in application.rb, which will enforce the setting globally.

5. Code changes

For migrating from Sidekiq to Solid Queue, we had to make the following changes to the syntax for enqueuing a job.

• Replaced .perform_async with .perform_later.
• Replaced .perform_at with .set(...).perform_later(...).

- SendMailJob.perform_async
+ SendMailJob.perform_later

- SendMailJob.perform_at(1.minute.from_now)
+ SendMailJob.set(wait: 1.minute).perform_later

At some places we were storing the Job ID on a record, for querying the job's status or for cancelling the job. For such cases, we made the following change.

def disable_form_at_deadline
- job_id = DisableFormJob.perform_at(deadline, self.id)
- self.disable_job_id = job_id
+ job_id = DisableFormJob.set(wait_until: deadline).perform_later(self.id)
+ self.disable_job_id = job.job_id
end

def cancel_form_deadline
- Sidekiq::Status.cancel(self.disable_job_id)
+ SolidQueue::Job.find_by(active_job_id: self.disable_job_id).destroy!
  self.disable_job_id = nil
end

6. Error handling and retries

Initially, we thought the on_thread_error configuration provided by Solid Queue can be used for error handling. However, during the development phase, we noticed that it wasn't capturing errors. We raised an issue with Solid Queue as we thought it was a bug.

Rosa Gutiérrez responded on the issue and clarified the following.

on_thread_error wasn't intended for errors on the job itself, but rather errors in the thread that's executing the job, but around the job itself. For example, if you had an Active Record's thread pool too small for your number of threads and you got an error when trying to check out a new connection, on_thread_error would be called with that.

For errors in the job itself, you could try to hook into Active Job's itself.

Based on the above information, we modified our SqBaseJob base class to handle the exceptions and report it to Honeybadger.

# sq_base_job.rb
class SqBaseJob < ApplicationJob
  self.queue_adapter = :solid_queue

  rescue_from(Exception) do |exception|
    context = {
      error_class: self.class.name,
      args: self.arguments,
      scheduled_at: self.scheduled_at,
      job_id: self.job_id
    }
    Honeybadger.notify(exception, context:)
    raise exception
  end
end

Remember we mentioned that ActionMailer doesn't inherit from ApplicationJob. So similarly, we would have to handle exceptions for Mailers separately.

# application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  # ...
  ActionMailer::MailDeliveryJob.rescue_from(Exception) do |exception|
    context = {
      error_class: self.class.name,
      args: self.arguments,
      scheduled_at: self.scheduled_at,
      job_id: self.job_id
    }
    Honeybadger.notify(exception, context:)
    raise exception
  end
end

For retries, unlike Sidekiq, Solid Queue doesn't include any automatic retry mechanism, it relies on Active Job for this. We wanted our application to retry sending emails in case of any errors. So we added the retry logic in the ApplicationMailer.

# application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  # ...
  ActionMailer::MailDeliveryJob.retry_on StandardError, attempts: 3
end

Note that, although the queue adapter configuration can be removed from application_mailer.rb once the entire application migrates to Solid Queue, error handling and retry override cannot be removed because of the way ActionMailer::MailDeliveryJob inherits from ActiveJob::Base rather than application's ApplicationJob.

7. Testing

Once all the above changes were done, it was obvious that a lot of tests were failing. Apart from fixing the usual failures related to the syntax changes, some of the tests were failing inconsistently. On debugging, we found that the affected tests were all related to controllers, specifically tests inheriting from ActionDispatch::IntegrationTest.

We tried debugging and searched for solutions when we stumbled upon Ben Sheldon's comment on one of Good Job's issues. Ben points out that this is actually an issue in Rails where Rails sometimes inconsistently overrides ActiveJob's queue_adapter setting with TestAdapter. A PR is already open for the fix. Thankfully, Ben, in the same comment, also mentioned a workaround for it until the fix has been added to Rails.

We added the workaround in our test helper_methods.rb and called the method in each of our controller tests which were failing.

# test/support/helper_methods.rb
def ensure_consistent_test_adapter_is_used
  # This is a hack mentioned here: https://github.com/bensheldon/good_job/issues/846#issuecomment-1432375562
  # The actual issue is in Rails for which a PR is pending merge
  # https://github.com/rails/rails/pull/48585
  (ActiveJob::Base.descendants + [ActiveJob::Base]).each(&:disable_test_adapter)
end

# test/controllers/exports_controller_test.rb
class ExportsControllerTest < ActionDispatch::IntegrationTest
  def setup
    ensure_consistent_test_adapter_is_used
    # ...
  end

  # ...
end

8. Monitoring

Basecamp has released mission_control-jobs, which can be used to monitor background jobs. It is generic, so it can be used with any compatible ActiveJob adapter.

Add gem "mission_control-jobs" to your Gemfile and run bundle install.

Mount the mission control route in your routes.rb file.

# routes.rb
Rails.application.routes.draw do
  # ...
  mount MissionControl::Jobs::Engine, at: "/jobs"

By default, mission control would try to load the adapter specified in your application.rb or individual environment-specific files. Currently, Sidekiq isn't compatible with mission control, so you will face an error while loading the dashboard at /jobs. The fix is to explicitly specify solid_queue to the list of mission control adapters.

# application.rb
# ...
config.mission_control.jobs.adapters = [:solid_queue]

Now, visiting /jobs on your site should load a dashboard where you can monitor your Solid Queue jobs.

But that isn't enough. There is no authentication. For development environments, it is fine, but the /jobs route would be exposed on production too. By default, Mission Control's controllers will extend the host app's ApplicationController. If no authentication is enforced, /jobs will be available to everyone.

To implement some kind of authentication, we can specify a different controller as the base class for Mission Control's controllers and add the authentication there.

# application.rb
# ...
MissionControl::Jobs.base_controller_class = "MissionControlController"

# app/controllers/mission_control_controller.rb
class MissionControlController < ApplicationController
  before_action :authenticate!, if: :restricted_env?

  private

    def authenticate!
      authenticate_or_request_with_http_basic do |username, password|
        username == "solidqueue" && password == Rails.application.secrets.mission_control_password
      end
    end

    def restricted_env?
      Rails.env.staging? || Rails.env.production?
    end
end

Here, we have specified that MissionControlController would be our base controller for mission control related controllers. Then in MissionControlController we implemented basic authentication for staging and production environments.

Observations

We haven't had any complaints so far. Solid Queue offers simplicity, requires no additional infrastructure and provides visibility for managing jobs since they are stored in the database.

In the coming days, we will migrate all of our 22 Neeto products to Solid Queue. And once cron-style job support lands in Solid Queue, we will completely migrate from Sidekiq.

In this blog post, we'll delve into the concept of rate limiting, exploring its significance and implementation in Rails 8.0.

The Need for Rate Limiting

It's crucial in ensuring the following:

• Security Shield: Guards against Denial-of-Service (DoS) attacks, thwarting malicious attempts to flood your app and crash it.
• Resource Balance: Prevents resource abuse by capping heavy users, ensuring fair resource distribution and maintaining optimal app performance.
• Brute-Force Defender: Stalls hackers attempting password guesses or exploiting vulnerabilities through relentless, repeated attempts.

How Rack::Attack Paved the Way

In the pre-Rails 8.0 era, the go-to solution for rate limiting was the rack-attack gem. This gem allowed developers to set up rate-limiting rules by adding custom code in the rack_attack.rb file. While effective, this approach required manual intervention for each endpoint and could become cumbersome in a growing application. This external dependency brought its own set of challenges, such as the need for custom code and constant vigilance over rate-limited endpoints.

Rails 8.0: Native Rate Limiting

Rails 8.0 brings a native rate-limiting feature to the Action Controller, streamlining the process and eliminating the need for external gems. Developers can now set rate limits directly within their controllers using the rate_limit method.

Let's take a look at the usage:

Define Limits

Utilize the rate_limit method within your controller actions, specifying the maximum allowed requests and the corresponding timeframe. The rate_limit method accepts the following parameters:

• to: The maximum number of requests allowed, beyond which the rate limiting error will be raised, that is with a 429 Too Many Requests response.
• within: The maximum number of requests allowed in a given time window.
• only: Which all controller actions ought to be rate limited.
• except: Which all controller actions to be omitted from rate limiting.

An example:

class SignupController < ApplicationController
  rate_limit to: 4, within: 1.minute, only: :create

  def create
    # ...
  end
end

Granular Control

Target specific actions or employ custom logic for nuanced control. For instance, you can limit requests based on domain instead of IP address.

rate_limit to: 4, within: 1.minute, by: -> { request.domain }, only: :create

Tailored Responses

By default, rate-limited requests receive a 429 Too Many Requests error. However, you can personalize the response for a more informative user experience.

rate_limit to: 4, within: 1.minute, with: -> { redirect_to(ip_restrictions_controller_url), alert: "Signup Attempts failed four times. Please try again later." }, only: :create

Custom Cache Stores

The Rails 8.0 rate-limiting implementation at the moment allows us to make use a wide range of backends as its store.

It includes:

• Memcached
• Redis (including alternative Redis clients like Dalli)
• Database-backed stores
• File-based stores

The Rate Limiting API seamlessly integrates with Rails' caching mechanisms, leveraging ActiveSupport::Cache stores. Developers can specify custom cache stores if they require separate handling for rate limits compared to other cache data. This integration ensures efficient storage and retrieval of rate limit data, optimizing performance. An example:

class SignupController < ApplicationController
  RATE_LIMIT_STORE = ActiveSupport::Cache::RedisCacheStore.new(url: ENV["REDIS_URL"])
  rate_limit to: 8, within: 2.minutes, store: RATE_LIMIT_STORE
end

What to look forward to

At the moment the built-in rate limiting feature is limited in its extensibility. There are cases where this feature cannot be used as a direct replacement for logic that's based on rack-attack gem. A good example is a scenario where we might want to use an exponential backoff based rate limiting algorithm, which can be implemented using rack-attack gem but not with the built-in Rails feature at the moment.

Right now, we are constrained to stick to the default cache counter algorithm that Rails comes shipped with. In the future, we can expect a more generic limiter interface, allowing users to explore advanced rate-limiting algorithms such as exponential backoff, leaky bucket, or token bucket, etc. This change would open the door for developers to swap the implementation based on their specific requirements.

Please check out the following pull requests for more details:

• https://github.com/rails/rails/pull/50490
• https://github.com/rails/rails/pull/50781

We are building neeto, and our technology stack is quite simple. On the front end, we use React.js. On the backend we use Ruby on Rails, PostgreSQL, Redis and Sidekiq.

The term globalProps might not ring a bell for most people. It was coined by the BigBinary team for our internal use. globalProps is data that is directly retrieved from our backend and assigned to the browser global object that's window. To view the global props, we can type globalProps in the browser console, which prints out useful information set by the backend service.

How is globalProps implemented

To understand where the globalProps came from and how it works, we need to examine the React-Rails gem. It uses Ruby on Rails asset pipeline to automatically transform JSX into Ruby on Rails compatible assets using the Ruby Babel transpiler.

react_component helper method takes a component name as the first argument, props as the second argument and a list of HTML attributes as the third argument. The documentation has more details.

<%= react_component("App", get_client_props, { class: "root-container" }) %>

Limitations of the default behavior

In the react-rails gem, the hash is set as the props of the component specified in the react_component method by default. In the example above, the hash returned by the get_client_props method is passed as props to the App component in the front end.

The limitation of this approach is that we need to pass down globalProps through all the components by prop-drilling or React Context.

The concept of nanos

At neeto, anything that does not contain product-specific business logic and can be extracted into a reusable tool is extracted into an independent package. We call them nanos.

You can read more about it in our blog on how nanos make Neeto better.

Limitations in accessing the props by nanos and utility functions

The issue with the above approaches in handling the props is that it won't be directly available in utility functions or nano. We explicitly need to pass it as arguments to utility functions after prop drilling. If we use React Context, it can only be accessed in React components or hooks, it cannot be accessed in utility functions. Also, we cannot directly obtain the reference of the Context within the nanos.

Why we didn't chose environment variables

Some of the variables inside the globalProps are environment variables, which is usually accessed as process.env.VARIABLE_NAME. If we set environment variables, they will be hardcoded into the JavaScript bundle at the time of bundling. This implies that whenever we need to change the environment variable, we must trigger a redeployment.

The solution is globalProps

The advantage of globalProps over these approaches is, it's accessible everywhere since it's in the browser global object window. All the nanos and utility functions that we integrate into the application have seamless access to the props without any extra step of wiring.

Seeding the hash at the backend into the browser's global object, window

Seeding the hash at the backend into the browser's global object, window, is accomplished using the above-mentioned helper method, react_component. As we discussed earlier, an HTML node is created that contains data-react-class representing the component name anddata-react-props attribute representing the hash we passed from the backend as an HTML-encoded string.

Decode the HTML-encoded string into JavaScript object

The next step is to decode the HTML-encoded string and parse it into a JavaScript object. The hash is read from the root-container HTML node and parsed into a JavaScript object.

const rootContainer = document.getElementsByClassName("root-container")[0];

const reactProps = JSON.parse(rootContainer?.dataset?.reactProps || "{}");

Convert the case of the keys in the object

The JavaScript object that we have obtained have the keys in snake case. We will convert them into camel case using the helper method keysToCamelCase.

We convert the case because React prefers camel case keys, while Rails prefers snake case keys.

const globalProps = keysToCamelCase(reactProps);

Deepfreeze the global props

Additionally, we take an extra step to deep-freeze the global props object, ensuring immutability. The helper function used here is deepFreezeObject. This prevents modifications to global props from within the product and thus ensures data integrity when working with different nanos. All these steps are performed before the initial rendering of the React component.

window.globalProps = globalProps;

deepFreezeObject(window.globalProps);

Let's see the final codeblock that seeds the hash at the backend to the browser's global object.

export default function initializeGlobalProps() {
  const rootContainer = document.getElementsByClassName("root-container");
  const htmlEncodedReactProps = rootContainer[0]?.dataset?.reactProps;

  const reactProps = JSON.parse(htmlEncodedReactProps || "{}");
  const globalProps = keysToCamelCase(reactProps);

  window.globalProps = globalProps;
  deepFreezeObject(window.globalProps);
}

If we take a closer look at the content of globalProps, we can see that it carries a lot of data. As discussed earlier this data is useful to other nanos. Some of the data being passed are appName, honeyBadgerApiKey, organization, user info, etc.

Recently, 37signals open sourced Solid Queue.

Solid Queue is database based queuing backend for Active Job. In contrast Sidekiq and Resque are Redis-based queuing backends.

In her blog Rosa Gutiérrez mentioned following lines which captured our attention.

In our case, one feature that PostgreSQL has had for quite some time and that was finally introduced in MySQL 8, has been crucial to our implementation:

SELECT ... FOR UPDATE SKIP LOCKED

This allows Solid Queue’s workers to fetch and lock jobs without locking other workers.

As per her, this feature had been in PostgreSQL for a while, and now this feature has landed in MySQL making it possible to build Solid Queue.

We had never heard of UPDATE SKIP LOCKED feature either in PostgreSQL or in MySQL. We were wondering what is this UPDATE SKIP LOCKED without which it was not possible to build Solid Queue. So we decided to look into it.

Processing jobs from a queue

Consider a case where we need to build a system where a bunch of jobs need to be processed in the background.

There are a bunch of workers waiting to grab a job and start processing the moment a job becomes available. The challenge is when multiple workers attempt to claim the same job simultaneously how do we ensure that only one of the workers claims the job for processing. At any point of time, a worker should claim only the "unclaimed" job, and an "unclaimed" job should be claimed by one and only one worker.

Here is how one might go about it implementing it.

START TRANSACTION
SELECT * FROM JOBS WHERE processed='no' LIMIT 1;
-- Process the job
COMMIT;

With the above code, it's possible that two workers might claim the same job.

One way to resolve this issue is by marking a particular row as locked for update.

START TRANSACTION;
SELECT * FROM JOBS WHERE processed='no' FOR UPDATE LIMIT 1;
-- Process the job
COMMIT

SELECT ... FOR UPDATE locks a particular row, and hence no one else can lock that record.

As soon as a new job comes in, multiple workers will execute the above query and will try to take a lock on that record. The database will ensure that only one of the workers gets the lock.

The first worker will take the lock on the record using FOR UPDATE. When other workers come to that record and they see that there is a lock FOR UPDATE, they will wait for the lock to be lifted. Yes, these workers will wait until the lock is released.

The lock will only be released when the transaction is committed. When the transaction is committed and the lock is released, then other workers will get hold of the record only to find that the job has already been processed. As you can see, this is a highly inefficient process.

That is where FOR UPDATE SKIP LOCKED comes in.

SKIP LOCKED skips locked rows

START TRANSACTION;
SELECT * FROM jobs_table FOR UPDATE SKIP LOCKED LIMIT 1;
-- Process the job
COMMIT;

Imagine the same scenario here. A job comes in. Multiple workers compete to claim the job. The database ensures that only one worker gets the lock. However in this case the other workers will move on to the next record. They will not wait. That's what SKIP LOCKED does.

MySQL has detailed documentation on how SKIP LOCKED works If you want to read in more detail.

Solid Queue uses FOR UPDATE SKIP LOCKED feature to ensure that a job is claimed by only one worker.

How GoodJob manages job processing without SKIP LOCKED

GoodJob burst into the scene around July, 2020. GoodJobs supports only PostgreSQL database because it uses advisory locks to guarantee that no two workers claim the same job.

PostgreSQL folks understand that the lock mechanism provided by the database would not satisfy all the variety of cases that might arise in an application. Advisory locks are a mechanism that allows applications to establish a communication channel to coordinate actions between different sessions or transactions. Unlike regular row-level locks enforced by the database system, advisory locks are implemented as a set of low-level functions that applications can use to acquire and release locks based on their requirements. We can read more about it here.

pg_advisory_lock function will lock the given resource. However, if another session already holds a lock on the same resourc,e then this function will wait. This is similar to the FOR UPDATE case we saw above.

However pg_try_advisory_lock function will either obtain the lock immediately and return true, or return false if the lock cannot be acquired immediately. As you can see, the name has the word try. This function attempts to acquire a lock. If it can't get the lock, then it won't wait. Now this function can be utilized to build a queuing system.

Any usage of an advisory lock means the application needs to coordinate action. It gives more power to the application but it also means more work by the application. In contrast, FOR UPDATE SKIP RECORD is natively supported by PostgreSQL.

Based on the discussions here and here, it seems GoodJob is evaluating the possibility of migrating from advisory locks to using FOR UPDATE SKIP LOCKED for better performance. Going through these issues was quite revealing and I got to learn a lot about things I was unaware of.

Delayed job implementation

DelayedJob has been there since 2009, long before Sidekiq. It doesn't use SKIP LOCK. Instead it uses a row level locking system by updating a field in the job record to indicate that the job is being processed. In short DelayedJob ensures that no two workers take the same job at the application level without taking any help in this direction from the database.

What about SQLite

So far, we discussed PostgreSQL and MySQL. What about SQLite? Does it support SKIP LOCK. No, it doesn't support it, but it's ok. As per the documentation, it supports only writer at any instant in time.

High Concurrency

SQLite supports an unlimited number of simultaneous readers, but it will only allow one writer at any instant in time. For many situations, this is not a problem. Writers queue up. Each application does its database work quickly and moves on, and no lock lasts for more than a few dozen milliseconds. But there are some applications that require more concurrency, and those applications may need to seek a different solution.

NOWAIT

For completeness, let's discuss the NOWAIT feature. We saw earlier that if we take a lock on a row using FOR UPDATE, then other workers will wait until the lock is released.

START TRANSACTION;
SELECT * FROM JOBS WHERE processed='no' FOR UPDATE NOWAIT LIMIT 1;
-- Process the job
COMMIT

NOWAIT feature allows other transactions to not wait for the lock to be released. In this case if a transaction is not able to get a lock on the given row then it will raise an error and the application needs to handle the error.

In contrast, SKIP LOCKED will allow the transaction to move on to the next row if a lock is already taken.

Redis backed queue vs database backed queue

Now that we looked at how FOR UPDATE SKIP LOCK helps build queuing system using database itself, let's see some pros and cons of each type of queuing system.

Simplicity and familiarity

Database-backed queues are often simpler to set up and manage, especially if your application is already using a relational database. There's no need for an additional dependency like Redis.

No Additional Infrastructure

Since the job information is stored in the same database as your application data, you don't need to set up and maintain a separate infrastructure like a Redis server.

Transactionality

Database-backed queues can leverage database transactions, ensuring that both the job creation and any related database operations are committed or rolled back together. This can be important in scenarios where data consistency is critical.

Modifiability

It is easier to modify the jobs stored in the database than Redis, but doing so requires caution, and it's generally not recommended. In Redis, jobs are often stored as serialized data, and modifying them directly is not as straightforward or common. Redis provides commands to interact with data, but modifying job data directly is not a standard practice and could result in data corruption.

One of the recent additions to this is the transaction.active_record event That is triggered when Active Record-managed transactions occur. This is particularly useful if you want to build a monitoring system like NewRelic, where you need to track and analyze database transactions for performance monitoring and optimization purposes.

The event payload contains the connection, outcome, and the timing details. The connection helps us identify the database where the transaction occurred, which is particularly valuable in a multi-database environment. The outcome, which may be one of the following: :commit, :rollback, :restart, or :incomplete signifies the transaction's result.

To make use of this, we can subscribe to the event in an initializer config/initializers/events.rb like this.

ActiveSupport::Notifications.subscribe(
  "transaction.active_record"
) do |event|
  MetricsLogger.record_transaction(event.payload)
end

In the above example, MetricLogger is responsible for recording the transaction details. It then analyzes and reports slow transactions so that proper action can be taken. You can modify this to suit your instrumentation needs.

Please check out these pull requests - 1, 2 for more details.

A CDN is a geographically distributed group of servers that cache content close to end users. Let's say that a user from London is hitting a website that has a JavaScript file hosted in Chicago. That JavaScript file has to travel all the way from Chicago to London. This means the site will load slowly.

A CDN will have a copy of that JavaScript file in London itself. In this way, for that user, that JavaScript file will be loaded from the London server and not from the Chicago server. This is what a CDN primarily does.

Here's how we set up Cloudflare as a CDN for our Rails applications.

1. Configure CNAME record for CDN subdomain in Cloudflare

We need a subdomain that will act as a CDN. For NeetoCal, we use https://cdn.neetocal.com as the subdomain. You can use any subdomain you want.

To set up this subdomain, follow these steps.

1. Add a new record with the type CNAME.
2. In the name field, enter *.
3. In the target field, enter the URL where the Rails app is deployed.

If we already have a CNAME record for * that handles all the subdomains, then we don't need to create a new one.

2. Turn on the Proxy feature in Cloudflare

In the previous step, when creating a new CNAME record, by default, Cloudflare will enable proxy for each record.

Once the Proxy status column is turned on, it will display 'Proxied' as shown below.

When we turn on "Proxy", then the DNS queries will resolve to the Cloudflare IP address instead of the target. It means Cloudflare will get all the requests and then Cloudflare in turn will forward those requests to the target.

Let's say that a user from Paris loads a webpage which is hosted in Chicago and Cloudflare has a server in Paris. When "Proxy" is turned on, then when the user makes the first request to the server, that request will go to Cloudflare. Cloudflare doesn't have the JavaScript file, so Cloudflare will forward that request to the server. The server will send that JavaScript file to Cloudflare. Cloudflare will cache this JavaScript file and serve the JavaScript file to the user.

Two seconds later, another user from a different part of Paris hits the same webpage. This time, Cloudflare has a cached copy of the JavaScript file and Cloudflare will serve the JavaScript from its server in Paris.

This has two benefits. The first benefit is that the user gets to see the webpage really fast. The second benefit is that the server cost is a lot lower since fewer requests need to be served by the server in Chicago.

What all things Cloudflare can cache

Cloudflare can only cache files that are stored on the domain that is proxied through Cloudflare. If we have an image file that is stored on Amazon S3 and is accessed via a direct S3 URL, then Cloudflare won't be able to cache it.

Cloudflare caches are based on file extensions, not MIME types. Here is the list of file extensions that will be cached by default. HTML is not on this list.

Cloudflare allows us to cache other file types by setting up suitable page rules.

3. Set appropriate Cache-Control headers in Rails

When utilizing Cloudflare as a CDN, it's crucial to configure our Rails app to serve assets with specific Cache-Control headers. This ensures optimal caching behavior for resources.

The Rails app should serve assets with a header similar to the one given below.

cache-control: public, max-age=31536000

The cache-control must be public, and max-age must be a positive value for Cloudflare to cache the resource.

For this, we need to modify the Rails configuration file config/environments/production.rb.

config.public_file_server.headers = {
  'Cache-Control' => 'public, max-age=31536000'
}

Here, Cache-Control is set to 'public', allowing any cache (including Cloudflare) to store the resource. The max-age is specified as 31,536,000 seconds (one year), indicating how long the cache should retain the object before considering it stale.

4. Configure asset_host in Rails.

We added the secrets for the asset_host in our Rails application's config/secrets.yml file.

production:
  asset_host: <%= ENV["ASSET_HOST"] %>

We also added an environment variable called ASSET_HOST that contains our new CDN host URL.

ASSET_HOST=cdn.neetoX.com

Once this is done all asset URLs will point to https://cdn.neetoX.com.

5. Verify changes

Once all the setup is done, it can take a few hours for the changes to be reflected. Hit the page and see the response header of .js file.

If the response header cf-cache-status had a value of HIT, then that means Cloudflare is serving that asset from its cache.

Prior to Rails 7.1, if you had your routes configured to be scoped under say, user_id as shown below, you would have to explicitly specify the user_id in every single place where you want to generate a link for the scoped routes, such as when writing view files.

#routes.rb
Rails.application.routes.draw do
  root "articles#index"

  scope ":user_id" do
    get "/articles", to: "articles#index"
    get "/articles/:id", to: "articles#show"
  end

  get "/categories", to: "categories#index"
end

<!-- app/views/articles/index.html.erb -->

<a href="<%= articles_path(user_id: @current_user.id) %>"> Articles </a>

This could be solved by updating ApplicationController and overwriting the default_url_options method:

# application_controller.rb
class ApplicationController < ActionController::Base
  def default_url_options
    { user_id: "unique-id" }
  end
end

The default_url_options method is used to overwrite and set default options for all the methods based on url_for. However, this meant that all routes, even those that did not belong to the user_id scope, would have the ?user_id=unique-id query param added to the end of them, resulting in the following output:

articles_path # => /unique-id/articles
categories_path # => /categories?user_id=unique-id

Rails 7.1 fixes this issue with the addition of the path_params option. If you pass a hash of parameters to this key, those parameters will only be used for the named segments of the route. If they aren't used, they are discarded instead of being appended to the end of the route as query params.

With this change, we can implement the default_url_options method as follows:

class ApplicationController < ActionController::Base
  def default_url_options
    { path_params: { user_id: "unique-id" } }
  end
end

The url_for helper method will now give you the following output:

articles_path # => /unique-id/articles
articles_path(user_id: "test-id") # => /test-id/articles
categories_path # => /categories
categories_path(user_id: "test-id") # => /categories

The view file can now be written as:

<a href="<%= articles_path %>"> Articles </a>

This is very useful in situations where you only want to add a required param that is part of the route's URL without introducing unnecessary query params for other routes.

Please check out this pull request for more details.

Before moving on to the configuration changes, let us understand what a PRAGMA is. PRAGMA is a special kind of SQL statement(only available in SQLite) which is used to query or manipulate various aspects of the database's behaviour and configuration.

Configuration Changes

1. journal_mode is now set use WAL instead of using the rollback journal

journal_mode is a configuration setting in SQLite that determines how the database handles transactions and maintains data integrity in the face of system crashes or unexpected shutdowns. Previously, a brand new Rails application used to use the rollback journal with DELETE journal mode. Now it uses a much more efficient journaling method named Write-Ahead Logging (WAL). Let us understand the difference between them.

Rollback journal: In this implementation, the database engine first records the original unchanged database content in a rollback journal, and the writes are made directly to the database file. If the system crashes, the rollback journal can be used to restore the database to the state it was in before the transaction began. The issue with this approach is that a writer can alter the database or readers can read from the database — but not both at the same time.

Write-ahead logging: In WAL mode, SQLite maintains a separate write-ahead log. Instead of writing directly to the database file, changes are written to the log first. When a reader needs a page of content, it first checks the WAL to see if the page appears there, and if so it pulls in the latest copy of the page in the WAL. If no copy of the page exists in the WAL, then the page is read from the original database file. So this means readers and writers can work together and there is no contention.

How is the database file made up-to-date with the WAL file?

SQLite routinely moves the WAL file transactions back into the database. This process is called checkpointing. By default, SQLite does a checkpoint automatically when the WAL file reaches a threshold size of 1000 pages.

If a system crash occurs, the last commit record has not been written to the WAL file. Without the commit record, the new data is not considered valid, and the database simply ignores it.

WAL is a better choice for web applications because of the increased concurrency it offers.

Here is a nice article on how WAL works under the hood.

2. synchronous is now set to NORMAL instead of FULL

The synchronous pragma controls how and when SQLite flushes content to disk. The two common options are FULL and NORMAL, which map to “sync on every write” and “sync every 1000 written pages” respectively. FULL synchronous is very safe but slow. When synchronous is NORMAL, the database engine will sync at the most critical moments, but less often than in FULL mode. We trade an aggressive approach to durability for speed.

The SQLite documentation suggests using NORMAL for applications running in WAL mode.

3. journal_size_limit is now capped at 64MB

The journal_size_limit pragma tells SQLite how much of the write-ahead log data to keep in the on-disk file. Previously, it was set to -1 which means there was no limit set on the journal size, which allows it to grow unbounded, thereby potentially affecting read performance. Now it is capped at an appropriate size of 64MB.

4. cache_size is now set to 8MB

The cache_size pragma sets the maximum number of database disk pages that SQLite will hold in memory at once, per open database file. The default value was -2000 i.e 2000 bytes. Please note that SQLite interprets a negative value as a byte limit and positive number as a page limit. Now the cache_size is set to 2000(pages) with a default page size of 4096 bytes, which means the cache limit is ~8MB.

5. mmap_size is now set to 128MB

The mmap_size pragma sets the maximum number of bytes that are set aside for memory-mapped I/O on a single database. Let us first understand what memory-mapped I/O is.

Memory-mapped (mmap) I/O is an OS-provided feature that maps the contents of a file on secondary storage into a program’s address space. The program then accesses pages via pointers as if the file resided entirely in memory. The OS transparently loads pages only when the program references them and automatically evicts pages if memory fills up. The advantage of using mmap is that it bypasses the step where we need to copy the pages from secondary to primary storage, thereby making it faster.

How is memory-mapped I/O implemented in SQLite?

SQLite accesses and updates database files using xRead() and xWrite() methods by default. These methods are typically implemented as read() and write() system calls, which cause the OS to copy disk content between the kernel buffer cache and user space. SQLite also has the option of accessing disk content directly using memory-mapped I/O via the xFetch() and xUnfetch() methods. Using the legacy xRead() method in SQLite, a page-sized heap memory block is allocated, and the xRead() call copies the entire database page content into this allocated memory. Whereas if memory-mapped I/O is enabled, it calls the xFetch() method. The xFetch() method asks the operating system to return a pointer to the requested page. If the requested page has been or can be mapped into the application address space, then xFetch() returns a pointer to that page for SQLite to use without having to copy anything. Skipping the copy step is what makes memory-mapped I/O faster.

The mmap_size is the maximum number of bytes of the database file that SQLite will try to map into the process address space at one time. Now it is set to 128MB.

With these changes, there is a considerable improvement in performance. That being said, SQLite makes a strong case for single-node production applications, as it is highly performant, especially when used in conjunction with NVMe disks.

Please check out this pull request for more details.

Before Rails 7.1

In Rails versions prior to 7.1, when batch processing records with a composite primary key, like id_1 and id_2, developers could use the :asc or :desc argument to control the sorting order. However, there was a limitation in how this sorting worked. When you specified the sorting order using :asc or :desc, it affected both id_1 and id_2 simultaneously. In other words, if you requested ascending order, both id_1 and id_2 would be sorted in ascending order together. Similarly, if you requested descending order, both id_1 and id_2 would be sorted in descending order together.

This limitation had practical implications, especially when you needed to sort records by different criteria for each part of the composite primary key.

After Rails 7.1

With the new enhancement in Rails 7.1, developers can now select the sorting order for each key within a composite primary key. Let's see this with an example.

Consider a scenario where you have a Product model with a composite primary key, category_id and product_id. You want to fetch products in descending order of category_id and ascending order of product_id. With Rails 7.1, this becomes straightforward:

class Product < ActiveRecord::Base
  self.primary_key = [:category_id, :product_id]
end

# Retrieving products in descending order of
# category_id and ascending order of product_id
Product.find_each(order: [:desc, :asc]) do |product|
  # Your processing logic for each product goes here
end

The find_each method is a part of ActiveRecord::Batches and is used for efficient batch processing of records from the database. It retrieves records in small batches, reducing memory consumption and improving performance. The method takes an optional order argument, which, as of Rails 7.1, can accept an array of symbols to specify the sorting order for each key within the composite primary key.

The enhancement for specifying sorting orders for composite primary keys is not limited to find_each. It applies to other batch processing methods provided by ActiveRecord::Batches, such as find_in_batches and in_batches. These methods allow you to retrieve and process records in batches efficiently, just like find_each.

Conclusion

In Rails 7.1, the support for multiple-column ordering in batches for models with composite primary keys brings more flexibility and control to your application's data retrieval process. Now you can tailor the sorting of composite keys to match your specific needs, providing a more powerful and versatile toolset for your Rails development endeavors.

Please check out this pull request for more details.

class UserMailer < ApplicationMailer
  default from: 'notifications@neeto.com'

  def welcome_email
    @user = params[:user]
    mail(to: @user.email, subject: 'Welcome to neeto')
  end
end

When the mail method is called in the above code, it just renders the mail template from a view. The mail is actually not sent. The actual delivery may happen synchronously or asynchronously. To send the mail, we need to call one of many deliver methods.

UserMailer.with(user: @user).welcome_email.deliver_later

Before Rails 7.1, we did not have any callbacks around the deliver methods to execute code around the delivery lifecycle. You would need to use interceptors and observers to hook into the mail delivery lifecycle. Say, you need to send emails to only a few allowed email addresses in the staging environment. You would need to use an interceptor and register it to Action Mailer.

# interceptors/staging_email_interceptor.rb
module Interceptors
  class StagingEmailInterceptor
    def self.delivering_email(mail)
      if rails.env.staging? && !allowed_emails.include?(mail.to)
        mail.perform_deliveries = false
      end
    end
  end
end

# config/initializers/action_mailer.rb
ActionMailer::Base.register_interceptor(Interceptors::StagingEmailInterceptor)

The new before_deliver callback allows you to handle this situation without using the Interceptors or Observers. You would just need to include the following in UserMailer.

class UserMailer < ApplicationMailer
  default from: 'notifications@neeto.com'
  before_deliver :filter_allowed_emails

  def welcome_email
    @user = params[:user]
    mail(to: @user.email, subject: 'Welcome to neeto')
  end

  private

    def filter_allowed_emails
      if rails.env.staging? && !allowed_emails.include?(mail.to)
        mail.perform_deliveries = false
      end
    end
end

Similarly, suppose you want to update the mail_delivered_at attribute of the user instance, you would have to use an observer and register it like so:

# observers/staging_email_interceptor.rb
module Observers
  class SetDeliveredAtObserver
    def self.delivered_email(mail)
      user = User.find_by(email: mail.to)
      user.update(mail_delivered_at: mail.date)
    end
  end
end

# config/initializers/action_mailer.rb
ActionMailer::Base.register_interceptor(Interceptors::StagingEmailInterceptor)
ActionMailer::Base.register_observer(Observers::SetDeliveredAtObserver)

With the new after_deliver callback this becomes as simple as defining a method in UserMailer.

class UserMailer < ApplicationMailer
  default from: 'notifications@neeto.com'
  before_deliver :filter_allowed_emails
  after_deliver :set_delivered_at

  def welcome_email
    @user = params[:user]
    mail(to: @user.email, subject: 'Welcome to neeto')
  end

  private

    def filter_allowed_emails
      if rails.env.staging? && !allowed_emails.include?(mail.to)
        mail.perform_deliveries = false
      end
    end

    def set_delivered_at
      @user.update(mail_delivered_at: mail.date)
    end
end

An important thing to keep in mind is the order of execution of the callbacks.

• before_action
• after_action
• before_deliver
• after_deliver

This makes sense as the deliver callbacks only wrap around the deliver methods while the action callbacks wrap around the mail render method.

Please check out this pull request for more details.

Understanding the EXPLAIN method

Before diving into the new options available in Rails 7.1, let's quickly recap the purpose and usage of the explain method in Active Record. The explain method is used to retrieve the execution plan of an SQL query chosen by the database optimizer. It provides insight into how the database intends to execute the query, including the sequence of operations, indexes used, and estimated costs.

By analyzing the query plan, we can identify potential performance bottlenecks, optimize database schema design, and fine-tune queries for better efficiency. However, prior to Rails 7.1, the level of detail available in the explain output was limited.

Detailed query plan analysis with options

In Rails 7.1, the explain method accepts options, enabling us to customize the output and obtain a more detailed query plan analysis. It's important to note that these options are the same ones that are already available in native SQL. Options may vary depending on the database used. In this blog post, we will focus on examples using PostgreSQL. Let us take a look at some of the available options.

ANALYZE

The analyze option causes the statement to be actually executed, not only planned. Then actual run time statistics are added to the display, including the total elapsed time expended within each plan node and the total number of rows it actually returned.

Service.where('age > ?', 25).joins(:user).explain(:analyze)

EXPLAIN (ANALYZE) SELECT "services".* FROM "services" INNER JOIN "users" ON "users"."id" = "services"."user_id" WHERE (age > 25)
                          QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------
 Hash Join  (cost=15.50..29.42 rows=103 width=232) (actual time=0.015..0.017 rows=0 loops=1)
   Hash Cond: (services.user_id = users.id)
   ->  Seq Scan on services  (cost=0.00..13.10 rows=310 width=232) (actual time=0.012..0.012 rows=0 loops=1)
   ->  Hash  (cost=14.12..14.12 rows=110 width=8) (never executed)
         ->  Seq Scan on users  (cost=0.00..14.12 rows=110 width=8) (never executed)
               Filter: (age > 25)
 Planning Time: 0.915 ms
 Execution Time: 0.472 ms

VERBOSE

The verbose option provides a more detailed output by including additional information about each step in the execution plan. This includes statistics, cost estimates, and other relevant details.

Service.where('age > ?', 25).joins(:user).explain(:verbose)

EXPLAIN (VERBOSE) SELECT "services".* FROM "services" INNER JOIN "users" ON "users"."id" = "services"."user_id" WHERE (age > 25)
                          QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------
 Hash Join  (cost=15.50..29.42 rows=103 width=232)
   Output: services.id, services.user_id, services.provider, services.uid, services.access_token, services.access_token_secret, services.refresh_token, services.expires_at, services.auth, services.created_at, services.updated_at
   Inner Unique: true
   Hash Cond: (services.user_id = users.id)
   ->  Seq Scan on public.services  (cost=0.00..13.10 rows=310 width=232)
         Output: services.id, services.user_id, services.provider, services.uid, services.access_token, services.access_token_secret, services.refresh_token, services.expires_at, services.auth, services.created_at, services.updated_at
   ->  Hash  (cost=14.12..14.12 rows=110 width=8)
         Output: users.id
         ->  Seq Scan on public.users  (cost=0.00..14.12 rows=110 width=8)
               Output: users.id
               Filter: (users.age > 25)
(11 rows)

For more options available in PostgreSQL's EXPLAIN command, you can refer to the official documentation.

Please check out this pull request for more details.

Before Rails 7.1, you could normalize attributes using before_save callback.

model User < ApplicationRecord
  before_save :downcase_email, if :email_present?

  private

    def email_present?
      email.present?
    end

    def downcase_email
      email.downcase!
    end
end

In Rails 7.1, you can refactor the above to the code below.

model User < ApplicationRecord
  normalizes :email, with: -> email { email.downcase }
end

The normalization is applied when the attribute is assigned or updated, and the normalized value will be persisted to the database. The normalization is also applied to the corresponding keyword argument of finder methods. This allows a record to be created and later queried using unnormalized values.

By default, normalization is not applied to nil values. To normalize nil value, you can enable it using :apply_to_nil option.

model User < ApplicationRecord
  normalizes :user_name, with:
    -> user_name { user_name.parameterize.underscore }

  normalizes :email, with: -> { _1.strip.downcase }

  normalizes :profile_image, with:
    -> profile_image {
      profile_image.present? ? URI.parse(profile_image).to_s :
      "https://source.boringavatars.com/beam" },
    apply_to_nil: true
end

# rails console
>> User.create!(user_name: "Eve Smith", email: "eveSmith@EXAMPLE.com")

#<User:0x000000010b757090 id: 1, user_name: "eve_smith", profile_image:"https://source.boringavatars.com/beam", email: "evesmith@example.com", created_at: Wed, 03 May 2023 07:49:20.067765000 UTC +00:00, updated_at: Wed, 03 May 2023 07:49:20.067765000 UTC +00:00>

>> user = User.find_by!(email: "EVESMITH@example.COM")
>> user.email # => "evesmith@example.com"

>> User.exists?(email: "EveSmith@Example.COM")          # => true

If a user's email was already stored in the database before the normalization statement was added to the model, the email will not be retrieved in the normalized format.

This is because in the database it's stored in mixed case, that is, without the normalization. If you have legacy data, you can normalize it explicitly using the Normalization#normalize_attribute method.

# rails console
>> legacy_user = User.find(1)
>> legacy_user.email  # => "adamSmith@EXAMPLE.com"
>> legacy_user.normalize_attribute(:email)
>> legacy_user.email  # => "adamsmith@example.com"
>> legacy_user.save

Please check out this pull request for more details.

[ActiveJob] Enqueued NotifySubscribersJob (Job ID: 5945980f-303e-4c3f-af5b-e22be170f7c5) to Sidekiq(default)
[ActiveJob] ↳ app/models/post.rb:14 in `notify_subscribers`

By examining the logs above, we can determine that the NotifySubscribersJob was enqueued from the notify_subscribers method in the Post model, providing us with a clear picture of the job's origin.

To enable this functionality, we need to set the following configuration in config/environments/development.rb.

config.active_job.verbose_enqueue_logs = true

However, Rails 7.1 ships with this configuration set by default.

It's important to note that using verbose enqueue logs in a production environment is not recommended, as it uses the Kernel#caller method to retrieve the execution stack, which can be slow.

Please check out this pull request for more details.

For example, if we want to send a welcome email to multiple users, we can do this:

welcome_email_jobs = users.map do |user|
  WelcomeEmailJob.new(user)
end

ActiveJob.perform_all_later(welcome_email_jobs)

The benefit of doing it this way, rather than looping through the user records and using perform_later is that perform_all_later cuts down on the number of round-trips to the queue datastore. That means reducing Redis round-trip latency if the queuing backend is Sidekiq. Whereas if we're using a queuing backend like GoodJob, which is backed by Postgres, perform_all_later will enqueue all the jobs using a single INSERT statement which is more performant.

Please note that if the queuing backend doesn't support bulk enqueuing, perform_all_later will fallback to enqueuing each job individually.

Active Job is designed to abstract away the differences between different job processing libraries and to provide a unified interface. It is made possible using adapters.

The popular queuing backend Sidekiq already has a push_bulk method. Hence, the author of this pull request has made changes to the Sidekiq adapter so that perform_all_later uses the push_bulk method from Sidekiq.

Recently, GoodJob has also added support for bulk enqueuing in this pull request.

Practically, ActiveJob.perform_all_later is only useful if we want to push thousands of jobs at once. For a smaller number of jobs, the performance benefits will not be significant.

Please check out this pull request for more details.

Situations like this can be handled using migrations. Using migrations, we can disable "foreign keys". However, this would mean writing migrations for all the tables and removing all "foreign keys". This could be cumbersome.

def change
  create_table :authors do |t|
    t.string :name
    t.timestamps
  end

  create_table :books do |t|
    t.belongs_to :author, foreign_key: false
    t.datetime :published_at
    t.timestamps
  end
end

Rails 7.1 adds an option to database.yml that enables skipping foreign key constraints usage even if the underlying database supports them and solves the above issue of writing migrations to disable foreign keys for all the tables.

development:
  <<: *default
  database: db/development.sqlite3
  foreign_keys: false

Please check out this pull request for more details.

insert_all : This method can insert multiple records with a single SQL INSERT statement.

upsert_all : This method updates the records if they exist or inserts them into the database with a single SQL INSERT statement.

alias_attribute : Allows you to make aliases for attributes, which include a getter, a setter, and a predicate.

Rails 7.1 allows the use of aliased attributes with insert_all and upsert_all. Previously whenever we added an alias for an attribute, we couldn't use it for insert_all and upsert_all.

class User < ApplicationRecord
  # database column is `name`. `full_name` is the alias.
  alias_attribute :full_name, :name
end

Before Rails 7.1

# rails console
> User.insert_all [{ full_name: "John Doe" }]
=> # unknown attribute 'full_name' for User. (ActiveModel::UnknownAttributeError)

After Rails 7.1

# rails console
> User.insert_all [{ full_name: "Jane Doe" }]
=> # User Insert

> User.last
=> #<User id: 6, name: "Jane Doe", created_at: Mon, 21 Nov 2022 18:07:11.349000000 UTC +00:00, updated_at: Mon, 21 Nov 2022 18:07:11.349000000 UTC +00:00>

Now we can use the alias attribute with insert_all and upsert_all.

Please check out this pull request for more details.

Here are the issues we faced during the upgrade.

Migrating to Active Record Encryption

This was the biggest challenge we faced during the upgrade. For encrypting columns, we had used the attr_encrypted gem. However Rails 7 came with Active Record Encryption. So we needed to decrypt the records in the production database and encrypt them using Active Record Encryption. We found that "attr_encrypted" gem was incompatible with Rails 7. So the only option was to remove the "attr_encrypted" gem and decrypt the records using a script. We used the following method to decrypt the records.

def decrypted_attribute(attribute_name, record)
  value = record.send(attribute_name)
  return if value.blank?

  value = Base64.decode64(value)

  cipher = OpenSSL::Cipher.new("aes-256-gcm")
  cipher.decrypt

  cipher.key = Rails.application.secrets.attr_encrypted[:encryption_key]
  cipher.iv = Base64.decode64(record.send(:"#{attribute_name}_iv"))

  cipher.auth_tag = value[-16..]
  cipher.auth_data = ""

  cipher.update(value[0..-17]) + cipher.final
end

Broken images in Active Storage

After the upgrade, we started getting broken images in some places. This happened for Active Storage links embedded in Rich Text. After some debugging, we found that we were getting incorrect Active Storage links because of a change in the key generation algorithm. The following configuration was loading images using the old algorithm.

config.active_support.key_generator_hash_digest_class = OpenSSL::Digest::SHA1

Since the new algorithm provides more security, we decided to migrate the links instead of using the old algorithm. We used the following code to migrate the old link to the new valid link.

# Usage:
# text_with_new_links = ActiveStorageKeyConverter.new(text_with_old_links).process
# If no links are there to replace, the original text will be returned as it is.

class ActiveStorageKeyConverter
  def initialize(text)
    @text = text
  end

  def process
    replace(@text)
  end

  private
    def convert_key(id)
      verifier_name = "ActiveStorage"
      key_generator =  ActiveSupport::KeyGenerator.new(Rails.application.secrets.
      secret_key_base, iterations: 1000, hash_digest_class: OpenSSL::Digest::SHA1)
      key_generator = ActiveSupport::CachingKeyGenerator.new(key_generator)
      secret = key_generator.generate_key(verifier_name.to_s)
      verifier = ActiveSupport::MessageVerifier.new(secret)

      ActiveStorage::Blob.find_by_id(verifier.verify(id, purpose: :blob_id))
      .try(:signed_id) rescue nil
    end

    def replace(text)
      keys = text.scan(URI.regexp).flatten.select{|x|
      x.to_s.include? ("rails/active_storage")}.map{|x| x.split("/")[-2]}
      keys.each do |key|
        new_key = convert_key(key)
        text = text.gsub(key, new_key) if new_key
      end
      text
    end
end

Following one time rake task was used to update the Active Storage links in the content column of Task model:

desc "Update active storage links embedded in rich text to support in rails 7"
task migrate_old_activestorage_links: :environment do

  table_colum_map = {
    "Task" => "content",
  }
  match_term = "%rails/active_storage%"

  table_colum_map.each do |model_name, column_name|
    model_name.to_s.constantize.where("#{column_name} ILIKE ?", match_term).find_each do|row|
      row.update_column(column_name, ActiveStorageKeyConverter.new(row[column_name]).process)
    end
  end
end

Test failures with the mailer jobs

After upgrading to Rails 7, tests related to mailers started to fail. This was because the mailer jobs were enqueued in the default queue instead of mailers. We fixed this by adding the following configuration.

config.action_mailer.deliver_later_queue_name = :mailers

Autoloading during initialization failed

After the upgrade, if we startthe Rails sever then we were getting the following error.

$ rails s
=> Booting Puma
=> Rails 7.0.3.1 application starting in development
=> Run `bin/rails server --help` for more startup options
Exiting
/Users/BB/Neeto/neeto_commons/lib/neeto_commons/initializers/session_store.rb:13:in
`session_store': uninitialized constant #<Class:NeetoCommons::Initializers>::ServerSideSession     (NameError)

    ActionDispatch::Session::ActiveRecordStore.session_class = ServerSideSession
                                                                ^^^^^^^^^^^^^^^^^
   from /Users/BB/Neeto/neeto-planner-web/config/initializers/common.rb:10:in `<main>'

That error was coming from our internal neeto-commons initializer called session_store.rb. The code looked like this.

#session_store.rb

module NeetoCommons
  module Initializers
    class << self
      def session_store
        Rails.application.config.session_store :active_record_store,
          key: Rails.application.secrets.session_cookie_name, expire_after: 10.years.to_i

        ActiveRecord::SessionStore::Session.table_name = "server_side_sessions"
        ActiveRecord::SessionStore::Session.primary_key = "session_id"
        ActiveRecord::SessionStore::Session.serializer = :json
        ActionDispatch::Session::ActiveRecordStore.session_class = ServerSideSession
      end
    end
  end
end

In order to fix the issue we had to put the last statement in a block like shown below.

Rails.application.config.after_initialize do
 ActionDispatch::Session::ActiveRecordStore.session_class = ServerSideSession
end

Missing template error with pdf render

After the Rails 7 upgrade the following test started failing.

def test_get_task_pdf_download_success
  get api_v1_project_section_tasks_download_path(@project.id, @section, @task, format: :pdf)

  assert_response :ok

  assert response.body.starts_with? "%PDF-1.4"
  assert response.body.ends_with? "%EOF\n"
end

The actual error is Missing template api/v1/projects/tasks/show.html.erb.

In order to fix it we renamed the file name from /tasks/show.html.erb to /tasks/show.pdf.erb. Similarly we changed the layout from /layouts/pdf.html.erb to /layouts/pdf.pdf.erb.

Initially the controller code looked like this.

format.pdf do
  render \
    template: "api/v1/projects/tasks/show.html.erb"
    pdf: pdf_file_name,
    layout: "pdf.html.erb"
end

After the change the code looked like this.

format.pdf do
  render \
    template: "api/v1/projects/tasks/show",
    pdf: pdf_file_name,
    layout: "pdf"
end

Open Redirect protection

After the Rails 7 upgrade the following test started failing.

def test_that_users_are_redirected_to_error_url_when_invalid_subdomain_is_entered
  invalid_subdomain = "invalid-subdomain"

  auth_subdomain_url = URI(app_secrets.auth_app[:url].gsub(
   app_secrets.app_subdomain, invalid_subdomain))

  auth_app_url = app_secrets.auth_app[:url]

  host! test_domain(invalid_subdomain)
  get "/"

  assert_redirected_to auth_app_url
end

In the test we are expecting the application to redirect to auth_app_url but we are getting UnsafeRedirectError error for open redirections. In Rails 7 the new Rails defaults protects applications against the Open Redirect Vulnerability.

To allow any external redirects we can pass allow_other_host: true.

redirect_to <External URL>, allow_other_host: true

Since we use open redirection in many places we disabled this protection globally.

config.action_controller.raise_on_open_redirects = false

Argument Error for Mailgun signing key

After the upgrade, we started getting the following error in production.

>> ArgumentError: Missing required Mailgun Signing key. Set action_mailbox.mailgun_signing_key
in your application's encrypted credentials or provide the MAILGUN_INGRESS_SIGNING_KEY
environment variable.

Before Rails 7, we used the MAILGUN_INGRESS_API_KEY environment variable to set up the Mailgun signing key . In Rails 7, that is changed to MAILGUN_INGRESS_SIGNING_KEY. So we renamed the environment variable to fix the problem.

All three videos were prepared for my team members to show how to go about debugging. The videos are being presented "as it was recorded".

First Case Study

When a job fails in Sidekiq, Sidekiq puts that job in RetrySet and retries that job until the job succeeds or the job reaches the maximum number of retries. By default the maximum number of retries is 25. If a job fails 25 times, then that job is moved to the DeadSet. By default, Sidekiq will store up to 10,000 jobs in the deadset.

We had a situation where Redis was running out of memory. Here is how the debugging was done.

Watch on YouTube

How to inspect the deadset

ds = Sidekiq::DeadSet.new
ds.each do |job|
  puts "Job #{job['jid']}: #{job['class']} failed at #{job['failed_at']}"
end

Running the following to view the latest entry to the dataset:

ds.first
ds.count

To see the memory usage following commands were executed in the Redis console.

> memory usage dead
30042467

> type dead
zset

As discussed in the video large amount of payload was being sent. This is not the right way to send data to the worker. Ideally, some sort of id should be sent to the worker and the worker should be able to get the necessary data from the database based on the received id.

References

1. How to increase the number of jobs in the Sidekiq deadset or disable deadset
2. Maximum number of job retries in Sidekiq
3. Maximum number of jobs in Sidekiq Deadset

Second case study

In this case, the Redis instance of neetochat was running out of memory. The Redis instance had 50MB capacity, but we were getting the following error.

ERROR: heartbeat: OOM command not allowed when used memory > 'maxmemory'.

We were pushing too many geo info records to Redis and that caused the memory to fill up. Here is the video capturing the debugging session.

Watch on YouTube

The following are the commands executed while debugging.

> ping
PONG

> info

> info memory

> info keyspace

> keys *failed*

> keys *process*

> keys *geocoder*

> get getocoder:http://ipinfo.io/41.174.30.55/geo?

Third Case Study

In this case, the authentication service of Neeto was failing because of memory exhaustion.

Here the number of keys was limited, but the payload data was huge and all that payload data was hogging the memory. Here is the video capturing the debugging session.

Watch on YouTube

The following are the commands executed while debugging.

> ping

> info keyspace
db0:keys=106, expires=86,avg_ttl=1233332728573

> key * (to see all the keys)

The last command listed all 106 keys. Next, we needed to find how much memory each of these keys are using. For that, the following commands were executed.

> memory usage organizations/subdomains/bigbinary/neeto_app_links
736 bytes

> memory usage failed
10316224 (10MB)

> memory usage dead
29871174 (29MB)

Book.where(purchases: 20..)
# Returns a collection of records with purchases from 20 and upwards.
Book.where(purchases: ..20)
# Returns a collection of records with purchases from 20 and below.

But the same idea of using infinite ranges in Active Model validations was limited in scope. Rails 7.1 extends this scope of usage by adding support for infinite ranges in Active Model validations. For example, validating the length of first_name without an upper bound for a User will be as simple as writing:

class User
    # ...
    validates_length_of :first_name, in: 20..
end

The length of the :first_name does not have an upper bound. As long as the length is greater than or equal to 20, it will remain valid.

The above example also holds true when using the :within option as well:

class User
    # ...
    validates_length_of :first_name, within: 20..
end

In a similar example, let's look at how we would use Active Model validations along with the :inclusion option:

class User
    # ...
    validates :age, inclusion: { in: proc { (25..) } }
end

The above example would validate the :age field such that, it's value needs to be 25 or above for the record to be valid.

Please check out the following pull requests for more details:

1. Infinite ranges for LengthValidator
2. Infinite ranges for Clusivity validator

Failed to replace attachments_attachments because one or more of the new records could not be saved.

This is not helpful, and it doesn't indicate where to make the required changes for Active Storage to be able to save the attachment(s). It also allows the application to start as if a valid service has been set for Active Storage to use. Starting Rails 7.1, if config.active_storage.service has not been explicitly set, then even attempting to start the application would throw a RuntimeError with the following error message:

Missing Active Storage service name. Specify Active Storage service name for config.active_storage.service in config/environments/production.rb

Please check out this pull request for more details.

These are the following callbacks, which can be registered:

1. before_command: This callback is invoked before any command can be processed by the channel.
2. around_command: This callback is generally invoked around the command. Any piece of code before yield is run before the actual command and any piece of code after yield is run after the command has been processed by the channel.
3. after_command: This callback is invoked after the command has been processed by the channel.

Let's take a look at some example code to understand this behaviour better:

class Connection < ActionCable::Connection::Base
  identified_by :current_user
  before_command :set_current_user
  around_command :register_telemetry_data
  after_command :update_current_user

  private

    def set_current_user
      if request.params["user_id"].present?
        self.current_user = User.find_by(request.params["user_id"])
      end
      reject_unauthorized_connection if self.current_user.nil?
    end

    def register_telemetry_data
      self.current_user.register_telemetry({ start: true })
      yield
      self.current_user.register_telemetry({ end: true })
    end

    def update_current_user
      self.current_user.touch(:updated_at)
    end
end

Here, we can expect set_current_user to be invoked before every command is processed. Similar to this, we can also expect update_current_user to be invoked after every command is processed. Whereas for the register_telemetry_data, self.current_user.register_telemetry({ start: true }) is run before the command is processed and then after the command is processed, self.current_user.register_telemetry({ end: true }) is run.

Please check out this pull request for more details.

Previously, the helper methods received only the asset path/url. To get the asset path of an Active Storage Attachment, we had to explicitly call polymorphic_path on the attachment, which returned the desired asset path.

Before

audio_tag(polymorphic_path(user.audio_file))
# => <audio src="/..."></audio>
video_tag(polymorphic_path(user.video_file))
# => <video src="/..."></video>

After

audio_tag(user.audio_file)
# => <audio src="/..."></audio>
video_tag(user.video_file)
# => <video src="/..."></video>

Please check out this pull request for more details.

Before

# rails console
>> @user = User.create!(name: "Josh")
>> @user.avatar.attach(params[:avatar])
=> true

After

# rails console
>> @user = User.create!(name: "Josh")
>> @user.avatar.attach(params[:avatar])
=> #<ActiveStorage::Attached::One:0x00007f075e592380 @name="avatar" @record=#<User:0x00007f075e5924e8 id: "1", name: "Josh">

Please check out this pull request for more details.

When the database field is a float column, the data will get serialized to the correct type. In the case of a JSON column, serialization doesn't take place.

As a resolution, Rails 7 has added only_numeric option to numericality validator.

We will see it in action.

To demonstrate, we need to generate a table that has a json/jsonb column.

# migration
create_table :users do |t|
  t.jsonb :personal
end

Before Validation

# Model
class User < ApplicationRecord
  store_accessor :personal, %i[age tooltips]
end

# rails console
>> User.create!(age: '29')
=> #<User id: 1, preferences: {"age"=>"29"}, created_at: Sun, 16 Jan 2022 14:09:43.045301000 UTC +00:00, updated_at: Sun, 16 Jan 2022 14:09:43.045301000 UTC +00:00>

After Validation

# Model
class User < ApplicationRecord
  store_accessor :personal, %i[age tooltips]
  validates_numericality_of :age, only_numeric: true, allow_nil: true
end

# rails console
>> User.create!(age: '29')
=> 'raise_validation_error': Validation failed: Age is not a number (ActiveRecord::RecordInvalid)

>> User.create!(age: 29)
=> #<User id: 2, preferences: {"age"=>29}, created_at: Sun, 16 Jan 2022 14:15:44.599934000 UTC +00:00, updated_at: Sun, 16 Jan 2022 14:15:44.599934000 UTC +00:00>

Please check out this pull request for more details.

Before

We need to first check whether the given file path exists or not to perform any other operations on it.

=> file_path = "config/schedule.yml"
=> Pathname.new(file_path).read if Pathname.new(file_path).exist?

Rails 7 onwards

The Pathname#existence method acts like Object#presence for file existence.

=> file_path = "config/schedule.yml"
=> Pathname.new(file_path).existence&.read

Please check out this pull request for more details.

class Entry < ApplicationRecord
  # Schema
  #  entryable_type, entryable_id, ...
  delegated_type :entryable, types: %w[ Message Comment ]
end

class Message
  # Schema
  #  subject, ...
end

class Comment
  # Schema
    #  content, ...
end

The accepts_nested_attributes_for option is very helpful while handling nested forms. We can easily create and update associated records by passing details along with the main object parameters when the accepts_nested_attributes_for option is enabled.

Before

The accepts_nested_attributes_for option is not available for delegated_type, hence we can't use nested forms for associated objects configured via delegated_type.

Rails 7 onwards

Rails 7 adds accepts_nested_attributes_for support to delegated_type, which allows to create and update records easily without needing to write specific methods or logic.

class Entry < ApplicationRecord
  delegated_type :entryable, types: %w[ Message Comment ]
  accepts_nested_attributes_for :entryable
end

params = {
  entry: {
    entryable_type: 'Message',
    entryable_attributes: { subject: 'Delegated Type' }
  }
}

message_entry = Entry.create(params[:entry])

params = {
  entry: {
    entryable_type: 'Comment',
    entryable_attributes: { content: 'Looks Cool!' }
  }
}

comment_entry = Entry.create(params[:entry])

If we want to deal with more logic or validations while creating the entryable objects, we'll have to create a method specifically and do the logic there.

class Entry < ApplicationRecord
  def self.create_with_comment(content)
    # Validation logic goes here
    create! entryable: Comment.new(content: content)
  end
end

Please check out this pull request for more details.

Rails 5.1 introduced assert_no_changes and assert_changes, making it easier to observe changes. Using ActiveSupport::TestCase#assert_no_changes we can easily assert that the result of evaluating an expression has not changed before and after calling the passed block.

To assert the expected change in the value of an object, we can use assert_changes.

assert_changes -> { user.address } do
  user.update address: 'Miami'
end

assert_changes also supports from and to options.

assert_changes -> { user.address }, from: 'San Francisco', to: 'Miami' do
  user.update address: 'Miami'
end

Similarly, assert_no_changes allows us to assert a value that is expected to not change.

assert_no_changes -> { user.address } do
  user.update address: 'Miami'
end

We can also specify an error message with assert_no_changes.

assert_no_changes -> { user.address }, 'Expect the address to not change' do
  user.update address: 'Miami'
end

Before

assert_no_changes did not support the from option, similar to assert_changes.

assert_no_changes -> { user.address } do
  user.update address: 'Miami'
end

However, Rails 7 has added from: option to ActiveSupport::TestCase#assert_no_changes, allowing us to assert on the initial value that is expected to not change.

Rails 7 onwards

Provides the optional from argument to specify the expected initial value.

assert_no_changes -> { user.address }, from: 'San Francisco' do
  user.update address: 'Miami'
end

Check out this pull request for more details.

Using byebug we can easily control the execution of a program and the debug inspector for call stack navigation. This allows us to handle and track the execution flow.

Here is byebug documentation and here is the pull request where it was added.

Rails 7 is replacing byebug with ruby/debug. debug is Ruby’s new debugger, which will be included in Ruby 3.1. To align Rails with Ruby debug has been added to Rails 7.

Let's see an example of debugging with both Byebug and Debug.

Before

Let's assume we have a NameController. Inside any Rails application, you can call the debugger by calling the byebug method.

# app/controllers/name_controller.rb
class NameController < ApplicationController
  def index
    name = "John Doe"
    byebug # Call to debugger
    city = "San Francisco"
  end
end

Then the invoked debugger results in the following.

    [1, 7] in app/controllers/test_controller.rb
    1: class NameController < ApplicationController
    2:   def index
    3:     name = "John Doe"
    4:     byebug # Call to debugger
=>  5:    city = "San Francisco"
    6:   end
    7: end

  (byebug) name # variable call
  "John Doe"

Rails 7 onwards

We can use the binding.break method for calling Ruby Debug.

# app/controllers/name_controller.rb
class NameController < ApplicationController
  def index
    name = "John Doe"
    binding.break # Call to debugger
    city = "San Francisco"
  end
end

The invoked debugger results in the following.

  [1, 7] in app/controllers/test_controller.rb
  1| class NameController < ApplicationController
  2|   def index
  3|     name = "John Doe"
  4|     binding.break # Call to debugger
> 5|    city = "San Francisco"
  6|   end
  7| end
>#0 NameController#index at ~/demo_app/app/controllers/name_controller.rb:5
  #1  ActionController::BasicImplicitRender#send_action(method="index", args=[])
(rdbg) name # variable call
"John Doe"

Check out this pull request for more details and for commands or features of Ruby Debug, please visit Ruby-Debug.

Using select helper we can easily create select boxes in HTML with one <option> element for each option to choose from.

For example, let's say we have a list of cities for the user to choose from.

# app/views/address/new.html.erb
<%= form.select :city, ["Pune", "Mumbai", "Delhi"] %>

Rails will generate the following markup.

<select name="city" id="city">
  <option value="Pune">Pune</option>
  <option value="Mumbai">Mumbai</option>
  <option value="Delhi">Delhi</option>
</select>

Previously for generating a select box for weekday select, we needed to write a custom helper. Rails did not have anything out of the box for the weekday selection.

However, Rails 7 has added weekday_options_for_select and weekday_select using which we can easily generate a dropdown field of selecting a weekday.

Before

In Rails 6.1, we can create a dropdown field for weekdays as shown here.

# app/views/users/new.html.erb
<%= form_with(model: user) do |form| %>
  <div class="field">
    <%= form.label :weekly_off %>
    <%= form.select :weekly_off, I18n.t('date.day_names') %>
  </div>
<% end %>

Or we can do something like this.

# app/views/users/new.html.erb
<%= form_with(model: user) do |form| %>
  <div class="field">
    <%= form.label :weekly_off %>
    <%= form.select :weekly_off, I18n.t('date.day_names').map.with_index.to_h %>
  </div>
<% end %>

Then the generated markup looks like this.

<select name="user[weekly_off]" id="user_weekly_off">
  <option value="Sunday">Sunday</option>
  <option value="Monday">Monday</option>
  <option value="Tuesday">Tuesday</option>
  <option value="Wednesday">Wednesday</option>
  <option value="Thursday">Thursday</option>
  <option value="Friday">Friday</option>
  <option value="Saturday">Saturday</option>
</select>

Here is how it would look if we go with the second option.

<select name="user[weekly_off]" id="user_weekly_off">
  <option value="0">Sunday</option>
  <option value="1">Monday</option>
  <option value="2">Tuesday</option>
  <option value="3">Wednesday</option>
  <option value="4">Thursday</option>
  <option value="5">Friday</option>
  <option value="6">Saturday</option>
</select>

Rails 7 onwards

We can use the weekday_options_for_select or weekday_select helper for generating a dropdown field for selecting a weekday.

# app/views/users/new.html.erb
<%= form_with(model: user) do |form| %>
  <div class="field">
    <%= form.label :weekly_off %>
    <%= form.select :weekly_off, weekday_options_for_select("Monday", day_format: :abbr_day_names) %>
  </div>
<% end %>

Or we can do something like this.

# app/views/users/new.html.erb
<%= form_with(model: user) do |form| %>
  <div class="field">
    <%= form.label :weekly_off %>
    <%= form.weekday_select :weekly_off, { selected: "Monday", day_format: :abbr_day_names } %>
  </div>
<% end %>

Then The generated markup looks like this.

<select name="user[weekly_off]" id="user_weekly_off">
  <option value="Sunday">Sunday</option>
  <option selected="selected" value="Monday">Monday</option>
  <option value="Tuesday">Tuesday</option>
  <option value="Wednesday">Wednesday</option>
  <option value="Thursday">Thursday</option>
  <option value="Friday">Friday</option>
  <option value="Saturday">Saturday</option>
</select>

weekday_options_for_select accepts a few options and all of them have a default value.

selected defaults to nil and if we provide this argument, the value passed will be used as the selected option.

 <!-- weekday_options_for_select("Friday") -->
<option value=\"Sunday\">Sunday</option>\n
<option value=\"Monday\">Monday</option>\n
<option value=\"Tuesday\">Tuesday</option>\n
<option value=\"Wednesday\">Wednesday</option>\n
<option value=\"Thursday\">Thursday</option>\n
<option selected=\"selected\" value=\"Friday\">Friday</option>\n
<option value=\"Saturday\">Saturday</option>"

index_as_value defaults to false and if true it will set the value of each option to the index of that day.

<!-- weekday_options_for_select(index_as_value: true) -->
<option value=\"0\">Sunday</option>\n
<option value=\"1\">Monday</option>\n
<option value=\"2\">Tuesday</option>\n
<option value=\"3\">Wednesday</option>\n
<option value=\"4\">Thursday</option>\n
<option value=\"5\">Friday</option>\n
<option value=\"6\">Saturday</option>"

day_format defaults to :day_names and passing this a different I18n key will use different formats for the option display names and their values.

<!-- weekday_options_for_select(day_format: :abbr_day_names) -->
<option value=\"Sun\">Sun</option>\n
<option value=\"Mon\">Mon</option>\n
<option value=\"Tue\">Tue</option>\n
<option value=\"We\">Wedn</option>\n
<option value=\"Thu\">Thu</option>\n
<option value=\"Fri\">Fri</option>\n
<option value=\"Sat\">Sat</option>"

Note that :abbr_day_names options are built into Rails, but we can define your array.

Check out this pull request for more details.