Identifying the problem

The crash happened inside Slots::SyncAllCalendarsService, a service that syncs multiple calendars concurrently using the Async gem. What made it particularly puzzling was that switching from Async::Barrier to Thread.new made the error disappear. This led us down a rabbit hole thinking it was an Async-specific issue.

The stack trace pointed to a strange location:

activerecord (7.2.1.1) lib/arel/visitors/to_sql.rb:898:in `each'
  from lib/arel/visitors/to_sql.rb:898:in `each_with_index'
  from lib/arel/visitors/to_sql.rb:898:in `inject_join'
  from lib/arel/visitors/to_sql.rb:627:in `visit_Arel_Nodes_Or'
  ... 1132 levels...

Over a thousand levels deep into visit_Arel_Nodes_Or. That's unusual. Why would visiting OR nodes in a SQL query cause such deep recursion?

The stack trace mentioned Slots::SyncAllCalendarsService, but that service itself had nothing obviously wrong. It was just iterating through calendars and syncing them. We traced deeper into the code and found the actual culprit in Neeto's Integrations::Icloud::SyncEventsService:

def abandoned_events
  pairs = event_params.map { |e| [e[:url], e[:recurrence_id]] }

  @_abandoned_events = calendar.events
    .where("start_time BETWEEN ? AND ?", start_time, end_time)
    .where.not([:url, :recurrence_id] => pairs)
end

This code finds "abandoned" events by excluding events that match specific URL and recurrence ID combinations. In production, pairs contained 233 entries. Nothing immediately suspicious, right?

To understand what was happening, we needed to see what Rails was actually doing with this query. The where.not([:url, :recurrence_id] => pairs) syntax is Rails' composite key feature. With 233 pairs, Rails generates SQL like this:

WHERE NOT (
  (url = 'url1' AND recurrence_id = 'rec1') OR
  (url = 'url2' AND recurrence_id = 'rec2') OR
  (url = 'url3' AND recurrence_id = 'rec3') OR
  -- ... 230 more OR clauses
)

233 OR clauses. That's a lot, but databases can handle it. The problem wasn't the SQL itself - it was how Rails built the internal representation of that query.

Going into Rails Internals

To really understand what changed, we used bundle open activerecord to look at the Rails source code. We compared the same code in Rails 7.1.5.2 and 7.2.1.1.

What We Found in Rails 7.1.5.2

In Rails 7.1.5.2, the code that visits OR nodes looked like this:

def visit_Arel_Nodes_Or(o, collector)
  stack = [o.right, o.left]

  while o = stack.pop
    if o.is_a?(Arel::Nodes::Or)
      stack.push o.right, o.left
    else
      visit o, collector
      collector << " OR " unless stack.empty?
    end
  end

  collector
end

This is an iterative approach using a manual stack. No matter how deeply nested the OR conditions are, this code only uses a single stack frame for the visit_Arel_Nodes_Or method. It's stack-safe.

What Changed in Rails 7.2.1.1

In Rails 7.2.1.1, the same method became much simpler:

def visit_Arel_Nodes_Or(o, collector)
  inject_join o.children, collector, " OR "
end

Where inject_join does:

def inject_join(list, collector, join_str)
  list.each_with_index do |x, i|
    collector << join_str unless i == 0
    collector = visit(x, collector)  # Recursive call
  end
  collector
end

This is a recursive approach. If an OR node contains another OR node, it calls visit, which calls visit_Arel_Nodes_Or, which calls inject_join again. The recursion depth grows with the nesting depth of the OR tree.

The Tree Structure Problem

But why is the OR tree so deeply nested? We found the answer in Active Record's PredicateBuilder#grouping_queries:

def grouping_queries(queries)
  if queries.one?
    queries.first
  else
    queries.map! { |query| query.reduce(&:and) }
    queries = queries.reduce { |result, query| Arel::Nodes::Or.new([result, query]) }
    Arel::Nodes::Grouping.new(queries)
  end
end

That reduce call is the key. It builds a left-deep nested tree:

Level 1:  Query1 OR Query2 = OR_Node_1
Level 2:  OR_Node_1 OR Query3 = OR_Node_2
Level 3:  OR_Node_2 OR Query4 = OR_Node_3
...
Level 232: OR_Node_231 OR Query233 = OR_Node_232

With 233 pairs, this creates 232 levels of nesting. Each level adds approximately 5 stack frames when Rails traverses it recursively. That's 1,160 stack frames just for the query building.

The Async Connection

Why did Thread.new work but Async::Barrier didn't?

We wrote a test script to measure the actual stack depth limits in different contexts:

def test_recursion_depth(context_name)
  depth = 0

  recurse = lambda do |n|
    depth = n
    recurse.call(n + 1)
  end

  begin
    recurse.call(0)
  rescue SystemStackError
    return depth
  end
end

thread_depth = nil
Thread.new do
  thread_depth = test_recursion_depth("thread")
end.join
puts "Thread recursion limit: ~#{thread_depth} calls"

async_depth = nil
Async do
  async_depth = test_recursion_depth("async")
end
puts "Async fiber recursion limit: ~#{async_depth} calls"

When we ran this, the results were revealing:

Thread recursion limit:      ~11910 calls
Async fiber recursion limit: ~1482 calls

Threads have roughly 8x more stack space than Async fibers. This is because threads use larger stacks (1MB-8MB) while fibers are designed to be lightweight with smaller stacks (512KB-1MB). This difference in stack allocation is why threads can handle deeper recursion before hitting the overflow limit.

With 233 pairs requiring ~1,160 stack frames, we were right at the edge of the Async fiber limit. But we were still well within the thread limit, which is why switching to Thread.new seemed to fix it.

When we tested with 500 pairs (which some of our larger calendars had), even Thread.new failed with a stack overflow. So it wasn't really a fix, it just pushed the problem further down the road.

The GitHub Trail

After we understood the problem, we searched the Rails repository and found the exact history:

1. April 4, 2024 - PR #51492: This PR changed OR nodes from binary to n-ary to handle multiple children in a single node.

2. September 24, 2024 - Issue #53031: Someone reported the exact issue we were seeing. Queries with 800+ OR conditions that worked in Rails 7.1 now crashed in Rails 7.2.

3. September 25, 2024 - PR #53032: fatkodima fixed it with a one-line change in PredicateBuilder#grouping_queries:

Before:

queries = queries.reduce { |result, query| Arel::Nodes::Or.new([result, query]) }

After:

queries = Arel::Nodes::Or.new(queries)

The difference is profound. The old code using reduce created a deeply nested structure like Russian nesting dolls:

Or([Or([Or([Query1, Query2]), Query3]), Query4])  # 232 levels with 233 pairs

The new code creates a flat structure:

Or([Query1, Query2, Query3, Query4, ...Query233])  # Just 1 level

With a flat structure, the recursive visitor only traverses one level instead of 232, eliminating the stack overflow. The fix is available in Rails 7.2.2+.

Our Solution

We're currently on Rails 7.2.1.1 and upgrading immediately would require extensive testing. So we implemented a workaround in our code:

def abandoned_events
  return @_abandoned_events if defined?(@_abandoned_events)

  pairs = event_params.map { |e| [e[:url], e[:recurrence_id]] }
  return if pairs.empty?

  events_in_time_range = calendar.events
    .where("start_time BETWEEN ? AND ?", start_time, end_time)

  # Get candidate events using simple IN clauses
  urls = pairs.map(&:first).uniq
  recurrence_ids = pairs.map(&:last).uniq

  candidate_events = events_in_time_range
    .where(url: urls)
    .where(recurrence_id: recurrence_ids)

  # Filter to exact pairs in memory
  pairs_set = pairs.to_set
  ids_to_exclude = candidate_events.select { |event|
    pairs_set.include?([event.url, event.recurrence_id])
  }.map(&:id)

  @_abandoned_events = if ids_to_exclude.empty?
    events_in_time_range
  else
    events_in_time_range.where.not(id: ids_to_exclude)
  end
end

This approach:

• Uses simple IN clauses instead of complex OR conditions
• Filters the exact pairs in memory (fast with a Set)
• Excludes by ID, which is a flat list
• Never builds deeply nested Arel nodes

The trade-off is we run 2 queries instead of 1, but the queries are simpler and more efficient. The in-memory filtering is negligible since we're already constrained by time range.

Measuring and Verifying

To prove our hypothesis, we wrote verification scripts that patched Rails internals to measure what was actually happening.

Counting Arel Visitor Calls

We created a script that patched visit_Arel_Nodes_Or to count how many times it was called and how deep the recursion went:

module Arel
  module Visitors
    class ToSql
      alias_method :original_visit_Arel_Nodes_Or, :visit_Arel_Nodes_Or

      def visit_Arel_Nodes_Or(o, collector)
        @or_call_count ||= 0
        @or_call_count += 1
        @max_depth ||= 0
        @current_depth ||= 0
        @current_depth += 1
        @max_depth = [@max_depth, @current_depth].max

        result = original_visit_Arel_Nodes_Or(o, collector)

        @current_depth -= 1
        result
      end
    end
  end
end

Results with 50 test pairs:

• Rails 7.1.5.2: visit_Arel_Nodes_Or was called 1 time and max recursion depth was 0.
• Rails 7.2.1.1: visit_Arel_Nodes_Or was called 49 times and max recursion depth was 100.

With 233 pairs in production:

• Rails 7.1.5.2: Still called 1 time, depth 0 (iterative approach)
• Rails 7.2.1.1: Called 232 times, depth ~466 (exceeds Async fiber limit)

Testing the Actual Breaking Points

We ran the problematic query with different pair counts to find exactly where it breaks:

# Test with Async
Async do
  pairs = calendar.events.limit(count).pluck(:url, :recurrence_id)
  calendar.events.where.not([:url, :recurrence_id] => pairs).count
end.wait

# Test with Thread
Thread.new do
  pairs = calendar.events.limit(count).pluck(:url, :recurrence_id)
  calendar.events.where.not([:url, :recurrence_id] => pairs).count
end.join

Results:

• Async Fiber: Breaks at approximately 233 pairs
• Thread.new: Breaks at approximately 500 pairs

This confirmed that Thread.new wasn't a real solution - it just had more headroom before hitting the same problem.

For now, we're sticking with our workaround. It works reliably, performs well, and doesn't require upgrading Rails immediately. When we do upgrade to Rails 7.2.2+, we can revert to the original clean syntax, knowing that the Rails team has fixed the underlying issue.

If you're on Rails 7.2.0 through 7.2.1.x and use composite key queries with large datasets, watch out for this issue. The fix is in Rails 7.2.2+, or you can work around it like we did.

References

• PR #51492 - Bug introduced
• Issue #53031 - Bug reported
• PR #53032 - Bug fixed

Marking an association as deprecated

Simply pass the deprecated: true option when declaring an association.

class User < ApplicationRecord
  has_many :meetings, deprecated: true
end

Now, every time the meeting association is invoked, we’ll get a deprecation warning in our logs.

> user.meetings
The association User#meetings is deprecated, the method meetings was invoked ((calendar):4:in '<main>')

> User.includes(:meetings).where(id: 1)
The association User#meetings is deprecated, referenced in query to preload records ()

> User.joins(:meetings).where(id: 1)
The association User#meetings is deprecated, referenced in query to join its table ()

Working with different association types

We can deprecate any association type.

class Order < ApplicationRecord
  # has_many
  has_many :line_items, deprecated: true

  # belongs_to
  belongs_to :customer, deprecated: true

  # has_one
  has_one :profile, deprecated: true

  # has_many through
  has_many :archived_comments, through: :comments, deprecated: true
end

Reporting modes and backtrace support

This feature supports three deprecation modes:

• :warn (default) Logs a warning to our Active Record logger.
• :raise Raises an exception when the deprecated association is used.
• :notify Emits an Active Support notification event with the key deprecated_association.active_record. This can be used to send notifications to external services like Honeybadger etc. We can check the details about its payload here.

Backtraces are disabled by default. If :backtrace is true, :warn mode will include a clean backtrace in the message, and :notify mode will have a backtrace key in the payload. Exceptions raised via :raise mode will always have a clean stack trace.

We can change the global default mode in an initializer.

# config/initializers/deprecated_associations.rb
ActiveRecord.deprecated_associations_options = {
  mode: :warn,      # :warn | :raise | :notify
  backtrace: true   # whether to include a cleaned backtrace
}

It can also be set at an environment level.

# config/environments/development.rb
Rails.application.configure do
  config.active_record.deprecated_associations_options = { mode: :raise, backtrace: true }
end

# config/environments/production.rb
Rails.application.configure do
  config.active_record.deprecated_associations_options = { mode: :warn, backtrace: true }
end

Why deprecate rather than remove?

In large applications, it’s often hard to guarantee complete test coverage. Some association usages may only surface in production.

Deprecating an association first, lets us:

• Identify every code path (tests, console, background jobs) that relies on it.
• Gradually refactor those references before removal.
• Ensure confidence that deleting the association won’t break anything unexpectedly.

This new feature in Rails provides a clean and intuitive way to phase out our associations with deprecation warnings, making it easier to maintain and refactor large codebases.

This feature was merged recently, and will be released in the next Rails minor/patch version.

Dual-booting is a process that allows you to run your application with different sets of dependencies, making it easy to switch between them. This approach enables you to quickly test your code with both the current and the newer version of what you are upgrading, ensuring that everything works smoothly before fully committing to the upgrade.

How to dual-boot Ruby?

The dual-boot technique involves maintaining two separate Gemfile.lock files within your Rails project: one for the current version of Ruby and another for the next version you're upgrading to.

To get started, first create a symbolic link for Gemfile.next with the following command:

ln -s Gemfile Gemfile.next

This command creates Gemfile.next file, which you'll use for the new Ruby version. This file isn't a separate copy but rather a pointer to your original Gemfile in the Rails project directory.

Next, add the following snippet at the top of your Gemfile:

def next?
  File.basename(__FILE__) == "Gemfile.next"
end

This code snippet helps determine which Gemfile is in use, whether it's the standard one or the Gemfile.next for the new Ruby version.

Now, let's utilize the next? method to dynamically set the Ruby version in the Gemfile using a conditional operator:

ruby_version = next? ? "3.3.0" : "3.2.2"
ruby ruby_version

This code snippet helps determine the Ruby version based on the next? method. If next? returns true, meaning you're operating within Gemfile.next, it sets the Ruby version to "3.3.0". Otherwise, if the standard Gemfile is being processed, it defaults to "3.2.2".

How to install dependencies?

With the dual-boot setup in place, you can effectively manage dependencies for both your current and next versions of the application.

Installing current dependencies: To install dependencies for your current version, simply run:

bundle install

This command uses the standard Gemfile to resolve and install dependencies for your current application.

Installing next dependencies: To install dependencies for the next version of your application, use the following command:

BUNDLE_GEMFILE=Gemfile.next bundle install

This command specifies Gemfile.next as the Gemfile, allowing you to install dependencies specifically for the next version.

Managing commands for the next version: To perform various operations for the next version, you can use the syntax BUNDLE_GEMFILE=Gemfile.next <command>. For example, to start the Rails server with the next version's dependencies, you would use:

BUNDLE_GEMFILE=Gemfile.next rails s

This approach ensures that Gemfile.next is explicitly used for the specified command, allowing you to work with the next version of your application while keeping dependencies and configurations correctly managed.

Considerations for dual-boot setup

When implementing a dual-boot setup, it's crucial to consider where both Ruby and gem versions are defined. While gem versions are usually specified only in the Gemfile, the Ruby version can be defined in several locations. The most common are:

• .ruby-version File: Used by version managers like rbenv to specify the Ruby version.
• Gemfile: This is where Bundler, RVM, Heroku, and similar tools reference the Ruby version to manage the Ruby environment.
• Gemfile.lock: Although the Ruby version in this file is mostly informative, it's worth noting.
• Dockerfile: Defines the Ruby version for Docker containers if you're using Docker.
• CI setup steps and configurations.
• Other less common locations within applications.

We need to make sure that dual-boot setup is compatible with all of them to ensure a smooth transition and consistency across different environments.

Pre-compiled Ruby binaries

Pre-compiled Ruby binaries are distribution-ready versions of Ruby that include optimized features for specific systems. These Ruby binaries save time by eliminating the need to compile Ruby source code manually. Pre-compiled Ruby binaries help users quickly deploy applications that use different versions of Ruby on multiple machines.

RVM (Ruby Version Manager) is widely used for managing Ruby installations on Unix-like systems. RVM provides customized pre-compiled Ruby binaries tailored for various CPU architectures. These binaries offer additional features like readline support and SSL/TLS support. You can find them at RVM binaries.

The Need for pre-compiled Ruby Binaries

NeetoCI must execute user code in a containerized environment. A Ruby environment is essential for running Ruby on Rails applications. However, relying on the system's Ruby version is impractical since it may differ from the user's required version. Although rbenv or rvm can be used to install the necessary Ruby version, this approach could be slow. To save time, we chose to leverage pre-compiled Ruby binaries.

As a CI/CD system, NeetoCI must ensure that all versions of Ruby that an application requires are always available. Hence, we decided to build our binaries instead of relying on binaries provided by RVM. Also, this would allow us to do more system-specific optimizations to the Ruby binary at build time.

Building pre-compiled Ruby binaries

We built a Ruby binary following the official documentation . We were able to execute it on our local development machines. But the same binary ran into an error in our CI/CD environment.

$ bundle config path vendor/bundle
./ruby: bad interpreter: No such file or directory

To debug the issue, we initially focused on $PATH. However, even after resolving the $PATH issues, the problem persisted. We conducted a thorough investigation to identify the root cause. Unfortunately, not much was written on the Internet about this error. There was no mention of it in the official Ruby documentation.

As the next step, we decided to download the binary for version 3.2.2 from RVM. While examining the configuration file, we noticed that the following arguments were used with the configure command during the Ruby binary build process:

configure_args="'--prefix=/usr/share/rvm/rubies/ruby-3.2.2' '--enable-load-relative' '--sysconfdir=/etc' '--disable-install-doc' '--enable-shared'"

Here are the explanations of the configuration arguments:

1. --prefix=/usr/share/rvm/rubies/ruby-3.2.2: This specifies the directory where the Ruby binaries, libraries and other files will be kept after the installation is done.

2. --enable-load-relative: This specifies that Ruby can load relative paths for dynamically linked libraries. It allows the usage of relative paths instead of absolute paths when loading shared libraries. This feature can be beneficial in specific deployment scenarios.

3. --sysconfdir=/etc: This argument sets the directory where Ruby's system configuration files will be installed. In this case, it specifies the /etc directory as the location for these files.

4. --disable-install-doc: When this option is enabled, the installation of documentation files during the build process is disabled. This can help speed up the build process and save disk space, especially if you do not require the documentation files.

5. --enable-shared: Enabling this option allows the building of shared libraries for Ruby. Shared libraries enable Ruby to dynamically link and load specific functionality at runtime, leading to potential performance improvements and reduced memory usage.

In simpler terms, when the --enable-load-relative flag is enabled, the compiled Ruby binary can search for shared libraries in its own directory using the $ORIGIN variable.

When I built the binary on the Docker registry, then the passed --prefix was something like /usr/share/neetoci. When the binary is built, then binary had /usr/share/neetoci is hard-coded at various places. When we download this binary and use in CI then in the CI environment, Ruby is looking for /user/share/neetoci to load dependencies.

By enabling --enable-load-relative flag while building the binary Ruby will not use the hard coded value. Rather Ruby will use $ORIGIN variable and will search for the dependencies in the directory mentioned in $ORIGIN.

This is particularly helpful when the Ruby binary is relocated to a different directory or system. By using relative paths with $ORIGIN, the binary can find its shared libraries regardless of its new location. Without this flag, shared libraries are loaded using absolute paths, which can cause issues if the binary is moved to a different location and cannot locate its shared libraries.

In our specific use case, where we create and download binaries in separate containers, we encountered an error due to the absolute paths. To overcome this, we enabled the --enable-load-relative flag. This allowed the binary to find its shared libraries successfully, and it worked as expected in our CI/CD environment.

To display test suites in a tree structure, we need to store some information about the parent-child relationship in the database. This is where Ancestry comes in. Ancestry is a gem that allows Rails Active Record models to be organized as a tree structure.

Normally, web applications implement pagination to show a list of records. But, implementing pagination for a tree-structured data can be challenging and can make the application more complex. To avoid pagination, the application must have enough performance to display an entire tree having a large number of nodes/records without significant delays.

In this blog, we will discuss on how we leveraged the full potential of the Ancestry gem to address the performance issues encountered while listing suites.

1. Migration to materialized_path2

There are several ways to store hierarchical data in a relational database, such as materialized paths, closure tree tables, adjacency lists, and nested sets. Ancestry Gem uses the materialized path pattern to store hierarchical data.

The materialized path pattern is a technique in which a single node is stored in the database as a record and it has an additional column to store the hierarchical information. In the case of the Ancestry gem, this additional column is named as ancestry. The ancestry column is used to store IDs of the ancestors of a node as a single string, separated by a delimiter.

In order to understand how Ancestry gem uses materialized path pattern, first let's look at the nodes we have in our example. In the screenshot posted above, we see the following four nodes:

In our example, the s1 is a root node, s11 and s12 are the children of node s1, and s111 is the child of s11.

In order to store the hierarchical data, the gem offers two types of ancestry formats, materialized_path and materialized_path2. In these techniques, each node is represented by a record in the database. Our example consists of four nodes, so there will be four records in the database. The only difference between materialized_path and materialized_path2 lies in the format in which, the IDs are stored in the ancestry column.

materialized_path

Here the IDs of ancestors are stored in the format "id-1/id-2/id-3", were id-1, id-2 and id-3 are the IDs of three nodes with / as the delimiter. The id-1 is the root node, id-2 is the child of id-1 and id-3 is the child of id-2. In case of a root node, the ancestry will be null.

The table below shows how the suites in our example are stored in the database using materialized_path:

This arrangement of node IDs as a single string makes it easier to query for all descendants of a node, as we use SQL string functions to match the ancestry column. Here is the SQL statement to get descendants of suite s1:

SELECT "suites".* FROM "suites" WHERE ("suites"."ancestry" LIKE 's1/%' OR "suites"."ancestry" = 's1')

The result of the above query is:

materialized_path2

materialized_path2 stores ancestors in the format "/id-1/id-2/id-3/", were id-1 is the root node, id-2 is the child of id-1, and id-3 is the child of id-2. Here the delimiter is / as same as materialized_path, but the ancestry will be starting with a / and ending with a /. For a root node the ancestry will be /.

The table below shows how the suites in our example are stored in the database using materialized_path2:

The SQL statement to get the descendants of suite s1 is:

SELECT "suites".* FROM "suites" WHERE "suites"."ancestry" LIKE '/s1/%'

The result of above query is:

If we compare the 2 SQL queries, we can see that materialized_path2 has one less "OR" statement. This gives materialized_path2 a slight advantage in performance.

In NeetoTestify, we previously used the materialized_path format, but now we have migrated to materialized_path2 for improved performance.

2. Added collation to the ancestry column

Collation In database systems, the index specifies how data is sorted and compared in a database. Collation provides the sorting rules, case sensitivity, and accent sensitivity properties for the data in the database.

As mentioned above, our resulting query for fetching the descendants of a node would be

SELECT "suites".* FROM "suites" WHERE "suites"."ancestry" LIKE '/s1/%'

It uses a LIKE query for comparison with a wildcard character(%). In general, when using the LIKE query with a wildcard character (%) on the right-hand side of the pattern, the database can utilize an index and potentially optimize the query performance. This optimization holds true for ASCII strings.

However, it's important to note that this optimization may not necessarily hold true for Unicode strings, as Unicode characters can have varying lengths and different sorting rules compared to ASCII characters.

In our case, the ancestry column contains only ASCII strings. If we let the database know about this constraint, we can optimize the database's query performance. To do that, we need to specify the collation type of the ancestry column.

From Postgres's documentation:

The C and POSIX collations both specify “traditional C” behavior, in which only the ASCII letters “A” through “Z” are treated as letters, and sorting is done strictly by character code byte values.

Since we are using Postgres in NeetoTestify, we use collation C. Instead if we use MySQL, then the ancestry suggests using collation utf8mb4_bin.

class AddAncestryToTable < ActiveRecord::Migration[6.1]
  def change
    change_table(:table) do |t|
      # postgres
      t.string "ancestry", collation: 'C', null: false
      t.index "ancestry"
      # mysql
      t.string "ancestry", collation: 'utf8mb4_bin', null: false
      t.index "ancestry"
    end
  end
end

3. Usage of arrange method

Previously, we were constructing the tree structure of the suites by fetching the children of each node individually from the database. For this, we first fetched the suites whose ancestry is / (root nodes). Then, for each of these suites, we fetched their children, and repeated this process until we reached leaf-level suites.

This recursive approach results in a large number of database queries, causing performance issues as the tree size increases. Constructing a tree with n(4) nodes required n+1(5) database queries, adding to the complexity of the process.

The arrange method provided by Ancestry gem, converts the array of nodes into nested hashes, utilizing the ancestry column information. Also by using this method the number of database queries will remain 1, even if the number of suites and nested suites increases.

suites = project.suites
# SELECT "suites".* FROM "suites" WHERE "suites"."project_id" = "p1"
# [
#   <Suite id: "s1", project_id: "p1", name: "Suite 1", ancestry: "/">,
#   <Suite id: "s11", project_id: "p1", name: "Suite 1.1", ancestry: "/s1/">,
#   <Suite id: "s111", project_id: "p1", name: "Suite 1.1.1", ancestry: "/s1/s11/">,
#   <Suite id: "s12", project_id: "p1", name: "Suite 1.2", ancestry: "/s1/">
# ]

suites.arrange
# {
#   <Suite id: s1, project_id: "p1", name: "Suite 1", ancestry: "/"> => {
#     <Suite id: s11, project_id: "p1", name: "Suite 1.1", ancestry: "/s1/"> => {
#       <Suite id: s111, project_id: "p1", name: "Suite 1.1.1", ancestry: "/s1/s11/"> => {}
#     },
#     <Suite id: s12, project_id: "p1", name: "Suite 1.2", ancestry: "/s1/"> => {}
#   }
# }

The recursive approach took 5.72 seconds to retrieve 170 suites, but the array conversion approach of arrange method, retrieved the same number of suites in 728.72 ms.

The above image shows the advantage in performance while using the arrange method over the recursive approach. For the comparison of two approaches 170 suites and 1650 test cases were considered.

By implementing the above 3 best practices, we were able to considerably improve the overall performance of the application.

All three videos were prepared for my team members to show how to go about debugging. The videos are being presented "as it was recorded".

First Case Study

When a job fails in Sidekiq, Sidekiq puts that job in RetrySet and retries that job until the job succeeds or the job reaches the maximum number of retries. By default the maximum number of retries is 25. If a job fails 25 times, then that job is moved to the DeadSet. By default, Sidekiq will store up to 10,000 jobs in the deadset.

We had a situation where Redis was running out of memory. Here is how the debugging was done.

Watch on YouTube

How to inspect the deadset

ds = Sidekiq::DeadSet.new
ds.each do |job|
  puts "Job #{job['jid']}: #{job['class']} failed at #{job['failed_at']}"
end

Running the following to view the latest entry to the dataset:

ds.first
ds.count

To see the memory usage following commands were executed in the Redis console.

> memory usage dead
30042467

> type dead
zset

As discussed in the video large amount of payload was being sent. This is not the right way to send data to the worker. Ideally, some sort of id should be sent to the worker and the worker should be able to get the necessary data from the database based on the received id.

References

1. How to increase the number of jobs in the Sidekiq deadset or disable deadset
2. Maximum number of job retries in Sidekiq
3. Maximum number of jobs in Sidekiq Deadset

Second case study

In this case, the Redis instance of neetochat was running out of memory. The Redis instance had 50MB capacity, but we were getting the following error.

ERROR: heartbeat: OOM command not allowed when used memory > 'maxmemory'.

We were pushing too many geo info records to Redis and that caused the memory to fill up. Here is the video capturing the debugging session.

Watch on YouTube

The following are the commands executed while debugging.

> ping
PONG

> info

> info memory

> info keyspace

> keys *failed*

> keys *process*

> keys *geocoder*

> get getocoder:http://ipinfo.io/41.174.30.55/geo?

Third Case Study

In this case, the authentication service of Neeto was failing because of memory exhaustion.

Here the number of keys was limited, but the payload data was huge and all that payload data was hogging the memory. Here is the video capturing the debugging session.

Watch on YouTube

The following are the commands executed while debugging.

> ping

> info keyspace
db0:keys=106, expires=86,avg_ttl=1233332728573

> key * (to see all the keys)

The last command listed all 106 keys. Next, we needed to find how much memory each of these keys are using. For that, the following commands were executed.

> memory usage organizations/subdomains/bigbinary/neeto_app_links
736 bytes

> memory usage failed
10316224 (10MB)

> memory usage dead
29871174 (29MB)

We can see many implementations for calculating all subclasses of a particular class from the Ruby community with different gems. The ActiveSupport::DescendantsTracker is one of such implementations used in Rails framework. Finally, Ruby has added the Class#subclasses native implementation for it's 3.1 version release.

After Ruby 3.1

=> class User; end
=> class Employee < User; end
=> class Client < User; end

=> class Manager < Employee; end
=> class Developer < Employee; end

=> User.subclasses
=> [Employee, Client]

=> Employee.subclasses
=> [Manager, Developer]

=> Developer.subclasses
=> []

Here's the relevant pull request and feature discussion for this change.

=> str = "Ruby 3.1 introduces MatchData#match method"
=>  m = /(?<lang>\S+)(\s)(?<version>[\d\.]+)/.match(str)
=> #<MatchData "Ruby 3.1" lang:"Ruby" version:"3.1">

Before Ruby 3.1

We have MatchData#[] method to access all the substring groups that are matched. We can access either a single match or multiple matches by giving a single index or a range of indexes, respectively, using MatchData#[] method.

=> m[:lang]
=> "Ruby"

=> m[1]
=> "Ruby"
=> m[:version]
=> "3.1"

=> m[0..2]
=> ["Ruby 3.1", "Ruby", "3.1"]

=> m[1..2]
=> ["Ruby", "3.1"]


=> m[0].length
=> 8

After Ruby 3.1

We have two new methods now to access the match data & its length. The MatchData#match & MatchData#match_length methods accepts either an index or a symbol as an argument to return the match data & its length.

=> m.match(0)
=> "Ruby 3.1"

=> m.match(:version)
=> "3.1"

=> m.match(3)
=> nil

=> m.match_length(0)
=> 8

=> m.match_length(:version)
=> 3

=> m.match_length(3)
=> nil

Please note that MatchData#match is not same as MatchData#[] method. The later accepts an optional length or range as an argument to return the matched data but the former doesn't accept it, instead it allows only a single index or symbol as an argument.

Here's the relevant pull request and feature discussion for this change.

We already know Array#intersection or Array#& methods which are used to find the common elements between arrays.

=> x = [1, 2, 5, 8]
=> y = [2, 4, 5, 9]
=> z = [3, 7]

=> x.intersection(y) # x & y
=> [2, 5]

=> x.intersection(z) # x & z
=> []

The intersection or & methods return an empty array or array having the common elements in it as result. We have to further call empty?, any? or blank? like methods to check whether two arrays intersect each other or not.

Before Ruby 3.1

=> x.intersection(y).empty?
=> false

=> (x & z).empty?
=> true

=> (y & z).any?
=> false

After Ruby 3.1

=> x.intersect?(y)
=> true

=> y.intersect?(z)
=> false

The Array#intersect? method accepts only single array as argument, but Array#intersection method can accept multiple arrays as arguments.

=> x.intersection(y, z) # x & y & z
=> []

The newly introduced intersect? method is faster than the above described checks using intersection or & since the new method avoids creating an intermediate array while evaluating for common elements. Also new method returns true as soon as it finds a common element between arrays.

Here's the relevant pull request and feature discussion for this change.

Ruby 3.1 introduces an optional hash argument for the Enumerable#tally method to count. If a hash is given, the total number of occurrences of each element is added to the hash values and the final hash is returned.

Ruby 2.7.0+

=> letters = ["a", "b", "c", "a", "d", "c", "a", "c", "a"]
=> result = letters.tally
=> {"a"=>4, "b"=>1, "c"=>3, "d"=>1}

Before Ruby 3.1

=> new_letters = ["a", "b", "c", "a", "c", "a"]
=> new_letters.tally(result)
=> ArgumentError (wrong number of arguments (given 1, expected 0))

After Ruby 3.1

=> new_letters = ["a", "b", "c", "a", "c", "a"]
=> new_letters.tally(result)
=> {"a"=>7, "b"=>2, "c"=>5, "d"=>1}

The value corresponding to each element in the hash must be an integer. Otherwise, the method raises TypeError on execution.

If the default value is defined for the given hash, it will be ignored and the count of occurrences will be added in the returned hash.

=> letters = ["a", "b", "c", "a"]
=> letters.tally(Hash.new(10))
=> {"a"=>2, "b"=>1, "c"=>1}

Here's the relevant pull request and feature discussion for this change.

Ruby 3.1 introduces the compact method in the Enumerable module. Now we can use the compact method along with the Enumerator and Enumerator::Lazy classes which include the Enumerable module.

Before Ruby 3.1

=> enum = [1, nil, 3, nil, 5].to_enum
=> #<Enumerator: ...>

=> enum.compact
=> NoMethodError (undefined method `compact' for #<Enumerator: [1, nil, 3, nil, 5]:each>)

=>  enum.reject { |x| x.nil? }
=> [1, 3, 5]

After Ruby 3.1

=> enum = [1, nil, 3, nil, 5].to_enum
=> #<Enumerator: ...>

=> enum.compact
=> [1, 3, 5]

We can access the compact method to remove all nil occurrences from any classes where we include the Enumerable module.

class Person
  include Enumerable

  attr_accessor :names

  def initialize(names = [])
    @names = names
  end

  def each &block
    @names.each(&block)
  end
end

=> list = Person.new(["John", nil, "James", nil])
=> #<Person:0x0000000101cd3de8 @names=["John", nil, "James", nil]>

=> list.compact
=> ["John", "James"]

Similarly, lazy evaluation can be chained with the compact method to remove all nil entries from the Enumerator collection.

=> enum = [1, nil, 3, nil, 5].to_enum.lazy.compact
=> #<Enumerator::Lazy: ...>

=> enum.force
=> [1, 3, 5]


=> list = Person.new(["John", nil, "James", nil]).lazy.compact
=> #<Enumerator::Lazy: ...>

=> list.force
=> ["John", "James"]

Here's the relevant pull request and feature discussion for this change.

Ruby 3 major updates

The number 3 is very significant in the Ruby 3 release. Be it release version number, making performance 3x faster, or the trio of core contributors(Matz, TenderLove, Koichi). Similarly, there were 3 major goals of Ruby 3: being faster, having better concurrency, and ensuring correctness.

1. Ruby 3 performance

One of the major focuses for Ruby 3 was the performance. In fact, the initial discussion for Ruby 3 was started around it. Matz had set a very ambitious goal of making Ruby 3 times faster.

What is Ruby 3x3?

Before discussing this, let's revisit Ruby's core philosophy.

"I hope to see Ruby help every programmer in the world to be productive, and to enjoy programming, and to be happy." - Matz

About Ruby 3x3, some asked whether the goal was to make Ruby the fastest language? The answer is no. The main goal of Ruby 3x3 was to make Ruby 3 times faster than Ruby 2.

“No language is fast enough.” - Matz

Ruby was not designed to be fastest and if it would have been the goal, Ruby wouldn't be the same as it is today. As Ruby language gets performance boost, it definitely helps our application be faster and scalable.

"In the design of the Ruby language we have been primarily focused on productivity and the joy of programming. As a result, Ruby was too slow." - Matz

There are two areas where performance can be measured: memory and CPU.

CPU optimization

Some enhancements in Ruby internals have been made to improve the speed. The Ruby team has optimized the JIT(Just In Time) compiler from previous versions. Ruby MJIT compiler was first introduced in Ruby 2.6. Ruby 3 MJIT comes with better security and seems to improve web apps’ performance to a greater extent.

MJIT implementation is different from the usual JIT. When methods get called repeatedly e.g. 10000 times, MJIT will pick such methods which can be compiled into native code and put them into a queue. Later MJIT will fetch the queue and convert them to native code.

Please check JIT vs MJIT for more details.

Memory optimization

Ruby 3 comes with an enhanced garbage collector. It has python's buffer-like API which helps in better memory utilization. Since Ruby 1.8, Ruby has continuously evolved in Garbage collection algorithms.

Automatic Garbage Compaction

The latest change in garbage collection is Garbage Compaction. It was introduced in Ruby 2.7 where the process was a bit manual. But in version 3 it is fully automatic. The compactor is invoked aptly to make sure proper memory utilization.

Objects Grouping

The garbage compactor moves objects in the heap. It groups dispersed objects together at a single place in the memory so that this memory can be used even by heavier objects later.

2. Parallelism and Concurrency in Ruby 3

Concurrency is one of the important aspects of any programming language. Matz feels that Threads are not the right level of abstraction for Ruby programmers to use.

“I regret adding Threads.” - Matz

Ruby 3 makes it a lot easier to make applications where concurrency is a major focus. There are several features and improvements added in Ruby 3 related to concurrency.

Fibers

Fibers are considered a disruptive addition in Ruby 3. Fibers are light-weight workers which appear like Threads but have some advantages. It consumes less memory than Threads. It gives greater control to the programmer to define code segments that can be paused or resumed resulting in better I/O handling.

Falcon Rack web server uses Async Fibers internally. This allows Falcon to not block on I/O. Asynchronously managing I/O gives a great uplift to the Falcon server to serve requests concurrently.

Fiber Scheduler

Fiber Scheduler is an experimental feature added in Ruby 3. It was introduced to intercept blocking operations such as I/O. The best thing is that it allows lightweight concurrency and can easily integrate into the existing codebase without changing the original logic. It's an interface and that can be implemented by creating a wrapper for a gem like EventMachine or Async. This interface design allows separation of concerns between the event loop implementation and the application code.

Following is an example to send multiple HTTP requests concurrently using Async.

require 'async'
require 'net/http'
require 'uri'

LINKS = [
  'https://bigbinary.com',
  'https://basecamp.com'
]

Async do
  LINKS.each do |link|
    Async do
      Net::HTTP.get(URI(link))
    end
  end
end

Please check fibers for more details.

Ractors (Guilds)

As we know Ruby’s global VM lock (GVL) prevents most Ruby Threads from computing in parallel. Ractors work around the GVL to offer better parallelism. Ractor is an Actor-Model like concurrent abstraction designed to provide a parallel execution without thread-safety concerns.

Ractors allows Threads in different Ractors to compute at the same time. Each Ractor has at least one thread, which may contain multiple fibers. In a Ractor, only a single thread is allowed to execute at a given time.

The following program returns the square root of a really large number. It calculates the result for both numbers in parallel.

# Math.sqrt(number) in ractor1, ractor2 run in parallel

ractor1, ractor2 = *(1..2).map do
  Ractor.new do
    number = Ractor.recv
    Math.sqrt(number)
  end
end

# send parameters
ractor1.send 3**71
ractor2.send 4**51

p ractor1.take #=> 8.665717809264115e+16
p ractor2.take #=> 2.251799813685248e+15

3. Static Analysis

We need tests to ensure correctness of our program. However by its very nature tests could mean code duplication.

“I hate tests because they aren't DRY.” - Matz

To ensure the correctness of a program, static analysis can be a great tool in addition to tests.

The static analysis relies on inline type annotations which aren't DRY. The solution to address this challenge is having .rbs files parallel to our .rb files.

RBS

RBS is a language to describe the structure of a Ruby program. It provides us an overview of the program and how overall classes, methods, etc. are defined. Using RBS, we can write the definition of Ruby classes, modules, methods, instance variables, variable types, and inheritance. It supports commonly used patterns in Ruby code, and advanced types like unions and duck typing.

The .rbs files are something similar to .d.ts files in TypeScript. Following is a small example of how a .rbs file looks like. The advantage of having a type definition is that it can be validated against both implementation and execution.

The below example is pretty self-explanatory. One thing we can note here though, each_post accepts a block or returns an enumerator.

# user.rbs

class User
  attr_reader name: String
  attr_reader email: String
  attr_reader age: Integer
  attr_reader posts: Array[Post]

  def initialize: (name: String,
                   email: String,
                   age: Integer) -> void

  def each_post: () { (Post) -> void } -> void
                   | () -> Enumerator[Post, void]
end

Please check RBS gem documentation for more details.

Typeprof

Introducing type definition was a challenge because there is already a lot of existing Ruby code around and we need a tool that could automatically generate the type signature. Typeprof is a type analysis tool that reads plain Ruby code and generates a prototype of type signature in RBS format by analyzing the methods, and its usage. Typeprof is an experimental feature. Right now only small subset of ruby is supported.

“Ruby is simple in appearance, but is very complex inside, just like our human body.” - Matz

Let's see an example.

# user.rb
class User

  def initialize(name:, email:, age:)
    @name, @email, @age = name, email, age
  end

  attr_reader :name, :email, :age
end

User.new(name: "John Doe", email: 'john@example.com', age: 23)

Output

$ typeprof user.rb

# Classes
class User

  attr_reader name : String
  attr_reader email : String
  attr_reader age : Integer

  def initialize : (name: String,
                    email: String,
                    age: Integer) -> [String, String, Integer]

end

Other Ruby 3 features and changes

In the 7-year period, the Ruby community has seen significant improvement in performance and other aspects. Apart from major goals, Ruby 3 is an exciting update with lots of new features, handy syntactic changes, and new enhancements. In this section, we will discuss some notable features.

“We are making Ruby even better.” - Matz

One-line pattern matching syntax change

Previously one-line pattern matching used the keyword in. Now it's replaced with =>.

Ruby 2.7

  { name: 'John', role: 'CTO' } in {name:}
  p name # => 'John'

Ruby 3.0

  { name: 'John', role: 'CTO' } => {name:}
  p name # => 'John'

Find pattern

The find pattern was introduced in Ruby 2.7 as an experimental feature. This is now part of Ruby 3.0. It is similar to pattern matching in Elixir or Haskell.

users = [
  { name: 'Oliver', role: 'CTO' },
  { name: 'Sam', role: 'Manager' },
  { role: 'customer' },
  { name: 'Eve', city: 'New York' },
  { name: 'Peter' },
  { city: 'Chicago' }
]

users.each do |person|
  case person
  in { name:, role: 'CTO' }
    p "#{name} is the Founder."
  in { name:, role: designation }
    p "#{name} is a #{designation}."
  in { name:, city: 'New York' }
    p "#{name} lives in New York."
  in {role: designation}
    p "Unknown is a #{designation}."
  in { name: }
    p "#{name}'s designation is unknown."
  else
    p "Pattern not found."
  end
end

"Oliver is the Founder."
"Sam is a Manager."
"Unknown is a customer."
"Eve lives in New York."
"Peter's designation is unknown."
"Pattern not found."

Endless Method definition

This is another syntax enhancement that is optional to use. It enables us to create method definitions without end keyword.

def: increment(x) = x + 1

p increment(42) #=> 43

Except method in Hash

Sometimes while working on a non Rails app I get undefined method except. The except method was available only in Rails. In Ruby 3 Hash#except was added to Ruby itself.

user = { name: 'Oliver', age: 29, role: 'CTO' }

user.except(:role) #=> {:name=> "Oliver", :age=> 29}

Memory View

This is again an experimental feature. This is a C-API that will allow extension libraries to exchange raw memory area. Extension libraries can also share metadata of memory area that consists of shape and element format. It was inspired by Python’s buffer protocol.

Arguments forwarding

Arguments forwarding (...) now supports leading arguments.

It is helpful in method_missing, where we need method name as well.

def method_missing(name, ...)
  if name.to_s.end_with?('?')
    self[name]
  else
    fallback(name, ...)
  end
end

Other Notable changes

• Pasting in IRB is much faster.
• The order of backtrace had been reversed. The error message and line number are printed first, rest of the backtrace is printed later.
• Hash#transform_keys accepts a hash that maps old keys with new keys.
• Interpolated String literals are no longer frozen when # frozen-string-literal: true is used.
• Symbol#to_proc now returns a lambda Proc.
• Symbol#name has been added, which returns the symbol's name as a frozen string.

Many other changes can be checked at Ruby 3 News for more details.

Transition

A lot of core libraries have been modified to fit the Ruby 3 goal needs. But this doesn't mean that our old applications will suddenly stop working. The Ruby team has made sure that these changes are backward compatible. We might see some deprecation warnings in our existing code. The developers can fix these warnings to smoothly transition from an old version to the new version. We are all set to use new features and get all new performance improvements.

Conclusion

With great improvements in performance, memory utilization, static analysis, and new features like Ractors and schedulers, we have great confidence in the future of Ruby. With Ruby 3, the applications can be more scalable and more enjoyable to work with. The coming 2021 is not just a new year but rather a new era for all Rubyists. We at BigBinary thank everyone who contributed towards the Ruby 3 release directly or indirectly.

Happy Holidays and Happy New Year folks!!!

Usage before Ruby 3

The following example shows how we used to apply transform_keys:

# 1. Declare address hash
irb(main)> address = {House: 'Kizhakkethara', house_no: 123, locality: 'India'}
=> {:House=>"Kizhakkethara", :house_no=>123, :locality=>"India"}

# 2. Lowercase all the keys
irb(main)> address.transform_keys(&:downcase)
=> {:house=>"Kizhakkethara", :house_no=>123, :locality=>"India"}

# 3. Replace a particular key with a new key along with lowercasing
irb(main)* address.transform_keys do |key|
irb(main)*   new_key = key
irb(main)*   if key == :locality
irb(main)*     new_key = :country
irb(main)*   end
irb(main)*   new_key.to_s.downcase.to_sym
irb(main)> end
=> {:house=>"Kizhakkethara", :house_no=>123, :country=>"India"}

Although the changes required are trivial, we ended up writing a block to do the job. But what happens when the number of keys that needs to be transformed increases? Do we need to write n-number of conditions within a block? Not anymore!

Introducing Hash#transform_keys with hash argument

Let's take the same example and provide a hash, which will be used for the transformation:

# 1. Declare address hash
irb(main)> address = {House: 'Kizhakkethara', house_no: 123, locality: 'India'}
=> {:House=>"Kizhakkethara", :house_no=>123, :locality=>"India"}

# 2. Provide hash with transform_keys
irb(main)> address.transform_keys({House: :house, locality: :country})
=> {:house=>"Kizhakkethara", :house_no=>123, :country=>"India"}

That does the job. But let's try to improve this code. Ultimately what happens when we invoke that method is that it goes through each of the keys in our variable and maps the existing keys to the new keys. The transform_keys method accepts a block as a parameter. Thus let's pass in the downcase method as a Proc argument:

# 1. Passing in block parameters
irb(main)> address.transform_keys({locality: :country}, &:downcase)
=> {:house=>"Kizhakkethara", :house_no=>123, :country=>"India"}

An important point to be noted about the block parameter is that, it's only applied to keys which are not specified in the hash argument.

Other common use cases

Transforming params received in the Rails controller

# 1. Declare params
irb(rails)> params = ActionController::Parameters.new({"firstName"=>"oliver", "lastName"=>"smith", "email"=>"oliver@bigbinary.com"})
=> <ActionController::Parameters {"firstName"=>"oliver", "lastName"=>"smith", "email"=>"oliver@bigbinary.com"} permitted: false>

# 2. Convert camelCase to snake_case using block parameter
irb(rails)> params.permit(:firstName, :lastName, :email).transform_keys(&:underscore)
=> <ActionController::Parameters {"first_name"=>"oliver", "last_name"=>"smith", "email"=>"oliver@bigbinary.com"} permitted: true>

# 3. Or using hash argument
irb(rails)> params.permit(:firstName, :lastName, :email).transform_keys({firstName: 'first_name', lastName: 'last_name'})
=> <ActionController::Parameters {"first_name"=>"oliver", "last_name"=>"smith", "email"=>"oliver@bigbinary.com"} permitted: true>

Slicing hash along with key transformation

irb(main)> address.transform_keys({locality: :country}).slice(:house_no, :country)
=> {:house_no=>123, :country=>"India"}

Transforming keys in place using bang counterpart

irb(main)> address.transform_keys!({locality: :country}, &:downcase)
irb(main)> address
=> {:house=>"Kizhakkethara", :house_no=>123, :country=>"India"}

References

• Discussions regarding this feature can be found here.
• Commit for this feature can be found here.

Why do we need Hash#except?

At times, we need to print or log everything except some sensitive data. Let's say we want to print user details in the logs but not passwords.

Before Ruby 3 we could have achieved it in the following ways:

irb(main):001:0> user_details = { name: 'Akhil', age: 25, address: 'India', password: 'T:%g6R' }

# 1. Reject with a block and include?
irb(main):003:0> puts user_details.reject { |key, _| key == :password }
=> { name: 'Akhil', age: 25, address: 'India' }

# 2. Clone the hash with dup, tap into it and delete that key/value from the clone
irb(main):005:0> puts user_details.dup.tap { |hash| hash.delete(:password) }
=> { name: 'Akhil', age: 25, address: 'India' }

We know that ActiveSupport already comes with Hash#except but for a simple Ruby application using ActiveSupport would be overkill.

Ruby 3

To make the above task easier and more explicit, Ruby 3 adds Hash#except to return a hash excluding the given keys and their values:

irb(main):001:0> user_details = { name: 'Akhil', age: 25, address: 'India', password: 'T:%g6R' }
irb(main):002:0> puts user_details.except(:password)
=> { name: 'Akhil', age: 25, address: 'India' }

irb(main):003:0> db_info = YAML.safe_load(File.read('./database.yml'))
irb(main):004:0> puts db_info.except(:username, :password)
=> { port: 5432, database_name: 'example_db_production' }

Check out the commit for more details. Discussion around it can be found here.

One of the important aspects of Ruby 3.0 is optimization. The part of that optimization is the introduction of name method for Symbol. In this blog, we will take a look at what name method of class Symbol does and why it was introduced. The new name method is introduced on Symbol to simply convert a symbol into a string. Symbol#name returns a string. Let's see how it works.

irb(main):001:0> :simba.name
=> 'simba'
irb(main):002:0> :simba.name.class
=> String
irb(main):003:0> :simba.name === :simba.name
=> true

Wait what? Don't we have to_s to convert a symbol into a string. Most of us have used to_s method on a Symbol. The to_s method returns a String object and we can simply use it. But why name?

Using to_s is okay in most cases. But the problem with to_s is that it creates a new String object every time we call it on a symbol. We can verify this in irb.

irb(main):023:0> :simba.to_s.object_id
=> 260
irb(main):024:0> :simba.to_s.object_id
=> 280

Creating a new object for every symbol to a string conversion allocates new memory which increases overhead. The light was thrown on this issue by schneems (Richard Schneeman) in a talk at RubyConf Thailand where he showed how Symbol#to_s allocation causes significant overhead in ActiveRecord. This inspired Ruby community to have a new method name on Symbol which returns a frozen string object. This reduces the string allocations dramatically which results in reducing overhead.

irb(main):001:0> :simba.name.frozen?
=> true
irb(main):002:0> :simba.name.object_id
=> 200
irb(main):003:0> :simba.name.object_id
=> 200

The reason to bring this feature was that most of the times we want a simple string representation for displaying purpose or to interpolate into another string. The result of to_s is rarely mutated directly. By introducing this method we save a lot of objects which helps in optimization. Now we know the benefits of name, we should prefer using name over to_s when we don't want to mutate a string.

For more information on discussion, official documentation, please head on to Feature #16150 discussion, Pull request and Ruby 3.0 official release preview.

# endless method definition
>> def raise_to_power(number, power) = number ** power

>> raise_to_power(2, 5)

=> 32

The discussion around it can be found here. Check out the pull request for more details on this.

(..100) is a Beginless Range and it is equivalent to (nil..100).

Let's see how Beginless Range could be used.

> array = (1..10).to_a

# Select first 6 elements
> array[..5]
=> [1, 2, 3, 4, 5, 6]

# Select first 5 elements
> array[...5]
=> [1, 2, 3, 4, 5]

# grep (INFINITY..5) in (1..5)
> (1..10).grep(..5)
=> [1, 2, 3, 4, 5]

# (..100) is equivalent to (nil..100)
> (..100) == (nil..100)
=> true

Here is another example where in the case statement the condition can be read as below the specified level.

case temperature
when ..-15
  puts "Deep Freeze"
when -15..8
  puts "Refrigerator"
when 8..15
  puts "Cold"
when 15..25
  puts "Room Temperature"
when (25..)   # Kindly notice the brackets here
  puts "Hot"
end

It can also be used for defining constants for ranges.

TEMPERATURE = {
  ..-15  => :deep_freeze,
  -15..8 => :refrigerator,
  8..15  => :cold,
  15..25 => :room_temperature,
  25..   => :hot
end

Using Beginless Range in DSL makes it easier to write conditions and it looks more natural.

# In Rails
User.where(created_at: (..DateTime.now))
# User Load (2.2ms)  SELECT "users".* FROM "users" WHERE "users"."created_at" <= $1 LIMIT $2  [["created_at", "2020-08-05 15:00:19.111217"], ["LIMIT", 11]]


# In RubySpec
ruby_version(..'1.9') do
# Tests for old Ruby
end

Here is the relevant commit and discussion regarding this change.

Before Ruby 2.7, we could have achieved the same with 2 iterations using select & map combination or map & compact combination.

irb> numbers = [3, 6, 7, 9, 5, 4]

# we can use select & map to find square of odd numbers

irb> numbers.select { |x| x.odd? }.map { |x| x**2 }
=> [9, 49, 81, 25]

# or we can use map & compact to find square of odd numbers

irb> numbers.map { |x| x**2 if x.odd? }.compact
=> [9, 49, 81, 25]

Ruby 2.7

Ruby 2.7 adds Enumerable#filter_map which can be used to filter & map the elements in a single iteration and which is more faster when compared to other options described above.

irb> numbers = [3, 6, 7, 9, 5, 4]
irb> numbers.filter_map { |x| x**2 if x.odd? }
=> [9, 49, 81, 25]

The original discussion had started 8 years back. Here is the latest thread and github commit for reference.

Ruby 2.7 NEWS has listed the spec of keyword arguments for Ruby 3.0. We will take the examples mentioned there and for each scenario we will look into how we can fix them in the existing codebase.

Scenario 1

When method definition accepts keyword arguments as the last argument.

def sum(a: 0, b: 0)
  a + b
end

Passing exact keyword arguments in a method call is acceptable, but in applications we usually pass a hash to a method call.

sum(a: 2, b: 4) # OK

sum({ a: 2, b: 4 }) # Warned

In this case, we can add a double splat operator to the hash to avoid deprecation warning.

sum(**{ a: 2, b: 4 }) # OK

Scenario 2

When method call passes keyword arguments but does not pass enough required positional arguments.

If the number of positional arguments doesn't match with method definition, then keyword arguments passed in method call will be considered as the last positional argument to the method.

def sum(num, x: 0)
  num.values.sum + x
end

sum(a: 2, b: 4) # Warned

sum(a: 2, b: 4, x: 6) # Warned

To avoid deprecation warning and for code to be compatible with Ruby 3, we should pass hash instead of keyword arguments in method call.

sum({ a: 2, b: 4 }) # OK

sum({ a: 2, b: 4}, x: 6) # OK

Scenario 3

When a method accepts a hash and keyword arguments but method call passes only hash or keyword arguments.

If a method arguments are a mix of symbol keys and non-symbol keys, and the method definition accepts either one of them then Ruby splits the keyword arguments but also raises a warning.

def sum(num={}, x: 0)
  num.values.sum + x
end

sum("x" => 2, x: 4) # Warned

sum(x: 2, "x" => 4) # Warned

To fix this warning, we should pass hash separately as defined in the method definition.

sum({ "x" => 4 }, x: 2) # OK

Scenario 4

When an empty hash with double splat operator is passed to a method that doesn't accept keyword arguments.

Passing keyword arguments using double splat operator to a method that doesn't accept keyword argument will send empty hash similar to earlier version of Ruby but will raise a warning.

def sum(num)
  num.values.sum
end

numbers = {}
sum(**numbers) # Warned

To avoid this warning, we should change method call to pass hash instead of using double splat operator.

numbers = {}
sum(numbers) # OK

Added support for non-symbol keys

In Ruby 2.6.0, support for non-symbol keys in method call was removed. It is added back in Ruby 2.7. When method accepts arbitrary keyword arguments using double splat operator then non-symbol keys can also be passed.

def sum(**num)
  num.values.sum
end

ruby 2.6.5

sum("x" => 4, "y" => 3)
=> ArgumentError (wrong number of arguments (given 1, expected 0))

sum(x: 4, y: 3)
=> 7

ruby 2.7.0

sum("x" => 4, "y" => 3)
=> 7

sum(x: 4, y: 3)
=> 7

Added support for **nil

Ruby 2.7 added support for **nil to explicitly mention if a method doesn't accept any keyword arguments in method call.

def sum(a, b, **nil)
  a + b
end

sum(2, 3, x: 4)
=> ArgumentError (no keywords accepted)

To suppress above deprecation warnings we can use -W:no-deprecated option.

In conclusion, Ruby 2.7 has worked big steps towards changing specification of keyword arguments which will be completely changed in Ruby 3.

For more information on discussion, code changes and official documentation, please head to Feature #14183 discussion, pull request and NEWS release.

By applying Enumerable#lazy method on any enumerable object, we can convert that object into Enumerator::Lazy object. The chains of actions on this lazy enumerator will be evaluated only when it is needed. It helps us in processing operations on large collections, files and infinite sequences seamlessly.

# This line of code will hang and you will have to quit the console by Ctrl+C.
irb> list = (1..Float::INFINITY).select { |i| i%3 == 0 }.reject(&:even?)

# Just adding `lazy`, the above line of code now executes properly
# and returns result without going to infinite loop. Here the chains of
# operations are performed as and when it is needed.
irb> lazy_list = (1..Float::INFINITY).lazy.select { |i| i%3 == 0 }.reject(&:even?)
=> #<Enumerator::Lazy: ...>

irb> lazy_list.first(5)
=> [3, 9, 15, 21, 27]

When we chain more operations on Enumerable#lazy object, it again returns lazy object without executing it. So, when we pass lazy objects to any method which expects a normal enumerable object as an argument, we have to force evaluation on lazy object by calling to_a method or it's alias force.

# Define a lazy enumerator object.
irb> list = (1..30).lazy.select { |i| i%3 == 0 }.reject(&:even?)
=> #<Enumerator::Lazy: #<Enumerator::Lazy: ... 1..30>:select>:reject>

# The chains of operations will return again a lazy enumerator.
irb> result = list.select { |x| x if x <= 15 }
=> #<Enumerator::Lazy: #<Enumerator::Lazy: ... 1..30>:select>:reject>:map>

# It returns error when we call usual array methods on result.
irb> result.sample
irb> NoMethodError (undefined method `sample'
irb> for #<Enumerator::Lazy:0x00007faab182a5d8>)

irb> result.length
irb> NoMethodError (undefined method `length'
irb> for #<Enumerator::Lazy:0x00007faab182a5d8>)

# We can call the normal array methods on lazy object after forcing
# its actual execution with methods as mentioned above.
irb> result.force.sample
=> 9

irb> result.to_a.length
=> 3

The Enumerable#eager method returns a normal enumerator from a lazy enumerator, so that lazy enumerator object can be passed to any methods which expects a normal enumerable object as an argument. Also, we can call other usual array methods on the collection to get desired results.

# By adding eager on lazy object, the chains of operations would return
# actual result here. If lazy object is passed to any method, the
# processed result will be received as an argument.
irb> eager_list = (1..30).lazy.select { |i| i%3 == 0 }.reject(&:even?).eager
=> #<Enumerator: #<Enumerator::Lazy: ... 1..30>:select>:reject>:each>

irb> result = eager_list.select { |x| x if x <= 15 }
irb> result.sample
=> 9

irb> result.length
=> 3

The same way, we can use eager method when we pass lazy enumerator as an argument to any method which expects a normal enumerator.

irb> list = (1..10).lazy.select { |i| i%3 == 0 }.reject(&:even?)
irb> def display(enum)
irb>   enum.map { |x| p x }
irb> end

irb>  display(list)
=> #<Enumerator::Lazy: #<Enumerator::Lazy: ... 1..30>:select>:reject>:map>

irb> eager_list = (1..10).lazy.select { |i| i%3 == 0 }.reject(&:even?).eager
irb> display(eager_list)
=> 3
=> 9

Here's the relevant commit and feature discussion for this change.

> (1..10).each { |n| p n * 3 }

> { a: [1, 2, 3], b: [2, 4, 6], c: [3, 6, 9] }.each { |_k, v| p v }

> [10, 100, 1000].each_with_index { |n, i| p n, i }

Ruby 2.7 introduces a new way to access block parameters. Ruby 2.7 onwards, if block parameters are obvious and we wish to not use absurd names like n or i etc, we can use numbered parameters which are available inside a block by default.

We can use **1_ for first parameter, **2_ for second parameter and so on.

Here's how Ruby 2.7 provides numbered parameters inside a block. Below shown are the examples from above, only this time using numbered parameters.

> (1..10).each { p _1 * 3 }

> { a: [1, 2, 3], b: [2, 4, 6], c: [3, 6, 9] }.each { p _2 }

> [10, 100, 1000].each_with_index { p _1, _2 }

Like mentioned in News-2.7.0 docs, Ruby now raises a warning if we try to define local variable in the format _1. Local variable will have precedence over numbered parameter inside the block.

> _1 = 0
> => warning: `_1' is reserved for numbered parameter; consider another name

> [10].each { p _1 }
> => 0

Numbered parameters are not accessible inside the block if we define ordinary parameters. If we try to access _1 when ordinary parameters are defined, then ruby raises SyntaxError like shown below.

> ["a", "b", "c"].each_with_index { |alphabet, index| p _1, _2}

=> SyntaxError ((irb):1: ordinary parameter is defined)

This feature was suggested 9 years back and came back in discussion last year. After many suggestions community agreed to use _1 syntax.

Head to following links to read the discussion behind numbered parameters, Feature #4475 and Discussion #15723.

Here's relevant commit for this feature.

Before Ruby 2.7, we could have achieved it using group_by or inject.

irb> scores = [100, 35, 70, 100, 70, 30, 35, 100, 45, 30]

# we can use group_by to group the scores

irb> scores.group_by { |v| v }.map { |k, v| [k, v.size] }.to_h
=> {100=>3, 35=>2, 70=>2, 30=>2, 45=>1}

# or we can use inject to group the scores

irb> scores.inject(Hash.new(0)) {|hash, score| hash[score] += 1; hash }
=> {100=>3, 35=>2, 70=>2, 30=>2, 45=>1}

Ruby 2.7

Ruby 2.7 adds Enumerable#tally which can be used to find the frequency. Tally makes the code more readable and intuitive. It returns a hash where keys are the unique elements and values are its corresponding frequency.

irb> scores = [100, 35, 70, 100, 70, 30, 35, 100, 45, 30]
irb> scores.tally
=> {100=>3, 35=>2, 70=>2, 30=>2, 45=>1}

Check out the github commit for more details on this.

JIT stands for Just-In-Time compiler. JIT converts repetitive code into bytecode which can then be sent to the processor directly, hence, saving time by not compiling the same piece of code over and over.

Ruby 2.6

MJIT is introduced in Ruby 2.6. It is most commonly known as MRI JIT or Method Based JIT.

It is a part of the Ruby 3x3 project started by Matz. The name "Ruby 3x3" signifies Ruby 3.0 will be 3 times faster than Ruby 2.0 and it will focus mainly on performance. In addition to performance, it also aims for the following things:

1. Portability
2. Stability
3. Security

MJIT is still in development, therefore, MJIT is optional in Ruby 2.6. If you are running Ruby 2.6, then you can execute the following command.

ruby --help

You will see following options.

--Jit-wait # Wait program execution until code compiles.
--jit-verbose=num # Level information MJIT compiler prints for Ruby program.
--jit-min-calls=num # Minimum count in loops for which MJIT should work.
--jit-max-cache
--jit-save-temps # Save compiled library to the file.

Vladimir Makarov proposed improving performance by replacing VM instructions with RTL(Register Transfer Language) and introducing the Method based JIT compiler.

Vladimir explained MJIT architecture in his RubyKaigi 2017 conference keynote.

Ruby's compiler converts the code to YARV(Yet Another Ruby VM) instructions and then these instructions are run by the Ruby Virtual Machine. Code that is executed too often is converted to RTL instructions, which runs faster.

Let's take a look at how MJIT works.

# mjit.rb

require 'benchmark'

puts Benchmark.measure {
def test_while
start_time = Time.now
i = 0

    while i < 4
      i += 1
    end

    i
    puts Time.now - start_time

end

4.times { test_while }
}

Let's run this code with MJIT options and check what we got.

ruby --jit --jit-verbose=1 --jit-wait --disable-gems mjit.rb

Time taken is 4.0e-06
Time taken is 0.0
Time taken is 0.0
Time taken is 0.0
0.000082 0.000032 0.000114 ( 0.000105)
Successful MJIT finish

Nothing interesting right? And why is that? because we are iterating the loop for 4 times and default value for MJIT to work is 5. We can always decide after how many calls MJIT should work by providing --jit-min-calls=#number option.

Let's tweak the program a bit so MJIT gets to work.

require 'benchmark'

puts Benchmark.measure {
def test_while
start_time = Time.now
i = 0

    while i < 4_00_00_000
      i += 1
    end

    puts "Time taken is #{Time.now - start_time}"

end

10.times { test_while }
}

After running the above code we can see some work done by MJIT.

Time taken is 0.457916
Time taken is 0.455921
Time taken is 0.454672
Time taken is 0.452823
JIT success (72.5ms): block (2 levels) in <main>@mjit.rb:15 -> /var/folders/v6/\_6sh53vn5gl3lct18w533gr80000gn/T//\_ruby_mjit_p66220u0.c
JIT success (140.9ms): test_while@mjit.rb:4 -> /var/folders/v6/\_6sh53vn5gl3lct18w533gr80000gn/T//\_ruby_mjit_p66220u1.c
JIT compaction (23.0ms): Compacted 2 methods -> /var/folders/v6/\_6sh53vn5gl3lct18w533gr80000gn/T//\_ruby_mjit_p66220u2.bundle
Time taken is 0.463703
Time taken is 0.102852
Time taken is 0.103335
Time taken is 0.103299
Time taken is 0.103252
Time taken is 0.103261
2.797843 0.005357 3.141944 ( 2.801391)
Successful MJIT finish

Here's what's happening. Method ran 4 times and on the 5th call it found it is running same code again. So MJIT started a separate thread to convert the code into RTL instructions, which created a shared object library. Next, threads took that shared code and executed directly. As we passed option --jit-verbose=1 we can see what MJIT did.

What we are seeing in output is the following:

1. Time taken to compile.
2. What block of code is compiled by JIT.
3. Location of compiled code.

We can open the file and see how MJIT converted the piece of code to binary instructions but for that we need to pass another option which is --jit-save-temps and then just inspect those files.

After compiling the code to RTL instructions, take a look at the execution time. It dropped down to 0.10 ms from 0.46 ms. That's a neat speed bump.

Here is a comparison across some of the Ruby versions for some basic operations.


Rails comparison on Ruby 2.5, Ruby 2.6 and Ruby 2.6 with JIT

Create a rails application with different Ruby versions and start a server. We can start the rails server with the JIT option, as shown below.

RUBYOPT="--jit" bundle exec rails s

Now, we can start testing the performance on servers. We found that Ruby 2.6 is faster than Ruby 2.5, but enabling JIT in Ruby 2.6 does not add more value to the Rails application.

MJIT status and future directions

• It is in an early development stage.
• Does not work on windows.
• Needs more time to mature.
• Needs more optimisations.
• MJIT can use GCC or LLVM in the future C Compilers.

Further reading

1. Ruby 3x3 Performance Goal
2. The method JIT compiler for Ruby2.6
3. Vladimir Makarov's Ruby Edition

(1..10).cover?(5)
=> true

Range#cover? returns false if the object passed as an argument is non-comparable or is not in the range.

Before Ruby 2.6, Range#cover? used to return false if a Range object is passed as an argument.

>> (1..10).cover?(2..5)
=> false

Ruby 2.6

In Ruby 2.6 Range#cover? can accept a Range object as an argument. It returns true if the argument Range is equal to or a subset of the Range.

(1..100).cover?(10..20)
=> true

(1..10).cover?(2..5)
=> true

(5..).cover?(4..)
=> false

("a".."d").cover?("x".."z")
=> false

Here is relevant commit and discussion for this change.

As of now, RubyVM::AST supports two methods: parse and parse_file.

parse method takes a string as a parameter and returns the root node of the tree in the form of the object, RubyVM::AST::Node (Link is not available).

parse_file method takes the file name as a parameter and returns the root node of the tree in the form of the object, RubyVM::AST::Node.

Ruby 2.6.0-preview2

irb> RubyVM::AST.parse("(1..100).select { |num| num % 5 == 0 }")
=> #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >

irb> RubyVM::AST.parse_file("/Users/amit/app.rb")
=> #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >

RubyVM::AST::Node has seven public instance methods - children, first_column, first_lineno, inspect, last_column, last_lineno and type.

Ruby 2.6.0-preview2

irb> ast_node = RubyVM::AST.parse("(1..100).select { |num| num % 5 == 0 }")
=> #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >

irb> ast_node.children
=> [nil, #<RubyVM::AST::Node(NODE_ITER(9) 1:0, 1:38): >]

irb> ast_node.first_column
=> 0

irb> ast_node.first_lineno
=> 1

irb> ast_node.inspect
=> "#<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:38): >"

irb> ast_node.last_column
=> 38

irb> ast_node.last_lineno
=> 1

irb> ast_node.type
=> "NODE_SCOPE"

This module will majorly help in building a static code analyzer and formatter.

Let's take an example in which we have to select/filter all numbers which are divisible by 5 from a range.

Ruby 2.5

irb> (1..100).select { |num| num % 5 == 0 }
=> [5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]

irb> (1..100).filter { |num| num % 5 == 0 }
=> Traceback (most recent call last):
2: from /Users/amit/.rvm/rubies/ruby-2.5.1/bin/irb:11:in `<main>' 1: from (irb):2 NoMethodError (undefined method`filter' for 1..100:Range)

Ruby 2.6.0-preview2

irb> (1..100).select { |num| num % 5 == 0 }
=> [5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]

irb> (1..100).filter { |num| num % 5 == 0 }
=> [5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]

Also note, along with Enumerable#filter, Enumerable#filter!, Enumerable#select! was also added as an alias.

Here is the relevant commit and discussion.

Below code will raise class/module name must be CONSTANT exception.

  class Большойдвоичный
  end

We can use above non-ASCII character as a method name or variable name though.

Below code will run without any exception

  class NonAsciiMethodAndVariable
    def Большойдвоичный
      Имя = "BigBinary"
    end
  end

"Имя" is treated as a variable name in above example, even though first letter(И) is a capital non-ASCII character.

Ruby 2.6

Ruby 2.6 relaxes above mentioned limitation. We can now define constants in languages other than English. Languages having capital letters like Russian and Greek can be used to define constant name.

Below code will run without exception in any Ruby 2.6.

  class Большойдвоичный
  end

As capital non-Ascii characters are now treated as constant, below code will raise a warning in Ruby 2.6.

  irb(main):001:0> Имя = "BigBinary"
  => "BigBinary"
  irb(main):002:0> Имя = "BigBinary"
  (irb):2: warning: already initialized constant Имя
  (irb):1: warning: previous definition of Имя was here

Above code will run without any warnings on Ruby versions prior to 2.6

Here is relevant commit and discussion for this change.

An alternative Geocoder

Our search for an alternative geocoder landed us on Nominatim. Written in C, with a PHP web interface, Nominatim was performant enough for our requirements. Once set up, Nominatim required 8GB of RAM to run and this included RAM for the PostgreSQL (+ PostGIS) as well.

The rest of the blog discusses how to setup Nominatim and the tips and tricks that we learned along the way and how it compares with the geocoding solution offered by Google.

Setting up Nominatim

We started off by looking for Amazon Machine Images with Nominatim setup and could only find one which was hosted by OpenStreetMap but the magnet link was dead.

Next, we went through the official installation document. We decided to give docker a shot and found that there are many Nominatim docker builds. We used https://github.com/merlinnot/nominatim-docker since it seemed to follow all the steps mentioned in the official installation guide.

Issues faced during Setup

Out of Memory Errors

The official documentation recommends using 32GB of RAM for initial import but we needed to double the memory to 64GB to make it work.

Also any time docker build failed, due to the large amount of data that is generated on each run, we also ran out of disk space on subsequent docker builds since docker caches layers across builds.

Merging Multiple Regions

We wanted to geocode locations from USA, Mexico, Canada and Sri Lanka. USA, Mexico and Canada are included by default in North America data extract but we had to merge data for Sri Lanka with North America to get it in a format required for initial import.

The following snippet pre-processes map data for North America and Sri Lanka into a single data.osm.pbf file that can be directly used by Nominatim installer.

RUN curl -L 'http://download.geofabrik.de/north-america-latest.osm.pbf' \
    --create-dirs -o /srv/nominatim/src/north-america-latest.osm.pbf
RUN curl -L 'http://download.geofabrik.de/asia/sri-lanka-latest.osm.pbf' \
    --create-dirs -o /srv/nominatim/src/sri-lanka-latest.osm.pbf

RUN osmconvert /srv/nominatim/src/north-america-latest.osm.pbf \
    -o=/srv/nominatim/src/north-america-latest.o5m
RUN osmconvert /srv/nominatim/src/sri-lanka-latest.osm.pbf \
    -o=/srv/nominatim/src/sri-lanka-latest.o5m

RUN osmconvert /srv/nominatim/src/north-america-latest.o5m \
    /srv/nominatim/src/sri-lanka-latest.o5m \
    -o=/srv/nominatim/src/data.o5m

RUN osmconvert /srv/nominatim/src/data.o5m \
    -o=/srv/nominatim/src/data.osm.pbf

Slow Search times

Once the installation was done, we tried running simple location searches like this one, but the search timed out. Usually Nominatim can provide a lot of information from its web-interface by appending &debug=true to the search query.

# from
https://nominatim.openstreetmap.org/search.php?q=New+York&polygon_geojson=1&viewbox=
# to
https://nominatim.openstreetmap.org/search.php?q=New+York&polygon_geojson=1&viewbox=&debug=true

We created an issue in Nominatim repository and got very prompt replies from Nominatim maintainers, especially from Sarah Hoffman .

# runs analyze on the entire nominatim database
psql -d nominatim -c 'ANALYZE VERBOSE'

PostgreSQL query planner depends on statistics collected by postgres statistics collector while executing a query. In our case, query planner took an enormous amount of time to plan queries as there were no stats collected since we had a fresh installation.

Comparing Nominatim and Google Geocoder

We compared 2500 addresses and we found that Google geocoded 99% of those addresses. In comparison Nominatim could only geocode 47% of the addresses.

It means we still need to geocode ~50% of addresses using Google geocoder. We found that we could increase geocoding efficiency by normalizing the addresses we had.

Address Normalization using libpostal

Libpostal is an address normalizer, which uses statistical natural-language processing to normalize addresses. Libpostal also has ruby bindings which made it quite easy to use it for our test purposes.

Once libpostal and its ruby-bindings were installed (installation is straightforward and steps are available in ruby-postal's github page), we gave libpostal + Nominatim a go.

require 'geocoder'
require 'ruby_postal/expand'
require 'ruby_postal/parser'

Geocoder.configure({lookup: :nominatim, nominatim: { host: "nominatim_host:port"}})

full_address = [... address for normalization ...]
expanded_addresses = Postal::Expand.expand_address(full_address)
parsed_addresses = expanded_addresses.map do |address|
  Postal::Parser.parse_address(address)
end

parsed_addresses.each do | address |
  parsed_address = [:house_number, :road, :city, :state, :postcode, :country].inject([]) do |acc, key|
    # address is of format
    # [{label: 'postcode', value: 12345}, {label: 'city', value: 'NY'} .. ]
    key_value = address.detect { |address| address[:label] == key }
    if key_value
        acc << "#{key_value_pair[:value]}".titleize
    end
    acc
  end

  coordinates = Geocoder.coordinates(parsed_address.join(", "))
  if (coordinates.is_a? Array) && coordinates.present?
    puts "By Libpostal #{coordinates} => #{parsed_address.join(", ")}"
    break
  end
end

With this, we were able to improve our geocoding efficiency by 10% as Nominatim + Libpostal combination could geocode ~ 59% of addresses.

Similarly, write_timeout attribute and write_timeout= method is added to Net::BufferedIO class.

If any chunk of response is not written within number of seconds provided to write_timeout, Net::WriteTimeout exception is raised. Net::WriteTimeout exception is not raised on Windows systems.

Example

# server.rb

require 'socket'

server = TCPServer.new('localhost', 2345)
loop do
socket = server.accept
end

Ruby 2.5.1

# client.rb

require 'net/http'

connection = Net::HTTP.new('localhost', 2345)
connection.open_timeout = 1
connection.read_timeout = 3
connection.start

post = Net::HTTP::Post.new('/')
body = (('a' _ 1023) + "\n") _ 5_000
post.body = body

puts "Sending #{body.bytesize} bytes"
connection.request(post)

Output

\$ RBENV_VERSION=2.5.1 ruby client.rb

Sending 5120000 bytes

Ruby 2.5.1 processes request endlessly unless above program is interrupted.

Ruby 2.6.0-dev

Add write_timeout attribute to Net::HTTP instance in client.rb program.

# client.rb

require 'net/http'

connection = Net::HTTP.new('localhost', 2345)
connection.open_timeout = 1
connection.read_timeout = 3

# set write_timeout to 10 seconds

connection.write_timeout = 10

connection.start

post = Net::HTTP::Post.new('/')
body = (('a' _ 1023) + "\n") _ 5_000
post.body = body

puts "Sending #{body.bytesize} bytes"
connection.request(post)

Output

\$ RBENV_VERSION=2.6.0-dev ruby client.rb

Sending 5120000 bytes
Traceback (most recent call last):
13: `from client.rb:17:in `<main>`` 12: `from /net/http.rb:1479:in `request``
11: `from /net/http.rb:1506:in `transport_request`` 10: `from /net/http.rb:1506:in `catch``
9: `from /net/http.rb:1507:in `block in transport_request`` 8: `from /net/http/generic_request.rb:123:in `exec``
7: `from /net/http/generic_request.rb:189:in `send_request_with_body`` 6: `from /net/protocol.rb:221:in `write``
5: `from /net/protocol.rb:239:in `writing`` 4: `from /net/protocol.rb:222:in `block in write``
3: `from /net/protocol.rb:249:in `write0`` 2: `from /net/protocol.rb:249:in `each_with_index``
1: `from /net/protocol.rb:249:in `each`` `/net/protocol.rb:270:in `block in write0`: Net::WriteTimeout (Net::WriteTimeout)`

In Ruby 2.6.0, above program is terminated raising Net::WriteTimeout exception after 10 seconds (value set to write_timeout attribute).

Here is relevant commit and discussion for this change.

In Ruby 2.6, same methods are added as instance methods on Dir class. Dir#children (Link is not available) returns array of all the filenames except . and .. in the directory. Dir#each_child (Link is not available) yields all the filenames and operates on it.

Let's have a look at examples to understand it better.

Dir#children

directory = Dir.new('/Users/tejaswinichile/workspace')

directory.children
=> ["panda.png", "apple.png", "banana.png", "camera.jpg"]

Dir#each_child iterates and calls block for each file entry in the given directory. It uses filename as a parameter to the block.

Dir#each_child

directory = Dir.new('/Users/tejaswinichile/workspace')

directory.each_child { |filename| puts "Currently reading: #{filename}"}

Currently reading: panda.png
Currently reading: apple.png
Currently reading: banana.png
Currently reading: camera.jpg
=> #<Dir:/Users/tejaswinichile/Desktop>

If we don't pass any block to each_child, it returns enumerator instead.

directory = Dir.new('/Users/tejaswinichile/workspace')

directory.each_child

=> #<Enumerator: #<Dir:/Users/tejaswinichile/Desktop>:each_child>

Here is relevant commit and discussion for this change.

> > "1one".to_i
> > => 1

> > Integer("1one")
> > ArgumentError: invalid value for Integer(): "1one"

    from (irb):2:in `Integer'
    from (irb):2
    from /Users/prathamesh/.rbenv/versions/2.4.0/bin/irb:11:in `<main>'

> >

The to_i method tries to convert the given input to integer as much as possible whereas the Integer method throws an ArgumentError if it can't covert the input to integer. The Integer and Float methods parse more strictly compared to to_i and to_f respectively.

Some times, we might need the strictness of Integer and Float but ability to not raise an exception every time the input can't be parsed.

Before Ruby 2.6 it was possible to achieve it in following way.

> > Integer("msg") rescue nil
> >

In Ruby 2.6, the Integer and Float methods accept a keyword argument exception which can be either true or false. If it is false then no exception is raised if the input can't be parsed and nil is returned.

> > Float("foo", exception: false)
> > => nil
> > Integer("foo", exception: false)
> > => nil
> >

This is also faster than rescuing the exception and returning nil.

> > Benchmark.ips do |x|
> > ?> x.report("rescue") {
> > ?> Integer('foo') rescue nil
> > }
> > x.report("kwarg") {
> > ?> Integer('foo', exception: false)
> > }
> > x.compare!
> > end
> > Warming up --------------------------------------

              rescue    41.896k i/100ms
               kwarg    81.459k i/100ms

Calculating -------------------------------------
rescue 488.006k (± 4.5%) i/s - 2.472M in 5.076848s
kwarg 1.024M (±11.8%) i/s - 5.050M in 5.024937s

Comparison:
kwarg: 1023555.3 i/s
rescue: 488006.0 i/s - 2.10x slower

As we can see, rescuing the exception is twice slower than using the new keyword argument. We can still use the older technique if we want to return a different value from nil.

> > Integer('foo') rescue 42
> > => 42
> >

By default, the keyword argument exception is set to true for backward compatibility.

The Chinese version of this blog is available here.

binding.eval('[__FILE__, __LINE__]')
=> ["/Users/taha/blog/app/controllers/application_controller", 2]

Ruby 2.6 adds more readable method Binding#source_location (Link is not available) to achieve similar result.

binding.source_location
=> ["/Users/taha/blog/app/controllers/application_controller", 2]

Here is relevant commit and discussion for this change.

The Chinese version of this blog is available here.

In Ruby 2.6, a block can be passed to String#split (Link is not available) which yields each split string and operates on it. This avoids creating an array and thus is memory efficient.

We will add method is_fruit? to understand how to use split with a block.

def is_fruit?(value)
%w(apple mango banana watermelon grapes guava lychee).include?(value)
end

Input is a comma separated string with vegetables and fruits names. Goal is to fetch names of fruits from input string and store it in an array.

String#split

input_str = "apple, mango, potato, banana, cabbage, watermelon, grapes"

splitted_values = input_str.split(", ")
=> ["apple", "mango", "potato", "banana", "cabbage", "watermelon", "grapes"]

fruits = splitted_values.select { |value| is_fruit?(value) }
=> ["apple", "mango", "banana", "watermelon", "grapes"]

Using split an intermediate array is created which contains both fruits and vegetables names.

String#split with a block

fruits = []

input_str = "apple, mango, potato, banana, cabbage, watermelon, grapes"

input_str.split(", ") { |value| fruits << value if is_fruit?(value) }
=> "apple, mango, potato, banana, cabbage, watermelon, grapes"

fruits
=> ["apple", "mango", "banana", "watermelon", "grapes"]

When a block is passed to split, it returns the string on which split was called and does not create an array. String#split yields block on each split string, which in our case was to push fruit names in a separate array.

Update

Benchmark

We created a large random string to benchmark performance of split and split with block

require 'securerandom'

test_string = ''

100_000.times.each do
test_string += SecureRandom.alphanumeric(10)
test_string += ' '
end

require 'benchmark'

Benchmark.bmbm do |bench|

bench.report('split') do
arr = test_string.split(' ')
str_starts_with_a = arr.select { |str| str.start_with?('a') }
end

bench.report('split with block') do
str_starts_with_a = []
test_string.split(' ') { |str| str_starts_with_a << str if str.start_with?('a') }
end

end

Results

Rehearsal ----------------------------------------------------
split              0.023764   0.000911   0.024675 (  0.024686)
split with block   0.012892   0.000553   0.013445 (  0.013486)
------------------------------------------- total: 0.038120sec

                       user     system      total        real
split              0.024107   0.000487   0.024594 (  0.024622)
split with block   0.010613   0.000334   0.010947 (  0.010991)

We did another iteration of benchmarking using benchmark/ips.

require 'benchmark/ips'
Benchmark.ips do |bench|

bench.report('split') do
splitted_arr = test_string.split(' ')
str_starts_with_a = splitted_arr.select { |str| str.start_with?('a') }
end

bench.report('split with block') do
str_starts_with_a = []
test_string.split(' ') { |str| str_starts_with_a << str if str.start_with?('a') }
end

bench.compare!
end

Results

Warming up --------------------------------------
               split     4.000  i/100ms
    split with block    10.000  i/100ms
Calculating -------------------------------------
               split     46.906  (± 2.1%) i/s -    236.000  in   5.033343s
    split with block    107.301  (± 1.9%) i/s -    540.000  in   5.033614s

Comparison:
    split with block:      107.3 i/s
               split:       46.9 i/s - 2.29x  slower

This benchmark shows that split with block is about 2 times faster than split.

Here is relevant commit and discussion for this change.

The Chinese version of this blog is available here.

If we use else without rescue inside begin..end block in Ruby 2.5, it gives a warning.

irb(main):001:0> begin
irb(main):002:1> puts "Inside begin block"
irb(main):003:1> else
irb(main):004:1> puts "Inside else block"
irb(main):005:1> end
(irb):5: warning: else without rescue is useless

This warning is present as code inside else block will never get executed

Ruby 2.6

In Ruby 2.6 it will raise an exception if we use else without rescue in begin..end block. This commit changed warning into exception in Ruby 2.6. Changes made in the commit are experimental.

irb(main):001:0> begin
irb(main):002:1> puts "Inside begin block"
irb(main):003:1> else
irb(main):004:1> puts "Inside else block"
irb(main):005:1> end
Traceback (most recent call last):
1: from /usr/local/bin/irb:11:in `<main>'
SyntaxError ((irb):3: else without rescue is useless)

The Chinese version of this blog is available here.

Ruby 2.5.0

irb> (1..Float::INFINITY).each do |n|
irb\* # logic goes here
irb> end

OR

irb> 1.step.each do |n|
irb\* # logic goes here
irb> end

Ruby 2.6.0

Ruby 2.6 makes infinite loop more readable by changing mandatory second argument in range to optional. Internally, Ruby changes second argument to nil if second argument is not provided. So, both (0..) and (0..nil) are same in Ruby 2.6.

Using endless loop in Ruby 2.6

irb> (0..).each do |n|
irb\* # logic goes here
irb> end

irb> (0..nil).size
=> Infinity
irb> (0..).size
=> Infinity

In Ruby 2.5, nil is not an acceptable argument and (0..nil) would throw ArgumentError.

irb> (0..nil)
ArgumentError (bad value for range)

Here is the relevant commit and discussion for this change.

The Chinese version of this blog is available here.

Using Concurrent::Future

In one of our applications, to improve performance we added threaded code using Concurrent::Future. It worked really well for us until one day it stopped working.

"Why threads?" one might ask. The code in question was a textbook threading use case. It had a few API calls, some DB requests and finally an action that was performed on all the data that was aggregated.

Let us look at what this code looks like.

Non threaded code

selected_shipping_companies.each do | carrier |
  # api calls
  distance_in_miles = find_distance_from_origin_to_destination
  historical_average_rate = historical_average_for_this_particular_carrier

  # action performed
  build_price_details_for_this_carrier(distance_in_miles,
                                       historical_average_rate)
end

Converting the above code to use Concurrent::Future is trivial.

futures = selected_shipping_companies.map do |carrier|
  Concurrent::Future.execute do
    # api calls
    distance_in_miles = find_distance_from_origin_to_destination
    historical_average_rate = historical_average_for_this_particular_carrier

    # action performed
    build_price_details_for_this_carrier(distance_in_miles,
                                         historical_average_rate)
  end
end

futures.map(&:value)

A bit more about Concurrent::Future

It is often intimidating to work with threads. They can bring in complexity and can have unpredictable behaviors due to lack of thread-safety. Ruby, being a language of mutable references, we often find it difficult to write 100% thread-safe code.

Inspired by Clojure's Future function, Concurrent::Future is a primitive that guarantees thread safety. It takes a block of work and performs the work asynchronously using Concurrent Ruby's global thread-pool. Once a block of work is scheduled, Concurrent Ruby gives us a handle to this future work, on which when #value (or #deref) is called block's value is returned.

The Bug

Usually, when an exception occurs in the main thread, the interpreter stops and gathers the exception data. In the case of Ruby Threads, any unhandled exceptions are reported only when Thread#join is called. Setting Thread#abort_on_exception to true, is an better alternative which will cause all threads to exit when an exception is raised in any running thread. We published a blog recently which talks about this in great detail.

Exception handling in Concurrent Ruby

future = Concurrent::Future.execute {
            raise StandardError.new("Boom!")
          }

sleep(0.1) # giving arbitrary time for future to execute

future.value     #=> nil

Where did the exception go? This code fails silently and swallows the exceptions. How can we find out if the code executed successfully?

future = Concurrent::Future.execute {
              raise StandardError.new("Boom!")
          }

sleep(0.1) # giving arbitrary time for future to execute

future.value     #=> nil

future.rejected? #=> true
future.reason    #=> "#<StandardError: Boom!>"

How we fixed our issue

We found places in our application where Concurrent::Future was used in a way that would swallow exceptions. It is also a possibility that people might overlook the explicit need to manually report exception. We addressed these concerns with the following wrapper class.

module ConcurrentExecutor
  class Error < StandardError
    def initialize(exceptions)
      @exceptions = exceptions
      super
    end

    def message
      @exceptions.map { | e | e.message }.join "\n"
    end

    def backtrace
      traces = @exceptions.map { |e| e.backtrace }
      ["ConcurrentExecutor::Error START", traces, "END"].flatten
    end
  end

  class Future
    def initialize(pool: nil)
      @pool = pool || Concurrent::FixedThreadPool.new(20)
      @exceptions = Concurrent::Array.new
    end

    # Sample Usage
    # executor = ConcurrentExecutor::Future.new(pool: pool)
    # executor.execute(carriers) do | carrier |
    #   ...
    # end
    #
    # values = executor.resolve

    def execute array, &block
      @futures = array.map do | element |
        Concurrent::Future.execute({ executor: @pool }) do
          yield(element)
        end.rescue do | exception |
          @exceptions << exception
        end
      end

      self
    end

    def resolve
      values = @futures.map(&:value)

      if @exceptions.length > 0
        raise ConcurrentExecutor::Error.new(@exceptions)
      end

      values
    end
  end
end

Please note that using Concurrent Ruby Futures caused segmentation fault while running specs in Circle CI. As of this writing, we are using normal looping instead of Futures in Circle CI until the reason for the segfault is isolated and fixed.

Update

Concurrent::Future also gives us another API which not only returns the value of the block but also posts/raises any exceptions that occur into the main thread.

thread_pool = Concurrent::FixedThreadPool.new(20)
executors = [1, 2, 3, 4].map do |random_number|
  Concurrent::Future.execute({ executor: thread_pool }) do
    random_number / (random_number.even? ? 0 : 1)
  end
end

executors.map(&:value)
=> [1, nil, 3, nil]

executors.map(&:value!)

> ZeroDivisionError: divided by 0
> from (pry):4:in `/'

We thank Jonathan Rochkind for pointing us to this undocumented api in his reddit post.

In Ruby 2.4 when we pass a block to a method, which further passes to another method, Ruby creates a new Proc object by the given block before passing this proc to the another method.

This creates unnecessary objects even when the block parameter is not accessed. It also creates a chain of Proc objects when the block parameter is passed through various methods.

Proc creation is one a heavyweight operation because we need to store all local variables (represented by Env objects in MRI internal) in the heap.

Ruby 2.5 introduced a lazy proc allocation. Ruby 2.5 will not create a Proc object when passing a block to another method. Instead, it will pass the block information. If the block is accessed somewhere else, then it creates a Proc object by the given block.

This results in lesser memory allocation and faster execution.

Ruby 2.4

irb> require 'benchmark'
  => true
irb> def greet
irb>   yield
irb> end
  => :greet
irb>
irb> def greet_with_welcome(&block)
irb>   puts 'Welcome'
irb>   greet(&block)
irb> end
  => :greet_with_welcome
irb>
irb> Benchmark.measure { 1000.times { greet_with_welcome { 'BigBinary' } } }
Welcome
Welcome
...
...
...
  => #<Benchmark::Tms:0x007fe6ab929de0 @label="", @real=0.022295999999187188, @cstime=0.0, @cutime=0.0, @stime=0.01, @utime=0.0, @total=0.01>

Ruby 2.5

irb> require 'benchmark'
  => true
irb> def greet
irb>   yield
irb> end
  => :greet
irb>
irb> def greet_with_welcome(&block)
irb>   puts 'Welcome'
irb>   greet(&block)
irb> end
  => :greet_with_welcome
irb>
  irb> Benchmark.measure { 1000.times { greet_with_welcome { 'BigBinary' } } }
Welcome
Welcome
...
...
...
  => #<Benchmark::Tms:0x00007fa4400871b8 @label="", @real=0.004612999997334555, @cstime=0.0, @cutime=0.0, @stime=0.001524000000000001, @utime=0.0030690000000000023, @total=0.004593000000000003>

As we can see, there is considerable improvement in execution time when a block param is passed in Ruby 2.5.

Here is the relevant commit and discussion.

Let's suppose we need to run migrations using rake db:migrate in a Rails project setup script. We can use the Kernel#system method.

irb> system('rake db:migrate')

Ruby 2.5.0

Executing system returns true or false. Another feature of system is that it eats up the exceptions.

Let's suppose our migrations can run successfully. In this case the system command for running migrations will return true.

irb> system('rake db:migrate')
 => true

Let's suppose we have a migration that is trying to add a column to a table which does not exist. In this case, the system command for running migrations will return false.

irb> system('rake db:migrate')
== 20180311211836 AddFirstNameToAdmins: migrating =============================
-- add_column(:admins, :first_name, :string)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::UndefinedTable: ERROR:  relation "admins" does not exist
: ALTER TABLE "admins" ADD "first_name" character varying
.
.
.
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
 => false

As we can see, even when there is a failure in executing system commands, the return value is false. Ruby does not raise an exception in those cases.

However, we can use raise explicitly to raise an exception and halt the setup script execution.

irb> system('rake db:migrate') || raise('Failed to run migrations')
== 20180311211836 AddFirstNameToAdmins: migrating =============================
-- add_column(:admins, :first_name, :string)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::UndefinedTable: ERROR:  relation "admins" does not exist
: ALTER TABLE "admins" ADD "first_name" character varying
.
.
.
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
Traceback (most recent call last):
        2: from /Users/amit/.rvm/rubies/ruby-2.5.0/bin/irb:11:in `<main>'
        1: from (irb):4
RuntimeError (Failed to run migrations)

Ruby 2.6.0-preview1

Ruby 2.6 make our lives easier by providing an option exception: true so that we do not need to use raise explicitly to halt script execution.

irb> system('rake db:migrate', exception: true)
== 20180311211836 AddFirstNameToAdmins: migrating =============================
-- add_column(:admins, :first_name, :string)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::UndefinedTable: ERROR:  relation "admins" does not exist
: ALTER TABLE "admins" ADD "first_name" character varying
.
.
.
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
Traceback (most recent call last):
        3: from /Users/amit/.rvm/rubies/ruby-2.6.0-preview1/bin/irb:11:in `<main>'
        2: from (irb):2
        1: from (irb):2:in `system'
RuntimeError (Command failed with exit 1: rake)

Ruby 2.6 works the same way as previous Ruby versions when used without the exception option or used with exception set as false.

irb> system('rake db:migrate', exception: false)
== 20180311211836 AddFirstNameToAdmins: migrating =============================
-- add_column(:admins, :first_name, :string)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::UndefinedTable: ERROR:  relation "admins" does not exist
: ALTER TABLE "admins" ADD "first_name" character varying
.
.
.
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
 => false

Here is the relevant commit and discussion for this change.

system is not the only way to execute scripts like these. We wrote a blog 6 years ago which discusses the differences between running commands using backtick, exec, sh, popen3, popen2e and Process.spawn.

The Chinese version of this blog is available here.

division_thread = Thread.new do
  puts "Calculating 4/0 in division_thread"
  puts "Result is: #{4/0}"
  puts "Exiting from division_thread"
end

sleep 1

puts "In the main thread"

Execution of it looks like this.

$ RBENV_VERSION=2.4.0 ruby thread_example_1.rb

Calculating 4/0 in division_thread

In the main thread

Note that the last two lines from the block were not printed. Also notice that after failing in the thread the program continued to run in main thread. That's why we got the message "In the main thread".

This is because the default behavior of Ruby is to silently ignore exceptions in threads and then to continue to execute in the main thread.

Enabling abort_on_exception to stop on failure

If we want an exception in a thread to stop further processing both in the thread and in the main thread then we can enable Thread[.#]abort_on_exception on that thread to achieve that.

Notice that in the below code we are using Thread.current.

division_thread = Thread.new do
  Thread.current.abort_on_exception = true

  puts "Calculating 4/0 in division_thread"
  puts "Result is: #{4/0}"
  puts "Exiting from division_thread"
end

sleep 1

puts "In the main thread"

$ RBENV_VERSION=2.4.0 ruby thread_example_2.rb

Calculating 4/0 in division_thread

thread_example_2.rb:5:in `/': divided by 0 (ZeroDivisionError)
  from thread_example_2.rb:5:in `block in <main>'

As we can see once an exception was encountered in the thread then processing stopped on both in the thread and in the main thread.

Note that Thread.current.abort_on_exception = true activates this behavior only for the current thread.

If we want this behavior globally for all the threads then we need to use Thread.abort_on_exception = true.

Running program with debug flag to stop on failure

Let's run the original code with --debug option.

$ RBENV_VERSION=2.4.0 ruby --debug thread_example_1.rb

thread_example_1.rb:1: warning: assigned but unused variable - division_thread

Calculating 4/0 in division_thread

Exception `ZeroDivisionError' at thread_example_1.rb:3 - divided by 0
Exception `ZeroDivisionError' at thread_example_1.rb:7 - divided by 0
thread_example_1.rb:3:in `/': divided by 0 (ZeroDivisionError)
  from thread_example_1.rb:3:in `block in <main>'

In this case the exception is printed in detail and the code in main thread was not executed.

Usually when we execute a program with --debug option then the behavior of the program does not change. We expect the program to print more stuff but we do not expect behavior to change. However in this case the --debug option changes the behavior of the program.

Running program with join on thread to stop on failure

If a thread raises an exception and abort_on_exception and $DEBUG flags are not set then that exception will be processed at the time of joining of the thread.

division_thread = Thread.new do
  puts "Calculating 4/0 in division_thread"
  puts "Result is: #{4/0}"
  puts "Exiting from division_thread"
end

division_thread.join

puts "In the main thread"

$ RBENV_VERSION=2.4.0 ruby thread_example_3.rb

Calculating 4/0 in division_thread

thread_example_3.rb:3:in `/': divided by 0 (ZeroDivisionError)
  from thread_example_3.rb:3:in `block in <main>'

Both Thread#join and Thread#value will stop processing in the thread and in the main thread once an exception is encountered.

Introduction of report_on_exception in Ruby 2.4

Almost 6 years ago, Charles Nutter (headius) had proposed that the exceptions raised in threads should be automatically logged and reported, by default. To make his point, he explained issues similar to what we discussed above about the Ruby's behavior of silently ignoring exceptions in threads. Here is the relevant discussion on his proposal.

Following are some of the notable points discussed.

• Enabling Thread[.#]abort_on_exception, by default, is not always a good idea.
• There should be a flag which, if enabled, would print the thread-killing exception info.
• In many cases, people spawn one-off threads which are not hard-referenced using Thread#join or Thread#value. Such threads gets garbage collected. Should it report the thread-killing exception at the time of garbage collection if such a flag is enabled?
• Should it warn using Warning#warn or redirect to STDERR device while reporting?

Charles Nutter suggested that a configurable global flag Thread.report_on_exception and instance-level flag Thread#report_on_exception should be implemented having its default value as true. When set to true, it should report print exception information.

Matz and other core members approved that Thread[.#]report_on_exception can be implemented having its default value set to false.

Charles Nutter, Benoit Daloze and other people demanded that it should be true by default so that programmers can be aware of the silently disappearing threads because of exceptions.

Shyouhei Urabe advised that due to some technical challenges, the default value should be set to false so as this feature could land in Ruby. Once this feature is in then the default value can be changed in a later release.

Nobuyoshi Nakada (nobu) pushed an implementation for Thread[.#]report_on_exception with a default value set to false. It was released in Ruby 2.4.0.

Let's try enabling report_on_exception globally using Thread.report_on_exception.

Thread.report_on_exception = true

division_thread = Thread.new do
  puts "Calculating 4/0 in division_thread"
  puts "Result is: #{4/0}"
  puts "Exiting from division_thread"
end

addition_thread = Thread.new do
  puts "Calculating nil+4 in addition_thread"
  puts "Result is: #{nil+4}"
  puts "Exiting from addition_thread"
end

sleep 1

puts "In the main thread"

$ RBENV_VERSION=2.4.0 ruby thread_example_4.rb

Calculating 4/0 in division_thread

#<Thread:0x007fb10f018200@thread_example_4.rb:3 run> terminated with exception:
thread_example_4.rb:5:in `/': divided by 0 (ZeroDivisionError)
  from thread_example_4.rb:5:in `block in <main>'

Calculating nil+4 in addition_thread

#<Thread:0x007fb10f01aca8@thread_example_4.rb:9 run> terminated with exception:
thread_example_4.rb:11:in `block in <main>': undefined method `+' for nil:NilClass (NoMethodError)

In the main thread

It now reports the exceptions in all threads. It prints that the Thread:0x007fb10f018200 was terminated with exception: divided by 0 (ZeroDivisionError). Similarly, another thread Thread:0x007fb10f01aca8 was terminated with exception: undefined method '+' for nil:NilClass (NoMethodError).

Instead of enabling it globally for all threads, we can enable it for a particular thread using instance-level Thread#report_on_exception.

division_thread = Thread.new do
  puts "Calculating 4/0 in division_thread"
  puts "Result is: #{4/0}"
  puts "Exiting from division_thread"
end

addition_thread = Thread.new do
  Thread.current.report_on_exception = true

  puts "Calculating nil+4 in addition_thread"
  puts "Result is: #{nil+4}"
  puts "Exiting from addition_thread"
end

sleep 1

puts "In the main thread"

In the above case we have enabled report_on_exception flag just for addition_thread.

Let's execute it.

$ RBENV_VERSION=2.4.0 ruby thread_example_5.rb

Calculating 4/0 in division_thread

Calculating nil+4 in addition_thread

#<Thread:0x007f8e6b007f70@thread_example_5.rb:7 run> terminated with exception:
thread_example_5.rb:11:in `block in <main>': undefined method `+' for nil:NilClass (NoMethodError)

In the main thread

Notice how it didn't report the exception which killed thread division_thread. As expected, it reported the exception that killed thread addition_thread.

With the above changes ruby reports the exception as soon as it encounters. However if these threads are joined then they will still raise exception.

division_thread = Thread.new do
  Thread.current.report_on_exception = true

  puts "Calculating 4/0 in division_thread"
  puts "Result is: #{4/0}"
  puts "Exiting from division_thread"
end

begin
  division_thread.join
rescue => exception
  puts "Explicitly caught - #{exception.class}: #{exception.message}"
end

puts "In the main thread"

$ RBENV_VERSION=2.4.0 ruby thread_example_6.rb

Calculating 4/0 in division_thread

#<Thread:0x007f969d00d828@thread_example_6.rb:1 run> terminated with exception:
thread_example_6.rb:5:in `/': divided by 0 (ZeroDivisionError)
  from thread_example_6.rb:5:in `block in <main>'

Explicitly caught - ZeroDivisionError: divided by 0

In the main thread

See how we were still be able to handle the exception raised in division_thread above after joining it despite it reported it before due to Thread#report_on_exception flag.

report_on_exception defaults to true in Ruby 2.5

Benoit Daloze (eregon) strongly advocated that both the Thread.report_on_exception and Thread#report_on_exception should have default value as true. Here is the relevant feature request.

After approval from Matz, Benoit Daloze pushed the implementation by fixing the failing tests and silencing the unnecessary verbose warnings.

It was released as part of Ruby 2.5.

Now in ruby 2.5 we can simply write like this.

division_thread = Thread.new do
  puts "Calculating 4/0 in division_thread"
  puts "Result is: #{4/0}"
  puts "Exiting from division_thread"
end

addition_thread = Thread.new do
  puts "Calculating nil+4 in addition_thread"
  puts "Result is: #{nil+4}"
  puts "Exiting from addition_thread"
end

sleep 1

puts "In the main thread"

Let's execute it with Ruby 2.5.

$ RBENV_VERSION=2.5.0 ruby thread_example_7.rb

Calculating 4/0 in division_thread

#<Thread:0x00007f827689a238@thread_example_7.rb:1 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
  1: from thread_example_7.rb:3:in `block in <main>'
thread_example_7.rb:3:in `/': divided by 0 (ZeroDivisionError)

Calculating nil+4 in addition_thread

#<Thread:0x00007f8276899b58@thread_example_7.rb:7 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
thread_example_7.rb:9:in `block in <main>': undefined method `+' for nil:NilClass (NoMethodError)

In the main thread

We can disable the thread exception reporting globally using Thread.report_on_exception = false or for a particular thread using Thread.current.report_on_exception = false.

Future Possibilities

In addition to this feature, Charles Nutter also suggested that it will be good if there exists a callback handler which can accept a block to be executed when a thread dies due to an exception. The callback handler can be at global level or it can be for a specific thread.

Thread.on_exception do
  # some stuff
end

In the absence of such handler libraries need to resort to custom code to handle exceptions. Here is how Sidekiq handles exceptions raised in threads.

Important thing to note is that report_on_exception does not change behavior of the code. It does more reporting when a thread dies and when it comes to thread dies more reporting is a good thing.