Let's take an example of sale at Amazon.

Amazon would like to remind users across different timezones about its upcoming sale by sending an email. This task of sending a reminder would be processed as a background job.

Before:

Before Rails 6, we had to pass timezone explicitly to the perform method of the job as shown below.

timezone = "Eastern Time (US & Canada)"

AmazonSaleJob.perform_later(Time.now, timezone)

class AmazonSaleJob < ApplicationJob
  queue_as :default

  def perform(time, timezone)

    time = time.in_time_zone(timezone)
    sale_start_time = localtime(2020, 12, 24)

    if time >= sale_start_time
      puts "Sale has started!"
      #Send an email stating Sale has started
    else
      sale_starts_in = (sale_start_time - time).div(3600)
      puts "Hang on! Sale will start in #{sale_starts_in} hours"
      #Send an email stating sales starts in sale_starts_in hours
     end
  end

  private

    def localtime(*args)
      Time.zone ? Time.zone.local(*args) : Time.utc(*args)
    end
end

After:

After the changes in Rails 6, passing timezone to Job is now taken care of by Rails.

timezone = "Eastern Time (US & Canada)"

Time.use_zone(timezone) do
  AmazonSaleJob.perform_later(Time.zone.now)
end

class AmazonSaleJob < ApplicationJob
  queue_as :default

  def perform(time)
    sale_start_time = localtime(2020, 12, 24)

    if time >= sale_start_time
      puts "Sale has started!"
      #Send an email stating Sale has started
    else
      sale_starts_in = (sale_start_time - time).div(3600)
      puts "Hang on! Sale will start in #{sale_starts_in} hours"
      #Send an email stating sales starts in sale_starts_in hours
     end
   end

  private

    def localtime(*args)
      Time.zone ? Time.zone.local(*args) : Time.utc(*args)
    end
end

Rails 6 also propagates timezone to all the subsequent nested jobs.

Array#including and Enumerable#including

including can be used to extend a collection in a more object oriented way. It does not mutate the original collection but returns a new collection which is the concatenation of the given collections.

# multiple arguments can be passed to including

> > [1, 2, 3].including(4, 5)
> > => [1, 2, 3, 4, 5]

# another enumerable can also be passed to including

> > [1, 2, 3].including([4, 5])
> > => [1, 2, 3, 4, 5]

> > %i(apple orange).including(:banana)
> > => [:apple, :orange, :banana]

# return customers whose country_code is IN along with the prime customers

> > Customer.where(country_code: "IN").including(Customer.where(prime: true))

Array#excluding and Enumerable#excluding

It returns a copy of the enumerable excluding the given collection.

# multiple arguments can be passed to including

> > [11, 22, 33, 44].excluding([22, 33])
> > => [11, 44]

> > %i(ant bat cat).excluding(:bat)
> > => [:ant, :cat]

# return all prime customers except those who haven't added their phone

> > Customer.where(prime: true).excluding(Customer.where(phone: nil))

Array#excluding and Enumerable#excluding replaces the already existing method without which in Rails 6 is now aliased to excluding.

> > [11, 22, 33, 44].without([22, 33])
> > => [11, 44]

excluding and including helps to shrink or extend a collection without using any operator.

Check out the commit for more details on this.

Let's checkout the bug in Rails 5.2 and the fix in Rails 6.

Rails 5.2

Let's define an after_commit callback in User model and try updating an invalid user object in a transaction block.

>> class User < ApplicationRecord
>>   validates :name, :email, presence: true
>>
>>   after_commit :show_success_message
>>
>>   private
>>
>>     def show_success_message
>>       p 'User has been successfully saved into the database.'
>>     end
>> end

=> :show_success_message

>> user = User.create(name: 'Jon Snow', email: 'jon@bigbinary.com')
begin transaction
User Create (0.8ms)  INSERT INTO "users" ("name", "email", "created_at", "updated_at") VALUES (?, ?, ?, ?)  [["name", "Jon Snow"], ["email", "jon@bigbinary.com"], ["created_at", "2019-07-14 15:35:33.517694"], ["updated_at", "2019-07-14 15:35:33.517694"]]
commit transaction
"User has been successfully saved into the database."

=> #<User id: 1, name: "Jon Snow", email: "jon@bigbinary.com", created_at: "2019-07-14 15:35:33", updated_at: "2019-07-14 15:35:33">

>> User.transaction do
>>   user.email = nil
>>   p user.valid?
>>   user.save
>> end
begin transaction
false
commit transaction
"User has been successfully saved into the database."

=> false

As we can see here, that that the after_commit callback show_success_message was called even if object was never saved in the transaction.

Rails 6.0.0.rc1

Now, let's try the same thing in Rails 6.

>> class User < ApplicationRecord
>>   validates :name, :email, presence: true
>>
>>   after_commit :show_success_message
>>
>>   private
>>
>>     def show_success_message
>>       p 'User has been successfully saved into the database.'
>>     end
>> end

=> :show_success_message

>> user = User.create(name: 'Jon Snow', email: 'jon@bigbinary.com')
SELECT sqlite_version(*)
begin transaction
User Create (1.0ms)  INSERT INTO "users" ("name", "email", "created_at", "updated_at") VALUES (?, ?, ?, ?)  [["name", "Jon Snow"], ["email", "jon@bigbinary.com"], ["created_at", "2019-07-14 15:40:54.022045"], ["updated_at", "2019-07-14 15:40:54.022045"]]
commit transaction
"User has been successfully saved into the database."

=> #<User id: 1, name: "Jon Snow", email: "jon@bigbinary.com", created_at: "2019-07-14 15:40:54", updated_at: "2019-07-14 15:40:54">

>> User.transaction do
>>   user.email = nil
>>   p user.valid?
>>   user.save
>>   end
false

=> false

Now, we can see that after_commit callback was never called if the object was not saved.

Here is the relevant issue and the pull request.

Before Rails 6, we had to run the following tasks to set up the database.

# create the database
rails db:create

# run the migrations
rails db:migrate

# prepopulate the database with initial/default data
rails db:seed

Rails 6

Rails 6, adds rails db:prepare to get rid of running all the above tasks individually. rails db:prepare first calls the migrate to run the migrations, but if the database doesn't exist, migrate throws an ActiveRecord::NoDatabaseError. Once it is catched, it performs the following operations:

• Creates the database.
• Loads the schema.
• Seeds the database.

Thus, rails db:prepare saves a lot of time spent on running database tasks individually while setting up an application and finishes it with just one command.

Here is the relevant pull request.

What is DNS Rebinding attack?

DNS Rebinding can be summarized as follows.

• An unsuspecting victim is tricked into loading rebinding.network which is resolved by a DNS server controlled by a malicious entity.
• Victims web browser sends a DNS query and gets the real IP address, say 24.56.78.99 of http://rebinding.network. This DNS server also sets a very short TTL value ( say 1 second ) on the response so that the client won't cache this response for long.
• The script on this webpage cannot attack services running in local network due to CORS restrictions imposed by victims web browser. Instead it starts sending a suspicious POST request to http://rebinding.network/setup/reboot with a JSON payload {params: factory-reset}.
• First few requests are indeed sent to 24.56.78.99 (real IP address), with the DNS info from the cache, but then the browser sends out a DNS query for rebinding.network when it observes that the cache has gone stale.
• When the malicious DNS server gets the request for a second time, instead of responding with 24.56.78.99 (which is the real IP address of rebinding.network), it responds with 192.168.1.90, an address at which, a poorly secured smart device runs.

Using this exploit, an attacker is able to factory-reset a device which relied on security provided by local network.

This attack is explained in much more detail in this blog post.

How does it affect Rails?

Rails’s web console was particularly vulnerable to a Remote Code Execution (RCE) via a DNS Rebinding.

In this blog post, Ben Murphy goes into technical details of exploiting this vulnerability to open Calculator app (only works in OS X).

How does Rails 6 mitigate DNS Rebinding?

Rails mitigates DNS Rebinding attack by maintaining a whitelist of domains from which it can receive requests. This is achieved with a new HostAuthorization middleware. This middleware leverages the fact that HOST request header is a forbidden header.

# taken from Rails documentation

# Allow requests from subdomains like `www.product.com` and
# `beta1.product.com`.
Rails.application.config.hosts << ".*\.product\.com/"

In the above example, Rails would render a blocked host template, if it receives requests from domains outside of above whitelist.

In development environment, default whitelist includes 0.0.0.0/0, ::0 (CIDR notations for IPv4 and IPv6 default routes) and localhost. For all other environments, config.hosts is empty and host header checks are not done.

>> blob = ActiveStorage::Blob.first
=> <ActiveStorage::Blob id: 1, key: "6qXeoibkvohP4VJiU4ytaEkH", filename: "Screenshot 2019-08-26 at 10.24.40 AM.png", ..., created_at: "2019-08-26 09:57:30">

>> blob.open do |tempfile|
>>   puts tempfile.path  #do some processing
>> end
# Output: /var/folders/67/3n96myxs1rn5q_c47z7dthj80000gn/T/ActiveStorage-1-20190826-73742-mve41j.png

Processing a blob

Let's take an example of a face detection application where the user images are uploaded. Let's assume that the images are uploaded on S3.

Before Rails 6, we will have to download the image in system's memory, process it with an image processing program and then send the processed image back to the S3 bucket.

The overhead

If the processing operation is successful, the original file can be deleted from the system. We need to take care of a lot of uncertain events from the download phase till the phase when the processed image is created.

ActiveStorage::Blob#open to the rescue

ActiveStorage::Blob#open, abstracts away all this complications and gives us a tempfile which is closed and unlinked once the block is executed.

 1. open takes care of handling all the fanfare of getting a blob object to a tempfile.  2. open takes care of the tempfile cleanup after the block.

> > blob = ActiveStorage::Blob.first
> > blob.open do |tempfile|
> > tempfile #do some processing
> > end

# once the given block is executed

# the tempfile is closed and unlinked

=> #<Tempfile: (closed)>

By default, tempfiles are created in Dir.tmpdir directory, but ActiveStorage::Blob#open also takes an optional argument tmpdir to set a custom directory for storing the tempfiles.

> > Dir.tmpdir
> > => "/var/folders/67/3n96myxs1rn5q_c47z7dthj80000gn/T"

> > blob = ActiveStorage::Blob.first
> > blob.open(tmpdir: "/desired/path/to/save") do |tempfile|
> > puts tempfile.path #do some processing
> > end
> >

Here is the relevant commit.

Before Rails 6, we had to join name and email address using string interpolation as mentioned in Rails 5.2 Guides and shown below.

  email_with_name = %("John Smith" <john@example.com>)
  mail(
    to: email_with_name,
    subject: 'Hey Rails 5.2!'
  )

Problem with string interpolation is it doesn't escape unexpected special characters like quotes(") in the name.

Here's an example.

Rails 5.2

irb(main):001:0> %("John P Smith" <john@example.com>)
=> "\"John P Smith\" <john@example.com>"

irb(main):002:0> %('John "P" Smith' <john@example.com>)
=> "'John \"P\" Smith' <john@example.com>"

Rails 6 adds ActionMailer::Base#email_address_with_name to join name and email address in the format "John Smith" <john@example.com> and take care of escaping special characters.

Rails 6.1.0.alpha

irb(main):001:0> ActionMailer::Base.email_address_with_name("john@example.com", "John P Smith")
=> "John P Smith <john@example.com>"

irb(main):002:0> ActionMailer::Base.email_address_with_name("john@example.com", 'John "P" Smith')
=> "\"John \\\"P\\\" Smith\" <john@example.com>"

mail(
to: email_address_with_name("john@example.com", "John Smith"),
subject: 'Hey Rails 6!'
)

Here's the relevant pull request for this change.

Let's take for an example that we want product :name to be as the default resource identifier instead of :id while defining routes for products. In this case, :param option comes handy. We will see below how we can use this option.

Before Rails 6, if resource custom param contains a colon, Rails used to consider that as an extra param which should not be the case because it sneaks in an extra param.

An issue was raised in Aug, 2017 which was later fixed in February this year.

So, now Rails 6 raises ArgumentError if a resource custom param contains a colon(:).

Let's checkout how it works.

Rails 5.2

Let's create routes for products with custom param as name/:pzn.

> > Rails.application.routes.draw do
> > resources :products, param: 'name/:pzn'
> > end
> >

\$ rake routes | grep products
products GET /products(.:format) products#index
POST /products(.:format) products#create
new_product GET /products/new(.:format) products#new
edit_product GET /products/:name/:pzn/edit(.:format) products#edit
product GET /products/:name/:pzn(.:format) products#show
PATCH /products/:name/:pzn(.:format) products#update
PUT /products/:name/:pzn(.:format) products#update
DELETE /products/:name/:pzn(.:format) products#destroy

As we can see, Rails also considers :pzn as a parameter.

Now let's see how it works in Rails 6.

Rails 6.0.0.rc1

> > Rails.application.routes.draw do
> > resources :products, param: 'name/:pzn'
> > end
> >

\$ rake routes | grep products

rake aborted!
ArgumentError: :param option can't contain colons
/Users/amit/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.0.rc1/lib/action_dispatch/routing/mapper.rb:1149:in `initialize' /Users/amit/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.0.rc1/lib/action_dispatch/routing/mapper.rb:1472:in `new'
/Users/amit/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.0.rc1/lib/action_dispatch/routing/mapper.rb:1472:in `block in resources'
...
...
...

Here is the relevant issue and the pull request.

Zeitwerk is available as a gem with no additional dependencies. It means any regular Ruby project can use Zeitwerk.

How to use Zeitwerk

Zeitwerk is baked in a Rails 6 project, thanks to the Zeitwerk-Rails integration. For a non-Rails project, adding the following into the project's entry point sets up Zeitwerk.

loader = Zeitwerk::Loader.new
loader.push_dir(...)
loader.setup

For gem maintainers, Zeitwerk provides the handy .for_gem utility method

The following example from Zeitwerk documentation illustrates the usage of Zeitwerk.for_gem method.

#lib/my_gem.rb (main file)

require "zeitwerk"
loader = Zeitwerk::Loader.for_gem
loader.setup

module MyGem
  # Since the setup has been performed, at this point we are already
  # able to reference project constants, in this case MyGem::MyLogger.
  include MyLogger
end

How does Zeitwerk work?

Before we look into Zeitwerk's internals, the following section provides a quick refresher on constant-resolution in Ruby and how classical code loader of Rails works.

Ruby's constant resolution looks for a constant in the following places.

• In each entry of Module.nesting
• In each entry of Module.ancestors

It triggers 'constant_missing' callback when it can't find the constant.

Ruby used to look for constants in Object.ancestors as well, but that seems not the case anymore. An in-depth explanation of constant resolution can be found at Conrad Irwin's blog.

Classical Code Loader in Rails

Classical code loader (code loader in Rails version < 6.0) achieves autoloading by overriding Module#const_missing and loads the missing constant without the need for an explicit require call as long as the code follows certain conventions.

• The file should be within a directory in ActiveSupport::Dependencies.autoload_paths
• A file should be named after the class, i.e Admin::RoutesController => admin/routes_controller.rb

Zeitwerk Mode

Zeitwerk takes an entirely different approach in autoloading by registering constants to be autoloaded by Ruby.

Consider the following configuration in which Zeitwerk manages lib directory and lib has automobile.rb file.

loader.push_dir('./lib')

Zeitwerk then uses Module.autoload to tell Ruby that "Automobile" can be found in "lib/automobile.rb".

autoload "Automobile", "lib/automobile.rb"

Unlike classical loader, Zeitwerk takes module nesting into account while loading constants by leveraging the new Tracepoint API to go look for constants defined in subdirectories when a new class or module is defined.

Let us look at an example to understand this better.

class Automobile
  # => Tracepoint hook triggers here.
  # include Engine
end

When the tracepoint hook triggers, Zeitwerk checks for an automobile directory in the same level as automobile.rb and sets up Module.autoload for that directory and all the files (in this case ./automobile/engine.rb) within that directory.

Conclusion

Previously in Rails, we had a code loader that was riddled with gotchas and struggled to be thread safe. Zeitwerk does a better job by leveraging the new Ruby standard API and matches Ruby's semantics for constants.

Rails introduced a way to resolve such frequent errors in development from error page itself.

Rails 6 added ActiveSupport::ActionableError module to define actions we want perform on errors, right from the error page.

For example, this is how PendingMigrationError page looks like in Rails 6.

By default, a button is added on error screen that says Run pending migrations. Clicking on this button would dispatch rails db:migrate action. Page will reload once migrations run successfully.

We can also define custom actions to execute on errors.

How to define actions on error?

We need to include ActiveSupport::ActionableError module in our error class. We can monkey patch an existing error class or define custom error class.

#action api is provided to define actions on error. First argument in #action is name of the action. This string would be displayed on the button on error page. Second argument is a block where we can write commands or code to fix the error.

Let's take an example of seeding posts data from controller, if posts not already present.

# app/controllers/posts_controller.rb

class PostsController < ApplicationController

def index
@posts = Post.all
if @posts.empty?
raise PostsMissingError
end
end

end

# app/errors/posts_missing_error.rb

class PostsMissingError < StandardError

include ActiveSupport::ActionableError

action "seed posts data" do
Rails::Command.invoke 'posts:seed'
end

end

# lib/tasks/posts.rake

namespace :posts do

desc 'posts seed task'
task :seed do
Post.create(title: 'First Post')
end

end

# app/views/posts/index.html.erb

<% @posts.each do |post| %>
<%= post.title %>
<% end %>

Let's check /posts (posts#index action) when no posts are present. We would get an error page with an action button on it as shown below.

Clicking on seed posts data action button will run our rake task and create posts. Rails will automatically reload /posts after running rake task.

ActionDispatch::ActionableExceptions middleware takes care of invoking actions from error page. ActionableExceptions middleware dispatches action to ActionableError and redirects back when action block has successfully run. Action buttons are added on error page from this middleware template.

Checkout the pull request for more information on actionable error.

It also provides remove_foreign_key to remove the foreign key constraint.

Before Rails 6, add_foreign_key and remove_foreign_key were not supported for SQLite3.

Rails 6 now adds this support. Now, we can create and remove foreign key constraints using add_foreign_key and remove_foreign_key in SQLite3.

Let's checkout how it works.

Rails 5.2

We have two tables named as orders and users. Now, let's add foreign key constraint of users in orders table using add_foreign_key and then try removing it using remove_foreign_key.

>> class AddUserReferenceToOrders < ActiveRecord::Migration[6.0]
>>   def change
>>     add_column :orders, :user_id, :integer
>>     add_foreign_key :orders, :users
>>   end
>> end

=> :change

>> AddUserReferenceToOrders.new.change
-- add_column(:orders, :user_id, :integer)
   (1.2ms)  ALTER TABLE "orders" ADD "user_id" integer
   -> 0.0058s
-- add_foreign_key(:orders, :users)
   -> 0.0000s

=> nil

>> class RemoveUserForeignKeyFromOrders < ActiveRecord::Migration[6.0]
>>   def change
>>     remove_foreign_key :orders, :users
>>   end
>> end

=> :change

>> RemoveUserForeignKeyFromOrders.new.change
-- remove_foreign_key(:orders, :users)
   -> 0.0001s

=> nil

We can see that add_foreign_key and remove_foreign_key are ignored by Rails 5.2 with SQLite3.

Rails 6.0.0.rc1

We have two tables named as orders and users. Now, let's add foreign key constraint of users in orders table using add_foreign_key.

>> class AddUserReferenceToOrders < ActiveRecord::Migration[6.0]
>>   def change
>>     add_column :orders, :user_id, :integer
>>     add_foreign_key :orders, :users
>>   end
>> end

=> :change

>> AddUserReferenceToOrders.new.change
-- add_column(:orders, :user_id, :integer)
   (1.0ms)  SELECT sqlite_version(*)
   (2.9ms)  ALTER TABLE "orders" ADD "user_id" integer
   -> 0.0091s
-- add_foreign_key(:orders, :users)
   (0.0ms)  begin transaction
   (0.1ms)  PRAGMA foreign_keys
   (0.1ms)  PRAGMA defer_foreign_keys
   (0.0ms)  PRAGMA defer_foreign_keys = ON
   (0.1ms)  PRAGMA foreign_keys = OFF
   (0.2ms)  CREATE TEMPORARY TABLE "aorders" ("id" integer NOT NULL PRIMARY KEY, "number" varchar DEFAULT NULL, "total" decimal DEFAULT NULL, "completed_at" datetime DEFAULT NULL, "created_at" datetime(6) NOT NULL, "updated_at" datetime(6) NOT NULL, "user_id" integer DEFAULT NULL)
   (0.1ms)  INSERT INTO "aorders" ("id","number","total","completed_at","created_at","updated_at","user_id")
                     SELECT "id","number","total","completed_at","created_at","updated_at","user_id" FROM "orders"
   (0.3ms)  DROP TABLE "orders"
   (0.1ms)  CREATE TABLE "orders" ("id" integer NOT NULL PRIMARY KEY, "number" varchar DEFAULT NULL, "total" decimal DEFAULT NULL, "completed_at" datetime DEFAULT NULL, "created_at" datetime(6) NOT NULL, "updated_at" datetime(6) NOT NULL, "user_id" integer DEFAULT NULL, CONSTRAINT "fk_rails_f868b47f6a"
FOREIGN KEY ("user_id")
  REFERENCES "users" ("id")
)
   (0.1ms)  INSERT INTO "orders" ("id","number","total","completed_at","created_at","updated_at","user_id")
                     SELECT "id","number","total","completed_at","created_at","updated_at","user_id" FROM "aorders"
   (0.1ms)  DROP TABLE "aorders"
   (0.0ms)  PRAGMA defer_foreign_keys = 0
   (0.0ms)  PRAGMA foreign_keys = 1
   (0.6ms)  commit transaction
   -> 0.0083s

=> []

>> class RemoveUserForeignKeyFromOrders < ActiveRecord::Migration[6.0]
>>   def change
>>     remove_foreign_key :orders, :users
>>   end
>> end

=> :change

>> RemoveUserForeignKeyFromOrders.new.change
-- remove_foreign_key(:orders, :users)
   (1.4ms)  SELECT sqlite_version(*)
   (0.0ms)  begin transaction
   (0.0ms)  PRAGMA foreign_keys
   (0.0ms)  PRAGMA defer_foreign_keys
   (0.0ms)  PRAGMA defer_foreign_keys = ON
   (0.0ms)  PRAGMA foreign_keys = OFF
   (0.2ms)  CREATE TEMPORARY TABLE "aorders" ("id" integer NOT NULL PRIMARY KEY, "number" varchar DEFAULT NULL, "total" decimal DEFAULT NULL, "completed_at" datetime DEFAULT NULL, "created_at" datetime(6) NOT NULL, "updated_at" datetime(6) NOT NULL, "user_id" integer DEFAULT NULL)
   (0.3ms)  INSERT INTO "aorders" ("id","number","total","completed_at","created_at","updated_at","user_id")
                     SELECT "id","number","total","completed_at","created_at","updated_at","user_id" FROM "orders"
   (0.4ms)  DROP TABLE "orders"
   (0.1ms)  CREATE TABLE "orders" ("id" integer NOT NULL PRIMARY KEY, "number" varchar DEFAULT NULL, "total" decimal DEFAULT NULL, "completed_at" datetime DEFAULT NULL, "created_at" datetime(6) NOT NULL, "updated_at" datetime(6) NOT NULL, "user_id" integer DEFAULT NULL)
   (0.1ms)  INSERT INTO "orders" ("id","number","total","completed_at","created_at","updated_at","user_id")
                     SELECT "id","number","total","completed_at","created_at","updated_at","user_id" FROM "aorders"
   (0.1ms)  DROP TABLE "aorders"
   (0.0ms)  PRAGMA defer_foreign_keys = 0
   (0.0ms)  PRAGMA foreign_keys = 1
   (0.7ms)  commit transaction
   -> 0.0179s

=> []

Now, let's remove foreign key constraint of users from orders table using remove_foreign_key.

>> class RemoveUserForeignKeyFromOrders < ActiveRecord::Migration[6.0]
>>   def change
>>     remove_foreign_key :orders, :users
>>   end
>> end

=> :change

>> RemoveUserForeignKeyFromOrders.new.change
-- remove_foreign_key(:orders, :users)
   (1.4ms)  SELECT sqlite_version(*)
   (0.0ms)  begin transaction
   (0.0ms)  PRAGMA foreign_keys
   (0.0ms)  PRAGMA defer_foreign_keys
   (0.0ms)  PRAGMA defer_foreign_keys = ON
   (0.0ms)  PRAGMA foreign_keys = OFF
   (0.2ms)  CREATE TEMPORARY TABLE "aorders" ("id" integer NOT NULL PRIMARY KEY, "number" varchar DEFAULT NULL, "total" decimal DEFAULT NULL, "completed_at" datetime DEFAULT NULL, "created_at" datetime(6) NOT NULL, "updated_at" datetime(6) NOT NULL, "user_id" integer DEFAULT NULL)
   (0.3ms)  INSERT INTO "aorders" ("id","number","total","completed_at","created_at","updated_at","user_id")
                     SELECT "id","number","total","completed_at","created_at","updated_at","user_id" FROM "orders"
   (0.4ms)  DROP TABLE "orders"
   (0.1ms)  CREATE TABLE "orders" ("id" integer NOT NULL PRIMARY KEY, "number" varchar DEFAULT NULL, "total" decimal DEFAULT NULL, "completed_at" datetime DEFAULT NULL, "created_at" datetime(6) NOT NULL, "updated_at" datetime(6) NOT NULL, "user_id" integer DEFAULT NULL)
   (0.1ms)  INSERT INTO "orders" ("id","number","total","completed_at","created_at","updated_at","user_id")
                     SELECT "id","number","total","completed_at","created_at","updated_at","user_id" FROM "aorders"
   (0.1ms)  DROP TABLE "aorders"
   (0.0ms)  PRAGMA defer_foreign_keys = 0
   (0.0ms)  PRAGMA foreign_keys = 1
   (0.7ms)  commit transaction
   -> 0.0179s

=> []

We can see here that with Rails 6, add_foreign_key and remove_foreign_key work and were able to add and remove foreign key constraint respectively.

Here is the relevant pull request.

This works the same way as Hash#dig (Link is not available).

It extracts the nested value specified by the sequence of keys.

Hash#dig (Link is not available) was introduced in Ruby 2.3.

Before Rails 6, we can achieve the same thing by first converting session to a hash and then calling Hash#dig (Link is not available) on it.

Let's checkout how it works.

Rails 5.2

Let's add some user information in session and use dig after converting it to a hash.

>> session[:user] = { email: 'jon@bigbinary.com', name: { first: 'Jon', last: 'Snow' }  }

=> {:email=>"jon@bigbinary.com", :name=>{:first=>"Jon", :last=>"Snow"}}

>> session.to_hash

=> {"session_id"=>"5fe8cc73c822361e53e2b161dcd20e47", "_csrf_token"=>"gyFd5nEEkFvWTnl6XeVbJ7qehgL923hJt8PyHVCH/DA=", "return_to"=>"http://localhost:3000", "user"=>{:email=>"jon@bigbinary.com", :name=>{:first=>"Jon", :last=>"Snow"}}}


>> session.to_hash.dig("user", :name, :first)

=> "Jon"

Rails 6.0.0.rc1

Let's add the same information to session and now use dig on session object without converting it to a hash.

>> session[:user] = { email: 'jon@bigbinary.com', name: { first: 'Jon', last: 'Snow' }  }

=> {:email=>"jon@bigbinary.com", :name=>{:first=>"Jon", :last=>"Snow"}}

>> session.dig(:user, :name, :first)

=> "Jon"

Here is the relevant pull request.

Before Rails 6, keys with the _html suffix in the language locale files are automatically marked as HTML safe. These HTML safe keys do not get escaped when used in the views.

# config/locales/en.yml

en:
  home:
    index:
      title_html: <h2>We build web & mobile applications</h2>
      description_html:
        We are a dynamic team of <em>developers</em> and <em>designers</em>.
      sections:
        blogs:
          title_html: <h3>Blogs & publications</h3>
          description_html:
            We regularly write our blog. Our blogs are covered by <strong>Ruby
            Inside</strong> and <strong>Ruby Weekly Newsletter</strong>.

<!-- app/views/home/index.html.erb -->

<%= t('.title_html') %>
<%= t('.description_html') %>

<%= t('.sections.blogs.title_html') %>
<%= t('.sections.blogs.description_html') %>

Once rendered, this page looks like this.

This way of marking translations as HTML safe by adding _html suffix to the keys does not work as expected when the value is an array.

# config/locales/en.yml

en:
home:
index:
title_html: <h2>We build web & mobile applications</h2>
description_html: We are a dynamic team of <em>developers</em> and <em>designers</em>.
sections:
blogs:
title_html: <h3>Blogs & publications</h3>
description_html: We regularly write our blog. Our blogs are covered by <strong>Ruby Inside</strong> and <strong>Ruby Weekly Newsletter</strong>.
services:
title_html: <h3>Services we offer</h3>
list_html: - <strong>Ruby on Rails</strong> - React.js &#9883; - React Native &#9883; &#128241;

<!-- app/views/home/index.html.erb -->

<%= t('.title_html') %>
<%= t('.description_html') %>

<%= t('.sections.blogs.title_html') %>
<%= t('.sections.blogs.description_html') %>

<%= t('.sections.services.title_html') %>

<ul>
  <% t('.sections.services.list_html').each do |service| %>
    <li><%= service %></li>
  <% end %>
<ul>

The rendered page escapes the unsafe HTML while rendering the array of translations for the key .sections.services.list_html even though that key has the _html suffix.

A workaround is to manually mark all the translations in that array as HTML safe using the methods such as #raw or #html_safe.

<!-- app/views/home/index.html.erb -->

<%= t('.title_html') %>
<%= t('.description_html') %>

<%= t('.sections.blogs.title_html') %>
<%= t('.sections.blogs.description_html') %>

<%= t('.sections.services.title_html') %>

<ul>
  <% t('.sections.services.list_html').each do |service| %>
    <li><%= service.html_safe %></li>
  <% end %>
<ul>

Arrays of translations are trusted as HTML safe by using the '_html' suffix in Rails 6

In Rails 6, the unexpected behavior of not marking an array of translations as HTML safe even though the key of that array has the _html suffix is fixed.

# config/locales/en.yml

en:
home:
index:
title_html: <h2>We build web & mobile applications</h2>
description_html: We are a dynamic team of <em>developers</em> and <em>designers</em>.
sections:
blogs:
title_html: <h3>Blogs & publications</h3>
description_html: We regularly write our blog. Our blogs are covered by <strong>Ruby Inside</strong> and <strong>Ruby Weekly Newsletter</strong>.
services:
title_html: <h3>Services we offer</h3>
list_html: - <strong>Ruby on Rails</strong> - React.js &#9883; - React Native &#9883; &#128241;

<!-- app/views/home/index.html.erb -->

<%= t('.title_html') %>
<%= t('.description_html') %>

<%= t('.sections.blogs.title_html') %>
<%= t('.sections.blogs.description_html') %>

<%= t('.sections.services.title_html') %>

<ul>
  <% t('.sections.services.list_html').each do |service| %>
    <li><%= service %></li>
  <% end %>
<ul>

We can see above that we no longer need to manually mark the translations as HTML safe for the key .sections.services.title_html using the methods such as #raw or #html_safe since that key has the _html suffix.

To learn more about this feature, please checkout rails/rails#32361.

Rails provides a way of doing this. We can add parameters to Rails.application.config.filter_parameters.

There is one more way of doing this in Rails. We can also use https://api.rubyonrails.org/classes/ActionDispatch/Http/FilterParameters.html.

However there is still a security issue when we call inspect on an ActiveRecord object for logging purposes. In this case, Rails does not consider Rails.application.config.filter_parameters and displays the sensitive information.

Rails 6 fixes this. It considers Rails.application.config.filter_parameters while inspecting an object.

Rails 6 also provides an alternative way to filter columns on ActiveRecord level by adding filter_attributes on ActiveRecord::Base.

In Rails 6, filter_attributes on ActiveRecord::Base takes priority over Rails.application.config.filter_parameters.

Let's checkout how it works.

Rails 6.0.0.rc1

Let's create a user record and call inspect on it.

>> class User < ApplicationRecord
>>  validates :email, :password, presence: true
>> end

=> {:presence=>true}

>> User.create(email: 'john@bigbinary.com', password: 'john_wick_bigbinary')
BEGIN
  User Create (0.6ms)  INSERT INTO "users" ("email", "password", "created_at", "updated_at") VALUES ($1, $2, $3, $4) RETURNING "id"  [["email", "john@bigbinary.com"], ["password", "john_wick_bigbinary"], ["created_at", "2019-05-17 21:34:34.504394"], ["updated_at", "2019-05-17 21:34:34.504394"]]
COMMIT

=> #<User id: 2, email: "john@bigbinary.com", password: [FILTERED], created_at: "2019-05-17 21:34:34", updated_at: "2019-05-17 21:34:34">

We can see that password is filtered as it is added to Rails.application.config.filter_parameters by default in config/initializers/filter_parameter_logging.rb.

Now let's add just :email to User.filter_attributes

>> User.filter_attributes = [:email]

=> [:email]

>> User.first.inspect
SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> "#<User id: 2, email: [FILTERED], password: \"john_wick_bigbinary\", created_at: \"2019-05-17 21:34:34\", updated_at: \"2019-05-17 21:34:34\">"

We can see here that User.filter_attributes took priority over Rails.application.config.filter_parameters and removed filtering from password and filtered just email.

Now, let's add both :email and :password to User.filter_attributes.

>> User.filter_attributes = [:email, :password]

=> [:email, :password]

>> User.first.inspect

=> "#<User id: 2, email: [FILTERED], password: [FILTERED], created_at: \"2019-05-17 21:34:34\", updated_at: \"2019-05-17 21:34:34\">"

We can see that now both email and password are filtered out.

Here is the relevant pull request.

Before Rails 6, it used to return ActiveRecord::ActiveRecordError.

Let's checkout how it works.

Rails 5.2

Let's create an orders table and try using :limit with a column named as quantity with data type integer.

> > class CreateOrders < ActiveRecord::Migration[5.2]
> > def change
> > create_table :orders do |t|
> > t.string :item
> > t.integer :quantity, limit: 10
> >
> >       t.timestamps
> >     end
> >
> > end
> > end

=> :change

> > CreateOrders.new.change
> > -- create_table(:orders)

=> Traceback (most recent call last):
2: from (irb):11
1: from (irb):3:in 'change'
ActiveRecord::ActiveRecordError (No integer type has byte size 10. Use a numeric with scale 0 instead.)

We can see that use of :limit with integer column raises ActiveRecord::ActiveRecordError in Rails 5.2.

Now let's try using :precision of 10 with a datetime column.

> > class CreateOrders < ActiveRecord::Migration[5.2]
> > def change
> > create_table :orders do |t|
> > t.string :item
> > t.integer :quantity
> > t.datetime :completed_at, precision: 10
> >
> >       t.timestamps
> >     end
> >
> > end
> > end

=> :change

> > CreateOrders.new.change
> > -- create_table(:orders)

=> Traceback (most recent call last):
2: from (irb):12
1: from (irb):3:in 'change'
ActiveRecord::ActiveRecordError (No timestamp type has precision of 10. The allowed range of precision is from 0 to 6)

We can see that invalid value of :precision with datetime column also raises ActiveRecord::ActiveRecordError in Rails 5.2.

Rails 6.0.0.rc1

Let's create an orders table and try using :limit with a column named as quantity with data type integer in Rails 6.

> > class CreateOrders < ActiveRecord::Migration[6.0]
> > def change
> > create_table :orders do |t|
> > t.string :item
> > t.integer :quantity, limit: 10
> >
> >       t.timestamps
> >     end
> >
> > end
> > end

=> :change

> > CreateOrders.new.change
> > -- create_table(:orders)

=> Traceback (most recent call last):
2: from (irb):11
1: from (irb):3:in 'change'
ArgumentError (No integer type has byte size 10. Use a numeric with scale 0 instead.)

We can see that use of :limit with integer column raises ArgumentError in Rails 6.

Now let's try using :precision of 10 with a datetime column.

> > class CreateOrders < ActiveRecord::Migration[6.0]
> > def change
> > create_table :orders do |t|
> > t.string :item
> > t.integer :quantity
> > t.datetime :completed_at, precision: 10
> >
> >       t.timestamps
> >     end
> >
> > end
> > end

=> :change

> > CreateOrders.new.change
> > -- create_table(:orders)

=> Traceback (most recent call last):
2: from (irb):12
1: from (irb):3:in 'change'
ArgumentError (No timestamp type has precision of 10. The allowed range of precision is from 0 to 6)

We can see that invalid value of :precision with datetime column also raises ArgumentError in Rails 6.

Here is the relevant pull request.

Custom configuration can be mentioned in either config/cable.yml or config/application.rb as shown below.

# config/cable.yml

production:
  url: redis://redis.example.com:6379
  adapter: redis
  channel_prefix: custom_

Or

# config/application.rb

config.action_cable.cable = { adapter: "redis", channel_prefix: "custom_" }

In some cases, we need another Action Cable server running separately from application with a different set of configuration.

Problem is that both approaches mentioned earlier set Action Cable server configuration on application boot up. This configuration can not be changed for the second server.

Rails 6 has added a provision to pass custom configuration. Rails 6 allows us to pass ActionCable::Server::Configuration object as an option when initializing a new Action Cable server.

config = ActionCable::Server::Configuration.new
config.cable = { adapter: "redis", channel_prefix: "custom_" }

ActionCable::Server::Base.new(config: config)

For more details on Action Cable configurations, head to Action Cable docs.

Here's the relevant pull request for this change.

Please note that documentation of ActiveSupport::HashWithIndifferentAccess#assoc in Rails 5.2 shows that ActiveSupport::HashWithIndifferentAccess#assoc works with symbol keys but it doesn't.

In Rails 6, ActiveSupport::HashWithIndifferentAccess implements a hash where string and symbol keys are considered to be the same.

Before Rails 6, HashWithIndifferentAccess#assoc used to work with just string keys.

Let's checkout how it works.

Rails 5.2

Let's create an object of ActiveSupport::HashWithIndifferentAccess and call assoc on that object.

>> info = { name: 'Mark', email: 'mark@bigbinary.com' }.with_indifferent_access

=> {"name"=>"Mark", "email"=>"mark@bigbinary.com"}

>> info.assoc(:name)

=> nil

>> info.assoc('name')

=> ["name", "Mark"]

We can see that assoc does not work with symbol keys with ActiveSupport::HashWithIndifferentAccess in Rails 5.2.

Rails 6.0.0.beta2

Now, let's call assoc on the same hash in Rails 6 with both string and symbol keys.

>> info = { name: 'Mark', email: 'mark@bigbinary.com' }.with_indifferent_access

=> {"name"=>"Mark", "email"=>"mark@bigbinary.com"}

>> info.assoc(:name)

=> ["name", "Mark"]

>> info.assoc('name')

=> ["name", "Mark"]

As we can see, assoc works perfectly fine with both string and symbol keys with ActiveSupport::HashWithIndifferentAccess in Rails 6.

Here is the relevant pull request.

Before Rails 6, calling #html_safe? on a slice of an HTML safe string returns nil.

>> html_content = "<div>Hello, world!</div>".html_safe
# => "<div>Hello, world!</div>"
>> html_content.html_safe?
# => true
>> html_content[0..-1].html_safe?
# => nil

Also, before Rails 6, the ActiveSupport::SafeBuffer#* method does not preserve the HTML safe status as well.

>> line_break = "<br />".html_safe
# => "<br />"
>> line_break.html_safe?
# => true
>> two_line_breaks = (line_break * 2)
# => "<br /><br />"
>> two_line_breaks.html_safe?
# => nil

Rails 6 returns expected status of #html_safe?

In Rails 6, both of the above cases have been fixed properly.

Therefore, we will now get the status of #html_safe? as expected.

>> html_content = "<div>Hello, world!</div>".html_safe
# => "<div>Hello, world!</div>"
>> html_content.html_safe?
# => true
>> html_content[0..-1].html_safe?
# => true

>> line_break = "<br />".html_safe
# => "<br />"
>> line_break.html_safe?
# => true
>> two_line_breaks = (line_break * 2)
# => "<br /><br />"
>> two_line_breaks.html_safe?
# => true

Please check rails/rails#33808 and rails/rails#36012 for the relevant changes.

Before Rails 6, if we use where.not with multiple attributes, it applies logical NOR (NOT(A) AND NOT(B)) in WHERE clause of the query. This does not always work as expected.

Let's look at an example to understand this better.

We have Post model with a polymorphic association.

Rails 5.2

>> Post.all
=> #<ActiveRecord::Relation [
#<Post id: 1, title: "First Post", source_type: "Feed", source_id: 100>,
#<Post id: 2, title: "Second Post", source_type: "Feed", source_id: 101>]>

>> Post.where(source_type: "Feed", source_id: 100)
=> #<ActiveRecord::Relation [#<Post id: 1, title: "First Post", source_type: "Feed", source_id: 100>]>

>> Post.where.not(source_type: "Feed", source_id: 100)
=> #<ActiveRecord::Relation []>

In the last query, we expect ActiveRecord to fetch one record.

Let's check SQL generated for the above case.

>> Post.where.not(source_type: "Feed", source_id: 100).to_sql

=> SELECT "posts".* FROM "posts" WHERE "posts"."source_type" != 'Feed' AND "posts"."source_id" != 100

where.not applies AND to the negation of source_type and source_id, and fails to fetch expected records.

In such cases, correct implementation of where.not would be logical NAND (NOT(A) OR NOT(B)).

Let us query posts table using NAND this time.

>> Post.where("source_type != 'Feed' OR source_id != 100")

   SELECT "posts".* FROM "posts" WHERE (source_type != 'Feed' OR source_id != 100)

=> #<ActiveRecord::Relation [#<Post id: 2, title: "Second Post", source_type: "Feed", source_id: 101>]>

Above query works as expected and returns one record. Rails 6.1 will change where.not working to NAND similar to the above query.

Rails 6.0.0.rc1

>> Post.where.not(source_type: "Feed", source_id: 100)

DEPRECATION WARNING: NOT conditions will no longer behave as NOR in Rails 6.1. To continue using NOR conditions, NOT each conditions manually (`.where.not(:source_type => ...).where.not(:source_id => ...)`). (called from irb_binding at (irb):1)

=> #<ActiveRecord::Relation []>

It is well mentioned in deprecation warning that if we wish to use NOR condition with multiple attributes, we can chain multiple where.not using a single predicate.

>> Post.where.not(source_type: "Feed").where.not(source_id: 100)

Here's the relevant discussion and pull request for this change.

What is Optimizer Hints?

Many relational database management systems (RDBMS) have a query optimizer. The job of the query optimizer is to determine the most efficient and fast plan to execute a given SQL query. Query optimizer has to consider all possible query execution plans before it can determine which plan is the optimal plan for executing the given SQL query and then compile and execute that query.

An optimal plan is chosen by the query optimizer by calculating the cost of each possible plans. Typically, when the number of tables referenced in a join query increases, then the time spent in query optimization grows exponentially which often affects the system's performance. The fewer the execution plans the query optimizer needs to evaluate, the lesser time is spent in compiling and executing the query.

As an application designer, we might have more context about the data stored in our database. With the contextual knowledge about our database, we might be able to choose a more efficient execution plan than the query optimizer.

This is where the optimizer hints or optimizer guidelines come into picture.

Optimizer hints allow us to control the query optimizer to choose a certain query execution plan based on the specific criteria. In other words, we can hint the optimizer to use or ignore certain optimization plans using optimizer hints.

Usually, optimizer hints should be provided only when executing a complex query involving multiple table joins.

Note that the optimizer hints only affect an individual SQL statement. To alter the optimization strategies at the global level, there are different mechanisms supported by different databases. Optimizer hints provide finer control over other mechanisms which allow altering optimization plans by other means.

Optimizer hints are supported by many databases such as MySQL, PostgreSQL with the help of pg_hint_plan extension, Oracle, MS SQL, IBM DB2, etc. with varying syntax and options.

Optimizer Hints in Rails 6

Before Rails 6, we have to execute a raw SQL query to use the optimizer hints.

query = "SELECT
            /*+ JOIN_ORDER(articles, users) MAX_EXECUTION_TIME(60000) */
            articles.*
         FROM articles
         INNER JOIN users
         ON users.id = articles.user_id
         WHERE (published_at > '2019-02-17 13:15:44')
        ".squish

ActiveRecord::Base.connection.execute(query)

In the above query, we provided two optimizer hints to MySQL .

/*+ HINT_HERE ANOTHER_HINT_HERE ... */

Another approach to use optimizer hints prior to Rails 6 is to use a monkey patch like this.

In Rails 6, using optimizer hints is easier.

The same example looks like this in Rails 6.

Article
  .joins(:user)
  .where("published_at > ?", 2.months.ago)
  .optimizer_hints(
    "JOIN_ORDER(articles, users)",
    "MAX_EXECUTION_TIME(60000)"
  )

This produces the same SQL query as above but the result is of type ActiveRecord::Relation.

In PostgreSQL (using the pg_hint_plan extension), the optimizer hints have a different syntax.

Article
  .joins(:user)
  .where("published_at > ?", 2.months.ago)
  .optimizer_hints("Leading(articles users)", "SeqScan(articles)")

Please checkout the documentation of each database separately to learn the support and syntax of optimizer hints.

To learn more, please checkout this PR which introduced the #optimization_hints method to Rails 6.

Bonus example: Using optimizer hints to speedup a slow SQL statement in MySQL

Consider that we have articles table with some indexes.

class CreateArticles < ActiveRecord::Migration[6.0]
  def change
    create_table :articles do |t|
      t.string :title, null: false
      t.string :slug, null: false
      t.references :user
      t.datetime :published_at
      t.text :description

      t.timestamps

      t.index :slug, unique: true
      t.index [:published_at]
      t.index [:slug, :user_id]
      t.index [:published_at, :user_id]
      t.index [:title, :slug]
    end
  end
end

Let's try to fetch all the articles which have been published in the last 2 months.

>> Article.joins(:user).where("published_at > ?", 2.months.ago)
# Article Load (10.5ms)  SELECT `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:38:18.647296')
=> #<ActiveRecord::Relation [#<Article id: 20, title: "Article 20", slug: "article-20", user_id: 1, ...]>

Let's use EXPLAIN to investigate why it is taking 10.5ms to execute this query.

>> Article.joins(:user).where("published_at > ?", 2.months.ago).explain
# Article Load (13.9ms)  SELECT `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:39:05.380577')
=> # EXPLAIN for: SELECT `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:39:05.380577')
# +--------+----------+----------------+-----------+------+----------+-------+
# | select |   table  | possible_keys  | key       | rows | filtered | Extra |
# | _type  |          |                |           |      |          |       |
# +--------+----------+----------------+-----------+------+----------+-------+
# | SIMPLE |   users  | PRIMARY        | PRIMARY   | 2    | 100.0    | Using |
# |        |          |                |           |      |          | index |
# +--------+----------+----------------+-----------+------+----------+-------+
# | SIMPLE | articles | index          | index     | 9866 | 10.0     | Using |
# |        |          | _articles      | _articles |      |          | where |
# |        |          | _on_user_id,   | _on       |      |          |       |
# |        |          | index          | _user_id  |      |          |       |
# |        |          | _articles      |           |      |          |       |
# |        |          | _on            |           |      |          |       |
# |        |          | _published_at, |           |      |          |       |
# |        |          | index          |           |      |          |       |
# |        |          | _articles      |           |      |          |       |
# |        |          | _on            |           |      |          |       |
# |        |          | _published_at  |           |      |          |       |
# |        |          | _and_user_id   |           |      |          |       |
# +--------+----------+----------------+-----------+------+----------+-------+

According to the above table, it appears that the query optimizer is considering users table first and then the articles table.

The rows column indicates the estimated number of rows the query optimizer must examine to execute the query.

The filtered column indicates an estimated percentage of table rows that will be filtered by the table condition.

The formula rows x filtered gives the number of rows that will be joined with the following table.

Also,

• For users table, the number of rows to be joined with the following table is 2 x 100% = 2,
• For articles table, the number of rows to be joined with the following table is 500 * 7.79 = 38.95.

Since the articles tables contain more records which references very few records from the users table, it would be better to consider the articles table first and then the users table.

We can hint MySQL to consider the articles table first as follows.

>> Article.joins(:user).where("published_at > ?", 2.months.ago).optimizer_hints("JOIN_ORDER(articles, users)")
# Article Load (2.2ms)  SELECT `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:54:06.230651')
=> #<ActiveRecord::Relation [#<Article id: 20, title: "Article 20", slug: "article-20", user_id: 1, ...]>

Note that it took 2.2ms now to fetch the same records by providing JOIN_ORDER(articles, users) optimization hint.

Let's try to EXPLAIN what changed by using this JOIN_ORDER(articles, users) optimization hint.

>> Article.joins(:user).where("published_at > ?", 2.months.ago).optimizer_hints("JOIN_ORDER(articles, users)").explain
# Article Load (4.1ms)  SELECT /*+ JOIN_ORDER(articles, users) */ `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:55:24.335152')
=> # EXPLAIN for: SELECT /*+ JOIN_ORDER(articles, users) */ `articles`.* FROM `articles` INNER JOIN `users` ON `users`.`id` = `articles`.`user_id` WHERE (published_at > '2019-02-17 11:55:24.335152')
# +--------+----------+----------------+-----------+------+----------+--------+
# | select |   table  | possible_keys  | key       | rows | filtered | Extra  |
# | _type  |          |                |           |      |          |        |
# +--------+----------+----------------+-----------+------+----------+--------+
# | SIMPLE | articles | index          | index     | 769  | 100.0    | Using  |
# |        |          | _articles      | _articles |      |          | index  |
# |        |          | _on_user_id,   | _on       |      |          | condi  |
# |        |          | index          | _publish  |      |          | tion;  |
# |        |          | _articles      | ed_at,    |      |          | Using  |
# |        |          | _on            |           |      |          | where  |
# |        |          | _published_at, |           |      |          |        |
# |        |          | index          |           |      |          |        |
# |        |          | _articles      |           |      |          |        |
# |        |          | _on            |           |      |          |        |
# |        |          | _published_at  |           |      |          |        |
# |        |          | _and_user_id   |           |      |          |        |
# +--------+----------+----------------+-----------+------+----------+--------+
# | SIMPLE | users    | PRIMARY        | PRIMARY   | 2    | 100.0    | Using  |
# |        |          |                |           |      |          | index  |
# +--------+----------+----------------+-----------+------+----------+--------+

The result of the EXPLAIN query shows that the articles table was considered first and then the users table as expected. We can also see that the index_articles_on_published_at index key was considered from the possible keys to execute the given query. The filtered column for both tables shows that the number of filtered rows was 100% which means no filtering of rows occurred.

We hope this example helps in understanding how to use #explain and #optimization_hints methods in order to investigate and debug the performance issues and then fixing it.

By taking the benefit of this feature, Rails 6 now reports the allocations made while rendering a view template, a partial and a collection.

Started GET "/articles" for ::1 at 2019-04-15 17:24:09 +0530
Processing by ArticlesController#index as HTML
Rendering articles/index.html.erb within layouts/application
Rendered shared/\_ad_banner.html.erb (Duration: 0.1ms | Allocations: 6)
Article Load (1.3ms) SELECT "articles".\* FROM "articles"
↳ app/views/articles/index.html.erb:5
Rendered collection of articles/\_article.html.erb [100 times] (Duration: 6.1ms | Allocations: 805)
Rendered articles/index.html.erb within layouts/application (Duration: 17.6ms | Allocations: 3901)
Completed 200 OK in 86ms (Views: 83.6ms | ActiveRecord: 1.3ms | Allocations: 29347)

Notice the Allocations: information in the above logs.

We can see that

• 6 objects were allocated while rendering shared/_ad_banner.html.erb view partial,
• 805 objects were allocated while rendering a collection of 100 articles/_article.html.erb view partials,
• and 3901 objects were allocated while rendering articles/index.html.erb view template.

We can use this information to understand how much time was spent while rendering a view template and how many objects were allocated in the process' memory between the time when that view template had started rendering and the time when that view template had finished rendering.

To learn more about this feature, please check rails/rails#34136.

Note that we can also collect this information by subscribing to Action View hooks.

ActiveSupport::Notifications.subscribe /^render\_.+.action_view\$/ do |event|
views_path = Rails.root.join("app/views/").to_s
template_identifier = event.payload[:identifier]
template_name = template_identifier.sub(views_path, "")
message = "[#{event.name}] #{template_name} (Allocations: #{event.allocations})"

ViewAllocationsLogger.log(message)
end

This should log something like this.

[render_partial.action_view] shared/\_ad_banner.html.erb (Allocations: 43)

[render_collection.action_view] articles/\_article.html.erb (Allocations: 842)

[render_template.action_view] articles/index.html.erb (Allocations: 4108)

Here is how it can be used.

> > User.annotate("User whose name starts with 'A'").where("name LIKE ?", "A%")

SELECT "users"._ FROM "users"
WHERE (name LIKE 'A%')
/_ User whose name starts with 'A' \*/
LIMIT ? [["LIMIT", 11]]

ActiveRecord::Relation#annotate allows to add multiple annotations on a query

> > bigbinary = Organization.find_by!(name: "BigBinary")
> > User.annotate("User whose name starts with 'A'")

       .annotate("AND belongs to BigBinary organization")
       .where("name LIKE ?", "A%")
       .where(organization: bigbinary)

SELECT "users"._ FROM "users"
WHERE (name LIKE 'A%') AND "users"."organization_id" = ?
/_ Users whose name starts with 'A' _/
/_ AND belongs to BigBinary organization \*/
LIMIT ? [["organization_id", 1], ["LIMIT", 11]]

Also, ActiveRecord::Relation#annotate allows annotating scopes and model associations.

class User < ActiveRecord::Base
scope :active, -> { where(status: 'active').annotate("Active users") }
end

> > User.active
> > SELECT "users"._ FROM "users"
> > /_ Active users \*/
> > LIMIT ? [["LIMIT", 11]]
> > ~~~

Check out the
[pull request](https://github.com/rails/rails/pull/35617)
for more details on this.

Before Rails 6, we have to provide a custom block to perform custom logging and monitoring around retries and discards of the jobs defined using Active Job framework.

class Container::DeleteJob < ActiveJob::Base
retry_on Timeout::Error, wait: 2.seconds, attempts: 3 do |job, error|
message = "Stopped retrying #{job.class} (JID #{job.job_id})
with #{job.arguments.join(', ')} due to
'#{error.class} - #{error.message}'.
This job was retried for #{job.executions} times.".squish

    BackgroundJob::ErrorLogger.log(message)

end

discard_on Container::NotFoundError do |job, error|
message = "Discarded #{job.class} (JID #{job.job_id})
with #{job.arguments.join(', ')} due to
'#{error.class} - #{error.message}' error.".squish

    BackgroundJob::ErrorLogger.log(message)

end

def perform(container_id)
Container::DeleteService(container_id).process

    # Will raise Container::NotFoundError
    # if no container is found with 'container_id'.

    # Might raise Timeout::Error when the remote API is not responding.

end
end

Notice the custom blocks provided to retry_on and discard_on methods to an individual job in the above example.

Extracting such custom logic to a base class or to a 3rd-party gem is possible but it will be non-standard and will be a bit difficult task.

An alternative approach is to subscribe to the hooks instrumented using Active Support Instrumentation API which is a standard and recommended way. Prior versions of Rails 6 already instruments some hooks such as enqueue_at.active_job, enqueue.active_job, perform_start.active_job, and perform.active_job. Unfortunately no hook is instrumented around retries and discards of an Active Job prior to Rails 6.

Rails 6

Rails 6 has introduced hooks to Active Job around retries and discards to which one can easily subscribe using Active Support Instrumentation API to perform custom logging and monitoring or to collect any custom information.

The newly introduced hooks are enqueue_retry.active_job, retry_stopped.active_job and discard.active_job.

Let's discuss each of these hooks in detail.

Note that whenever we say a job, it means a job of type ActiveJob.

enqueue_retry.active_job

The enqueue_retry.active_job hook is instrumented when a job is enqueued to retry again due to occurrence of an exception which is configured using the retry_on method in the job's definition. This hook is triggered only when above condition is satisfied and the number of executions of the job is less than the number of attempts defined using the retry_on method. The number of attempts is by default set to 5 if not defined explicitly.

This is how we would subscribe to this hook and perform custom logging in our Rails application.

ActiveSupport::Notifications.subscribe "enqueue_retry.active_job" do |*args|
event = ActiveSupport::Notifications::Event.new *args
payload = event.payload
job = payload[:job]
error = payload[:error]
message = "#{job.class} (JID #{job.job_id})
with arguments #{job.arguments.join(', ')}
will be retried again in #{payload[:wait]}s
due to error '#{error.class} - #{error.message}'.
It is executed #{job.executions} times so far.".squish

BackgroundJob::Logger.log(message)
end

Note that the BackgroundJob::Logger above is our custom logger. If we want, we can add any other logic instead.

We will change the definition of Container::DeleteJob job as below.

class Container::DeleteJob < ActiveJob::Base
retry_on Timeout::Error, wait: 2.seconds, attempts: 3

def perform(container_id)
Container::DeleteService(container_id).process

    # Will raise Timeout::Error when the remote API is not responding.

end
end

Let's enqueue this job.

Container::DeleteJob.perform_now("container-1234")

Assume that this job keeps throwing Timeout::Error exception due to a network issue. The job will be retried twice since it is configured to retry when a Timeout::Error exception occurs up to maximum 3 attempts. While retrying this job, Active Job will instrument enqueue_retry.active_job hook along with the necessary job payload.

Since we have already subscribed to this hook, our subscriber would log something like this with the help of BackgroundJob::Logger.log.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 1 times so far.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 2 times so far.

retry_stopped.active_job

The retry_stopped.active_job hook is triggered when a job is retried up to the available number of attempts.

Let's see how this hook is triggered.

Along with the subscription for the enqueue_retry.active_job hook, let's subscribe to the retry_stopped.active_job hook, too.

ActiveSupport::Notifications.subscribe "retry_stopped.active_job" do |*args|
event = ActiveSupport::Notifications::Event.new *args
payload = event.payload
job = payload[:job]
error = payload[:error]
message = "Stopped processing #{job.class} (JID #{job.job_id})
further with arguments #{job.arguments.join(', ')}
since it failed due to '#{error.class} - #{error.message}' error
which reoccurred #{job.executions} times.".squish

BackgroundJob::Logger.log(message)
end

Let's keep the Container::DeleteJob job's definition unchanged and enqueue it again.

Container::DeleteJob.perform_now("container-1234")

We will assume that the job will keep throwing Timeout::Error exception due to a network issue.

In the logs recorded using BackgroundJob::Logger.log, we should see something like this.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 1 times so far.

Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) with arguments container-1234 will be retried again in 2s due to error 'Timeout::Error - Couldn't establish connection to cluster within 10s'. It is executed 2 times so far.

Stopped processing Container::DeleteJob (JID 770f4f2a-daa7-4c1e-be51-24adc04b4cb2) further with arguments container-1234 since it failed due to 'Timeout::Error - Couldn't establish connection to cluster within 10s' error which reoccurred 3 times.

Notice the last entry in the logs above and its order.

discard.active_job

The discard.active_job hook is triggered when a job's further execution is discarded due to occurrence of an exception which is configured using discard_on method.

To see how this hook is triggered, we will subscribe to the discard.active_job hook.

ActiveSupport::Notifications.subscribe "discard.active_job" do |*args|
event = ActiveSupport::Notifications::Event.new *args
payload = event.payload
job = payload[:job]
error = payload[:error]
message = "Discarded #{job.class} (JID #{job.job_id})
with arguments #{job.arguments.join(', ')}
due to '#{error.class} - #{error.message}' error.".squish

BackgroundJob::Logger.log(message)
end

We will configure our Container::DeleteJob job to discard when Container::NotFoundError exception occurs while executing the job.

class Container::DeleteJob < ActiveJob::Base
discard_on Container::NotFoundError
retry_on Timeout::Error, wait: 2.seconds, attempts: 3

def perform(container_id)
Container::DeleteService(container_id).process

    # Will raise Container::NotFoundError
    # if no container is found with 'container_id'.

    # Will raise Timeout::Error when the remote API is not responding.

end
end

Let's enqueue this job and assume that it would throw Container::NotFoundError exception.

Container::DeleteJob.perform_now("unknown-container-9876")

We should see following in the logs recorded by BackgroundJob::Logger.log.

Discarded Container::DeleteJob (JID e9b1cb5c-6d2d-49ae-b1d7-fef44f09ab8d) with arguments unknown-container-9876 due to 'Container::NotFoundError - Container 'unknown-container-9876' was not found' error.

Notes

1. These new hooks are also instrumented for the jobs which are enqueued using perform_later method since both perform_now and perform_later calls perform method under the hood.

2. Active Job already subscribes to the these hooks and writes them using the Rails' default logger.

3. If a block is provided to retry_on or discard_on methods then an applicable hook is instrumented first and then the given block is yielded.

Rails 5.2 does not support storing credentials of different environments with different encryption keys. If we want environment specific encrypted credentials, we'll have to follow this workaround.

Rails 6 has added support for Multi Environment credentials. With this change, credentials that belong to different environments can be stored in separate files with their own encryption key.

Let's see how this works in Rails 6.0.0.beta3

Rails 6.0.0.beta3

If we want to add credentials to be used in staging environment, we can run

rails credentials:edit --environment staging

This will create the credentials file config/credentials/staging.yml.enc and a staging specific encryption key config/credentials/staging.key and open the credentials file in your text editor.

Let's add our AWS access key id here.

aws:
  access_key_id: "STAGING_KEY"

We can then access the access_key_id in staging environment.

>> RAILS_ENV=staging rails c

pry(main)> Rails.application.credentials.aws[:access_key_id]

=> "STAGING_KEY"

Which takes precedence: Global or Environment Specific credentials?

Credentials added to global file config/credentials.yml.enc will not be loaded in environments which have their own environment specific credentials file (config/credentials/$environment.yml.enc).

So if we decide to add the following to the global credentials file, these credentials will not be available in staging. Since we already have a environment specific credentials file for staging.

aws:
  access_key_id: "DEFAULT_KEY"
stripe:
  secret_key: "DEFAULT_SECRET_KEY"

>> RAILS_ENV=staging rails c

pry(main)> Rails.application.credentials.aws[:access_key_id]

=> "STAGING_KEY"

pry(main)> Rails.application.credentials.stripe[:secret_key]

Traceback (most recent call last):
        1: from (irb):6
NoMethodError (undefined method `[]' for nil:NilClass)

Here is the relevant pull request

before? and after? are aliases to < (less than) and > (greater than) methods respectively.

Let's checkout how it works.

Rails 5.2

Let's try calling before? on a date object in Rails 5.2.

> > Date.new(2019, 3, 31).before?(Date.new(2019, 4, 1))

=> NoMethodError: undefined method 'before?' for Sun, 31 Mar 2019:Date
from (irb):1

> > Date.new(2019, 3, 31) < Date.new(2019, 4, 1)

=> true

Rails 6.0.0.beta2

Now, let's compare Date , DateTime , Time and ActiveSupport::TimeWithZone objects using before? and after? in Rails 6.

> > Date.new(2019, 3, 31).before?(Date.new(2019, 4, 1))

=> true

> > Date.new(2019, 3, 31).after?(Date.new(2019, 4, 1))

=> false

> > DateTime.parse('2019-03-31').before?(DateTime.parse('2019-04-01'))

=> true

> > DateTime.parse('2019-03-31').after?(DateTime.parse('2019-04-01'))

=> false

> > Time.parse('2019-03-31').before?(Time.parse('2019-04-01'))

=> true

> > Time.parse('2019-03-31').after?(Time.parse('2019-04-01'))

=> false

> > ActiveSupport::TimeWithZone.new(Time.utc(2019, 3, 31, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]).before?(ActiveSupport::TimeWithZone.new(Time.utc(2019, 4, 1, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]))

=> true

> > ActiveSupport::TimeWithZone.new(Time.utc(2019, 3, 31, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]).after?(ActiveSupport::TimeWithZone.new(Time.utc(2019, 4, 1, 12, 0, 0), ActiveSupport::TimeZone["Eastern Time (US & Canada)"]))

=> false

Here is the relevant pull request for adding before? and after? methods and the pull request for moving before? and after? to DateAndTime::Calculations.

extract! is different from reject! in the way that reject! returns the array after removing the elements whereas extract! returns removed elements from the array.

Let's checkout how it works.

Rails 6.0.0.beta2

Let's pluck all the user emails and then extract emails which include gmail.com.

> > emails = User.pluck(:email)
> > SELECT "users"."email" FROM "users"

=> ["amit.choudhary@bigbinary.com", "amit@gmail.com", "mark@gmail.com", "sam@gmail.com"]

> > emails.extract! { |email| email.include?('gmail.com') }

=> ["amit@gmail.com", "mark@gmail.com", "sam@gmail.com"]

> > emails

=> ["amit.choudhary@bigbinary.com"]

> > emails = User.pluck(:email)
> > SELECT "users"."email" FROM "users"

=> ["amit.choudhary@bigbinary.com", "amit@gmail.com", "mark@gmail.com", "sam@gmail.com"]

> > emails.reject! { |email| email.include?('gmail.com') }

=> ["amit.choudhary@bigbinary.com"]

> > emails

=> ["amit.choudhary@bigbinary.com"]

Here is the relevant pull request.

Before Rails 6, we can achieve this by calling map along with to_h.

index_with takes both value or a block as a parameter.

Let's checkout how it works.

Rails 5.2

Let's create a hash from an array in Rails 5.2 using map and to_h.

> > address = Address.first
> > SELECT "addresses".\* FROM "addresses"
> > ORDER BY "addresses"."id" ASC LIMIT \$1 [["LIMIT", 1]]

=> #<Address id: 1, first_name: "Amit", last_name: "Choudhary", state: "California", created_at: "2019-03-21 10:03:57", updated_at: "2019-03-21 10:03:57">

> > NAME_ATTRIBUTES = [:first_name, :last_name]

=> [:first_name, :last_name]

> > NAME_ATTRIBUTES.map { |attr| [attr, address.public_send(attr)] }.to_h

=> {:first_name=>"Amit", :last_name=>"Choudhary"}

Rails 6.0.0.beta2

Now let's create the same hash from the array using index_with in Rails 6.

> > address = Address.first
> > SELECT "addresses".\* FROM "addresses"
> > ORDER BY "addresses"."id" ASC LIMIT \$1 [["LIMIT", 1]]

=> #<Address id: 1, first_name: "Amit", last_name: "Choudhary", state: "California", created_at: "2019-03-21 10:02:47", updated_at: "2019-03-21 10:02:47">

> > NAME_ATTRIBUTES = [:first_name, :last_name]

=> [:first_name, :last_name]

> > NAME_ATTRIBUTES.index_with { |attr| address.public_send(attr) }

=> {:first_name=>"Amit", :last_name=>"Choudhary"}

> > NAME_ATTRIBUTES.index_with('Default')

=> {:first_name=>"Default", :last_name=>"Default"}

Here is the relevant pull request.

Let's checkout how it works.

Rails 6.0.0.beta2

Let's create two models named as Address and Order. Let's also delegate validate_state method in Order to Address.

class Address < ApplicationRecord
  validates :first_name, :last_name, :state, presence: true

  DELIVERABLE_STATES = ['New York']

  def validate_state
    unless DELIVERABLE_STATES.include?(state)
      errors.add(:state, :invalid)
    end
  end
end

class Order < ApplicationRecord
  belongs_to :address

  delegate :validate_state, to: :address
end

>> Order.first
SELECT "orders".* FROM "orders"
ORDER BY "orders"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<Order id: 1, amount: 0.1e2, address_id: 1, created_at: "2019-03-21 10:02:58", updated_at: "2019-03-21 10:17:44">

>> Address.first
SELECT "addresses".* FROM "addresses"
ORDER BY "addresses"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<Address id: 1, first_name: "Amit", last_name: "Choudhary", state: "California", created_at: "2019-03-21 10:02:47", updated_at: "2019-03-21 10:02:47">

>> Order.first.validate_state
SELECT "orders".* FROM "orders"
ORDER BY "orders"."id" ASC LIMIT $1  [["LIMIT", 1]]

SELECT "addresses".* FROM "addresses"
WHERE "addresses"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]

=> ["is invalid"]

Now, let's add private: true to the delegation.

class Order < ApplicationRecord
  belongs_to :address

  delegate :validate_state, to: :address, private: true
end

>> Order.first.validate_state
SELECT "orders".* FROM "orders"
ORDER BY "orders"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> Traceback (most recent call last):
        1: from (irb):7
NoMethodError (private method 'validate_state' called for #<Order:0x00007fb9d72fc1f8>
Did you mean?  validate)

As we can see, Rails now raises an exception of private method called if private option is set with delegate method.

Here is the relevant pull request.

Let's checkout how it works.

Rails 5.2

Let's create a table user reviews in Rails 5.2.

>> class CreateUserReviews < ActiveRecord::Migration[5.2]
>>   def change
>>     create_table 'user reviews' do |t|
>>       t.string :value
>>
>>       t.timestamps
>>     end
>>   end
>> end

=> :change

>> CreateUserReviews.new.change
-- create_table("user reviews")
CREATE TABLE "user"."reviews" ("id" bigserial primary key, "value" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

=> Traceback (most recent call last):
        2: from (irb):10
        1: from (irb):3:in 'change'
ActiveRecord::StatementInvalid (PG::InvalidSchemaName: ERROR:  schema "user" does not exist)
LINE 1: CREATE TABLE "user"."reviews" ("id" bigserial primary key, "...
                     ^
: CREATE TABLE "user"."reviews" ("id" bigserial primary key, "value" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

We can see that Rails 5.2 raised an exception and tried to create table named as reviews in user schema.

Rails 6.0.0.beta2

Now, let's create a table user reviews in Rails 6.

>> class CreateUserReviews < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table 'user reviews' do |t|
>>       t.string :value
>>
>>       t.timestamps
>>     end
>>   end
>> end

=> :change

>> CreateUserReviews.new.change
-- create_table("user reviews")
CREATE TABLE "user reviews" ("id" bigserial primary key, "value" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> #<PG::Result:0x00007f9d633c5458 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

Now, we can see that the SQL generated is correct and Rails successfully created a table named as user reviews.

Here is the relevant pull request.

Before Rails 6, we could use ActiveRecord::Base.connection.table_exists?.

Default value of if_not_exists option is false.

Rails 5.2

Let's create users table in Rails 5.2.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users)
CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

=> #<PG::Result:0x00007fd73e711cf0 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

Now let's try creating users table again with if_not_exists option.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users, if_not_exists: true do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>true})
CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

=> Traceback (most recent call last):
        2: from (irb):121
        1: from (irb):114:in 'change'
ActiveRecord::StatementInvalid (PG::DuplicateTable: ERROR:  relation "users" already exists)
: CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL)

We can see that Rails 5.2 ignored if_not_exists option and tried creating the table again.

Now let's try ActiveRecord::Base.connection.table_exists? with Rails 5.2.

>> class CreateUsers < ActiveRecord::Migration[5.2]
>>   def change
>>     unless ActiveRecord::Base.connection.table_exists?('users')
>>       create_table :users do |t|
>>         t.string :name
>>
>>         t.timestamps
>>       end
>>     end
>>   end
>> end

>> CreateUsers.new.change

=> nil

We can see that create_table :users never executed because ActiveRecord::Base.connection.table_exists?('users') returned true.

Rails 6.0.0.beta2

Let's create users table in Rails 6 with if_not_exists option set as true.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users, if_not_exists: true do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>true})
CREATE TABLE IF NOT EXISTS "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> #<PG::Result:0x00007fc4614fef48 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>true})
CREATE TABLE IF NOT EXISTS "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> #<PG::Result:0x00007fc46513fde0 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>

We can see that no exception was raised when we tried creating users table the second time.

Now let's see what happens if we set if_not_exists to false.

>> class CreateUsers < ActiveRecord::Migration[6.0]
>>   def change
>>     create_table :users, if_not_exists: false do |t|
>>       t.string :name, index: { unique: true }
>>
>>       t.timestamps
>>     end
>>   end
>> end

>> CreateUsers.new.change
-- create_table(:users, {:if_not_exists=>false})
CREATE TABLE "users" ("id" bigserial primary key, "name" character varying, "created_at" timestamp(6) NOT NULL, "updated_at" timestamp(6) NOT NULL)

=> Traceback (most recent call last):
        2: from (irb):23
        1: from (irb):15:in `change'
ActiveRecord::StatementInvalid (PG::DuplicateTable: ERROR:  relation "users" already exists
)

As we can see, Rails raised an exception here because if_not_exists was set to false.

Here is the relevant pull request.

Before Rails 6, it was not possible to change the contents of database.yml automatically. But now a command has been added to do this automatically.

Let's say our app has started with sqlite and now we have to switch to MySQL.

$ rails db:system:change --to=mysql
    conflict  config/database.yml
Overwrite /Users/prathamesh/Projects/reproductions/squish_app/config/database.yml? (enter "h" for help) [Ynaqdhm] Y
       force  config/database.yml
        gsub  Gemfile
        gsub  Gemfile

Our database.yml is now changed to contain the configuration for MySQL database and the Gemfile also gets updated automatically with addition of mysql2 gem in place of sqlite3.

This command also takes care of using proper gem versions in the Gemfile when the database backend is changed.

Though CI tools like CircleCi and Travis CI provide a feature to run tests in parallel, there still wasn't a straightforward way to parallelize tests on local before Rails 6.

Before Rails 6, if we wanted to parallelize tests, we would use Parallel Tests.

Rails 6 adds the parallelization of tests by default. Rails 6 added parallelize as a class method on ActiveSupport::TestCase which takes a hash as a parameter with the keys workers and with. The worker key is responsible for setting the number of parallel workers. The default value of the worker key is :number_of_processors, which finds the number of processors on the machine and sets it as the number of parallel workers. with takes two values - :processes, which is the default one, and :threads as a value.

Rails 6 also added two hooks - parallelize_setup, which is called before the processes are forked, and parallelize_teardown, which is called after the processes are killed. Rails 6 also handles creation of multiple databases and namespacing of those databases for parallel tests out of the box.

If we want to disable parallel testing, we can set the value of workers as 1 or less.

Rails 6.0.0.beta2

class ActiveSupport::TestCase
  parallelize_setup do |worker|
    # setup databases
  end
   parallelize_teardown do |worker|
    # cleanup database
  end

  # Run tests in parallel with specified workers
  parallelize(workers: :number_of_processors)

  # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
  fixtures :all

  # Add more helper methods to be used by all tests here...
end

Rails 6 also provides an environment variable PARALLEL_WORKERS to set the number of parallel workers on runtime.

$ PARALLEL_WORKERS=10 bin/rails test

Here is the relevant pull request for adding parallelize and pull request for setting number of processors as default workers count.

Before Rails 6

Before Rails 6, the subscriber of an event can access the event's start time, end time and the duration along with the other event information.

To demonstrate how to access this information from an event, we will instrument a custom event custom_sleep_event and attach a subscriber to that event.

ActiveSupport::Notifications.subscribe('custom_sleep_event') do |*args|
  event = ActiveSupport::Notifications::Event.new(*args)

  puts "Event: #{event.inspect}"
  puts "Started: #{event.time}"
  puts "Finished: #{event.end}"
  puts "Duration (ms): #{event.duration}"
end

ActiveSupport::Notifications.instrument('custom_sleep_event') do
  sleep 2
end

The event subscriber should print something similar.

Event: #<ActiveSupport::Notifications::Event:0x00007f952fc6a0b8 @name="custom_sleep_event", @payload={}, @time=2019-04-11 16:58:52 +0530, @transaction_id="e82231ab65b7af3c85ec", @end=2019-04-11 16:58:54 +0530, @children=[], @duration=nil>

Started: 2019-04-11 16:58:52 +0530

Finished: 2019-04-11 16:58:54 +0530

Duration (ms): 2001.287

Improvements and additions made to ActiveSupport::Notifications::Event in Rails 6

Rails 6 has improved the way an event's duration is computed and also added useful information accessible on an event object such as CPU time, idle time and allocations.

Let's discuss it in more detail.

1. CLOCK_MONOTONIC instead of CLOCK_REALTIME

Before Rails 6, Time.now is used for recording the event's start time and end time. To avoid issues with the machine changing time, Rails 6 now uses Concurrent.monotonic_time instead of Time.now to record the event's both start time and end time accurately.

Initially Process.clock_gettime(Process::CLOCK_MONOTONIC) was used which later modified to use Concurrent.monotonic_time. Note that Concurrent.monotonic_time is same but returns more precise time than Process.clock_gettime(Process::CLOCK_MONOTONIC).

Time.now or Process.clock_gettime(Process::CLOCK_REALTIME) can jump forwards and backwards as the system time-of-day clock is changed. Whereas, clock time using CLOCK_MONOTONIC returns the absolute wall-clock time since an unspecified time in the past (for example, system start-up time, or the Epoch). The CLOCK_MONOTONIC does not change with the system time-of-day clock, it just keeps advances forwards at one tick per tick and resets if the system is rebooted. In general, CLOCK_MONOTONIC is recommended to compute the elapsed time between two events. To read more about the differences between CLOCK_REALTIME and CLOCK_MONOTONIC, please check the discussion on [this Stackoverflow thread](https://stackoverflow.com/questions/3523442/> difference-between-clock-realtime-and-clock-monotonic). Another article written by Luca Guidi on the same topic is a recommended read.

2. No need to create hand made event objects on our own

Since it is a common practice to initialize an event using ActiveSupport::Notifications::Event.new(*args) in the event subscriber block, Rails 6 now makes this a bit easy. If the block passed to the subscriber only takes one argument then the Active Support Notification framework now yields an event object to the block.

Therefore, the subscriber definition below

ActiveSupport::Notifications.subscribe('an_event') do |*args|
  event = ActiveSupport::Notifications::Event.new(*args)

  puts "Event #{event.name} received."
end

now can be simplified in Rails 6 as follows.

ActiveSupport::Notifications.subscribe('an_event') do |event|
  puts "Event #{event.name} received."
end

3. CPU time and idle time

Rails 6 now computes elapsed CPU time of an event with the help of Process.clock_gettime(Process::CLOCK_PROCESS_CPUTIME_ID).

System (kernel) keeps track of CPU time per process. The clock time returned using CLOCK_PROCESS_CPUTIME_ID represents the CPU time that has passed since the process started. Since a process may not always get all CPU cycles between start and finish of the process, the process often has to (sleep and) share CPU time among other processes. If the system puts a process to sleep, then the time spend waiting is not counted in the process' CPU time.

The CPU time of an event can be fetched using the #cpu_time method.

Also, Rails 6 now computes the idle time of an event, too. The idle time of an event represents the difference between the event's #duration and #cpu_time. Note that the #duration is computed using the difference between the event's monotonic time at the start (#time) and the monotonic time at the end (#end).

Let's see how to get these time values.

ActiveSupport::Notifications.subscribe('custom_sleep_event') do |event|
  puts "Event: #{event.inspect}"
  puts "Started: #{event.time}"
  puts "Finished: #{event.end}"
  puts "Duration (ms): #{event.duration}"
  puts "CPU time (ms): #{event.cpu_time}"
  puts "Idle time (ms): #{event.idle_time}"
end

ActiveSupport::Notifications.instrument('custom_sleep_event') do
  sleep 2
end

It prints this.

Event: #<ActiveSupport::Notifications::Event:0x00007fb02ac72400 @name="custom_sleep_event", @payload={}, @time=29514.525707, @transaction_id="43ca8e1c378b6b00d861", @end=29516.528971, @children=[], @duration=nil, @cpu_time_start=2.238801, @cpu_time_finish=2.238874, @allocation_count_start=835821, @allocation_count_finish=835821>

Started: 29514.525707

Finished: 29516.528971

Duration (ms): 2003.2639999990351

CPU time (ms): 0.07299999999998974

Idle time (ms): 2003.190999999035

Notice the @cpu_time_start and @cpu_time_finish counters in the inspected event object representation which are used to calculate the CPU time.

4. Allocations

We will now know how many objects were allocated between the start and end of an event using event's #allocations method.

ActiveSupport::Notifications.subscribe('custom_sleep_event') do |event|
  puts "Event: #{event.inspect}"
  puts "Started: #{event.time}"
  puts "Finished: #{event.end}"
  puts "Duration (ms): #{event.duration}"
  puts "CPU time (ms): #{event.cpu_time}"
  puts "Idle time (ms): #{event.idle_time}"
  puts "# of objects allocated: #{event.allocations}"
end

ActiveSupport::Notifications.instrument('custom_sleep_event') do
  sleep 2
end

The above example should print something like this.

Event: #<ActiveSupport::Notifications::Event:0x00007fed8c4e33c0 @name="custom_sleep_event", @payload={}, @time=30503.508897, @transaction_id="5330165dae2b49fbe143", @end=30505.513547, @children=[], @duration=nil, @cpu_time_start=2.813231, @cpu_time_finish=2.813404, @allocation_count_start=834227, @allocation_count_finish=834228>

Started: 30503.508897

Finished: 30505.513547

Duration (ms): 2004.6499999989464

CPU time (ms): 0.17299999999975668

Idle time (ms): 2004.4769999989467

# of objects allocated: 1

Notice the @allocation_count_finish and @allocation_count_start counters in the inspected event object representation which are used to calculate the number of objects allocated during an event whose difference is (834228 - 834227 = 1).

In case of JRuby, the allocations would be zero.

5. #start! and #finish!

Two public methods #start! and #finish! have been introduced to ActiveSupport::Notifications::Event which can be used to record more information.

The #start! method resets the @time, @cpu_time_start and @allocation_count_start counters. Similarly, the #finish! method also resets the @end, @cpu_time_finish and @allocation_count_finish counters.

To learn more about this feature, please check rails/rails#33449.

Before Rails 6, has_secure_password did not accept any attribute as a parameter. So, if we needed BCrypt encryption on a different column other than password_digest, we would have to manually encrypt the value before storing it.

Rails 6 makes it easy and allows custom attributes as a parameter to has_secure_password. has_secure_password still defaults to password so it works with previous versions of Rails. has_secure_password still needs the column named column_name_digest defined on the model.

has_secure_password also adds the authenticate_column_name method to authenticate the custom column.

Let's check out how it works.

Rails 5.2

>> class User < ApplicationRecord
>>   has_secure_password
>> end

=> [ActiveModel::Validations::ConfirmationValidator]

>> user = User.create(email: 'amit.choudhary@bigbinary.com', password: 'amit.choudhary')
BEGIN
User Create (0.8ms)  INSERT INTO "users" ("email", "password_digest", "created_at", "updated_at") VALUES ($1, $2, $3, $4) RETURNING "id"  [["email", "amit.choudhary@bigbinary.com"], ["password_digest", "$2a$10$g6ZJNgakn4I1w/qjAx3vM.I76QSNjFCHtTtT9ovko/9Th50SEmIBO"], ["created_at", "2019-03-17 23:30:13.754379"], ["updated_at", "2019-03-17 23:30:13.754379"]]
COMMIT

=> #<User id: 1, email: "amit.choudhary@bigbinary.com", password_digest: "$2a$10$g6ZJNgakn4I1w/qjAx3vM.I76QSNjFCHtTtT9ovko/9...", created_at: "2019-03-17 23:30:13", updated_at: "2019-03-17 23:30:13">

>> user.authenticate('amit.choudhary')

=> #<User id: 1, email: "amit.choudhary@bigbinary.com", password_digest: "$2a$10$g6ZJNgakn4I1w/qjAx3vM.I76QSNjFCHtTtT9ovko/9...", created_at: "2019-03-17 23:30:13", updated_at: "2019-03-17 23:30:13">

>> class User < ApplicationRecord
>>   has_secure_password :transaction_password
>> end

=> NoMethodError: undefined method 'fetch' for :transaction_password:Symbol
	from (irb):9:in '<class:User>'
	from (irb):8

Rails 6.0.0.beta2

>> class User < ApplicationRecord
>>   has_secure_password
>>   has_secure_password :transaction_password
>> end

=> [ActiveModel::Validations::ConfirmationValidator]

>> user = User.create(email: 'amit.choudhary@bigbinary.com', password: 'amit.choudhary', transaction_password: 'amit.choudhary')
BEGIN
User Create (0.5ms)  INSERT INTO "users" ("email", "password_digest", "transaction_password_digest", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["email", "amit.choudhary@bigbinary.com"], ["password_digest", "$2a$10$nUiO7E2XrIJx/sSdpG0JAOL00uFvPRH7kXHLk5f/6qA1zLPHIrpPy"], ["transaction_password_digest", "$2a$10$l6cTpHwV9xOEn2.OumI29OnualGpvr1CgrNrbuMuHyGTltko8eBG2"], ["created_at", "2019-03-17 23:42:28.723431"], ["updated_at", "2019-03-17 23:42:28.723431"]]
COMMIT

=> #<User id: 5, email: "amit.choudhary@bigbinary.com", password_digest: [FILTERED], transaction_password_digest: [FILTERED], created_at: "2019-03-17 23:42:28", updated_at: "2019-03-17 23:42:28">

>> user.authenticate('amit.choudhary')

=> #<User id: 5, email: "amit.choudhary@bigbinary.com", password_digest: [FILTERED], transaction_password_digest: [FILTERED], created_at: "2019-03-17 23:42:28", updated_at: "2019-03-17 23:42:28">

>> user.authenticate_transaction_password('amit.choudhary')

=> #<User id: 5, email: "amit.choudhary@bigbinary.com", password_digest: [FILTERED], transaction_password_digest: [FILTERED], created_at: "2019-03-17 23:42:28", updated_at: "2019-03-17 23:42:28">

Here is the relevant pull request.

Before Rails 6, the default format %{attribute} %{message} is used to display validation error message for a model's attribute.

> > article = Article.new
> > => #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
> > article.errors.full_message(:title, "cannot be blank")
> > => "Title cannot be blank"
> >

The default format can be overridden globally using a language-specific locale file.

# config/locales/en.yml

en:
errors:
format:
"'%{attribute}' %{message}"

With this change, the full error message is changed for all the attributes of all models.

> > article = Article.new
> > => #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
> > article.errors.full_message(:title, "cannot be blank")
> > => "'Title' cannot be blank"

> > user = User.new
> > => #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
> > user.errors.full_message(:first_name, "cannot be blank")
> > => "'First name' cannot be blank"
> >

This trick works in some cases but it doesn't work if we have to customize the error messages on the basis of specific models or attributes.

Before Rails 6, there is no easy way to generate error messages like shown below.

The article's title cannot be empty

or

First name of a person cannot be blank

If we change the errors.format to The article's %{attribute} %{message} in config/locales/en.yml then that format will be unexpectedly used for other models, too.

# config/locales/en.yml

en:
errors:
format:
"The article's %{attribute} %{message}"

This is what will happen if we make such a change.

> > article = Article.new
> > => #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
> > article.errors.full_message(:title, "cannot be empty")
> > => "The article's Title cannot be empty"

> > user = User.new
> > => #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
> > user.errors.full_message(:first_name, "cannot be blank")
> > => "The article's First name cannot be blank"
> >

Notice the error message generated for the :first_name attribute of User model. This does not look the way we want, right?

Let's see what is changed in Rails 6 to overcome this problem.

Enhancements made to ActiveModel::Errors#full_message in Rails 6

Overriding the format of error message globally using errors.format is still supported in Rails 6.

In addition to that, Rails 6 now also supports overriding the error message's format at the model level and at the attribute level.

In order to enable this support, we need to explicitly set config.active_model.i18n_customize_full_message to true in the Rails configuration file, preferably in config/application.rb which is implicitly set to false by default.

Overriding model level format

We can customize the full error message format for each model separately.

# config/locales/en.yml

en:
activerecord:
errors:
models:
article:
format: "`%{attribute}`: %{message}"
user:
format: "%{attribute} of the user %{message}"

The full error messages will look like this.

> > article = Article.new
> > => #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
> > article.errors.full_message(:title, "cannot be empty")
> > => "`Title`: cannot be empty"
> > article.valid?
> > => false
> > article.errors.full_messages
> > => ["`Title`: can't be blank"]

> > user = User.new
> > => #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
> > user.errors.full_message(:first_name, "cannot be blank")
> > => "First name of the user cannot be blank"

> > comment = Comment.new
> > => #<Comment id: nil, message: nil, author_id: nil, created_at: nil, updated_at: nil>
> > comment.errors.full_message(:message, "is required")
> > => "Message is required"
> >

Notice how the default format %{attribute} %{message} is used for generating the full error messages for the Comment model since its format is not being overridden.

Since the other methods such as ActiveModel::Errors#full_messages, ActiveModel::Errors#full_messages_for, ActiveModel::Errors#to_hash etc. use the ActiveModel::Errors#full_message method under the hood, we get the full error messages according to the custom format in the returned values of these methods respectively as expected.

Overriding attribute level format

Similar to customizing format at the model level, we can customize the error format for specific attributes of individual models.

# config/locales/en.yml

en:
activerecord:
errors:
models:
article:
attributes:
title:
format: "The article's title %{message}"
user:
attributes:
first_name:
format: "%{attribute} of a person %{message}"

With such a configuration, we get the customized error message for the title attribute of the Article model.

> > article = Article.new
> > => #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
> > article.errors.full_message(:title, "cannot be empty")
> > => "The article's title cannot be empty"
> > article.errors.full_message(:description, "cannot be empty")
> > => "Description cannot be empty"

> > user = User.new
> > => #<User id: nil, first_name: nil, last_name: nil, country: nil, created_at: nil, updated_at: nil>
> > user.errors.full_message(:first_name, "cannot be blank")
> > => "First name of a person cannot be blank"
> > user.errors.full_message(:last_name, "cannot be blank")
> > => "Last name cannot be blank"
> >

Note that the error messages for the rest of the attributes were generated using the default %{attribute} %{message} format for which we didn't add custom formats in the config/locales/en.yml manifest.

Overriding model level format of deeply nested models

# config/locales/en.yml

en:
activerecord:
errors:
models:
article/comments/attachments:
format: "%{message}"

> > article = Article.new
> > => #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
> > article.errors.full_message(:'comments/attachments.file_name', "is required")
> > => "is required"
> > article.errors.full_message(:'comments/attachments.path', "cannot be blank")
> > => "cannot be blank"
> > article.errors.full_message(:'comments.message', "cannot be blank")
> > => "Comments message cannot be blank"
> >

Overriding attribute level format of deeply nested models

# config/locales/en.yml

en:
activerecord:
errors:
models:
article/comments/attachments:
attributes:
file_name:
format: "File name of an attachment %{message}"

> > article = Article.new
> > => #<Article id: nil, title: nil, description: nil, created_at: nil, updated_at: nil>
> > article.errors.full_message(:'comments/attachments.file_name', "is required")
> > => "File name of an attachment is required"
> > article.errors.full_message(:'comments/attachments.path', "cannot be blank")
> > => "Comments/attachments path cannot be blank"
> >

Precedence

The custom formats specified in the locale file has the following precedence in the high to low order.

• activerecord.errors.models.article/comments/attachments.attributes.file_name.format
• activerecord.errors.models.article/comments/attachments.format
• activerecord.errors.models.article.attributes.title.format
• activerecord.errors.models.article.format
• errors.format

To learn more, please checkout rails/rails#32956 and rails/rails#35789.

For example, we want to fetch subscriptions of some users. The query would look as shown below.

Rails 5.2

User.where(blocked: false).preload(:subscriptions).collect(&:subscriptions)

=> # returns collection of subscription records

ActiveRecord::Relation#extract_associated provides a shorthand to achieve same result and is more readable than former.

Rails 6.0.0.beta3

User.where(blocked: false).extract_associated(:subscriptions)

=> # returns the same collection of subscription records

Here's the relevant pull request for this change.

This has impact on methods like first , last and many more where implicit ordering is used.

Let's checkout how it works.

Rails 5.2

>> class User < ApplicationRecord
>>   validates :name, presence: true
>> end

=> {:presence=>true}

>> User.first
SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 1, name: "Amit", created_at: "2019-03-11 00:18:41", updated_at: "2019-03-11 00:18:41">

>> User.last
SELECT "users".* FROM "users" ORDER BY "users"."id" DESC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 2, name: "Mark", created_at: "2019-03-11 00:20:42", updated_at: "2019-03-11 00:20:42">

>> class User < ApplicationRecord
>>   validates :name, presence: true
>>   self.implicit_order_column = "updated_at"
>> end

=> Traceback (most recent call last):
        2: from (irb):10
        1: from (irb):12:in '<class:User>'
NoMethodError (undefined method 'implicit_order_column=' for #<Class:0x00007faf4d6cb408>)

Rails 6.0.0.beta2

>> class User < ApplicationRecord
>>   validates :name, presence: true
>> end

=> {:presence=>true}

>> User.first
SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 1, name: "Amit", created_at: "2019-03-11 00:18:41", updated_at: "2019-03-11 00:18:41">

>> User.last
SELECT "users".* FROM "users" ORDER BY "users"."id" DESC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 2, name: "Mark", created_at: "2019-03-11 00:20:42", updated_at: "2019-03-11 00:20:42">

>> class User < ApplicationRecord
>>   validates :name, presence: true
>>   self.implicit_order_column = "updated_at"
>> end

=> "updated_at"

>> User.find(1).touch
SELECT "users".* FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]
UPDATE "users" SET "updated_at" = $1 WHERE "users"."id" = $2  [["updated_at", "2019-03-11 00:23:33.369021"], ["id", 1]]

=> true

>> User.first
SELECT "users".* FROM "users" ORDER BY "users"."updated_at" ASC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 2, name: "Mark", created_at: "2019-03-11 00:20:42", updated_at: "2019-03-11 00:23:09">

>> User.last
SELECT "users".* FROM "users" ORDER BY "users"."updated_at" DESC LIMIT $1  [["LIMIT", 1]]

=> #<User id: 1, name: "Amit", created_at: "2019-03-11 00:18:41", updated_at: "2019-03-11 00:23:33">

Here is the relevant pull request.

Bulk inserts can be performed using newly added methods: insert_all, insert_all! and upsert_all.

All of these new methods allow the insertion of multiple records of the same model into the database. A single INSERT SQL query is prepared by these methods and a single sql statement is sent to the database, without instantiating the model or invoking Active Record callbacks or validations.

During bulk insertion, violation of primary key, violation of unique indexes, and violation of unique constraints is possible. Rails leverages database-specific features to either skip, or upsert the duplicates, depending on the case.

Let's discuss insert_all, insert_all! and upsert_all methods in detail, which are all used to perform bulk insert.

We will create an articles table with two unique indexes.

create_table :articles do |t|
  t.string :title, null: false
  t.string :slug, null: false
  t.string :author, null: false
  t.text :description

  t.index :slug, unique: true
  t.index [:title, :author], unique: true
end

Note that we do not allow duplicate slug columns. We also prevent records from having duplicate title and author columns together.

To try out the examples provided in this blog post, please ensure to always clean up the articles table before running any example.

1. Performing bulk inserts by skipping duplicates

Let's say we want to insert multiple articles at once into the database. It is possible that certain records may violate the unique constraint(s) of the table. Such records are considered duplicates.

In other words, rows or records are determined to be unique by every unique index on the table by default.

To skip the duplicate rows or records, and insert the rest of the records at once, we can use ActiveRecord::Persistence#insert_all method.

1.1 Behavior with PostgreSQL

Let's run the following example on a PostgreSQL database.

result = Article.insert_all(
  [
    { id: 1,
      title: 'Handling 1M Requests Per Second',
      author: 'John',
      slug: '1m-req-per-second' },

    { id: 1, # duplicate 'id' here
      title: 'Type Safety in Elm',
      author: 'George',
      slug: 'elm-type-safety' },

    { id: 2,
      title: 'Authentication with Devise - Part 1',
      author: 'Laura',
      slug: 'devise-auth-1' },

    { id: 3,
      title: 'Authentication with Devise - Part 1',
      author: 'Laura', # duplicate 'title' & 'author' here
      slug: 'devise-auth-2' },

    { id: 4,
      title: 'Dockerizing and Deploying Rails App to Kubernetes',
      author: 'Paul',
      slug: 'rails-on-k8s' },

    { id: 5,
      title: 'Elm on Rails',
      author: 'Amanda',
      slug: '1m-req-per-second' }, # duplicate 'slug' here

    { id: 6,
      title: 'Working Remotely',
      author: 'Greg',
      slug: 'working-remotely' }
  ]
)
# Bulk Insert (2.3ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests  [...snip...] 'working-remotely') ON CONFLICT  DO NOTHING RETURNING "id"

puts result.inspect
#<ActiveRecord::Result:0x00007fb6612a1ad8 @columns=["id"], @rows=[[1], [2], [4], [6]], @hash_rows=nil, @column_types={"id"=>#<ActiveModel::Type::Integer:0x00007fb65f420078 @precision=nil, @scale=nil, @limit=8, @range=-9223372036854775808...9223372036854775808>}>

puts Article.count
# 4

The insert_all method accepts a mandatory argument which should be an array of hashes with the attributes of the same model. The keys in all hashes should be same.

Notice the ON CONFLICT DO NOTHING clause in the INSERT query. This clause is supported by PostgreSQL and SQLite databases. This instructs the database that when there is a conflict or a unique key constraint violation during bulk insert operation, to skip the conflicting record silently and proceed with the insertion of the next record.

In the above example, we have exactly 3 records which violate various unique constraints defined on the articles table.

One of the records being inserted has a duplicate id: 1 attribute, which violates unique primary key constraint. Another record that has duplicate title: 'Authentication with Devise - Part 1', author: 'Laura' attributes violates the multi-column unique index defined on title and author columns. Another record has duplicate slug: '1m-req-per-second' attributes violates the unique index defined on the slug column.

All of these records that violate any unique constraint or unique index are skipped and are not inserted into the database.

If successful, ActiveRecord::Persistence#insert_all returns an instance of ActiveRecord::Result. The contents of the result vary per database. In case of PostgreSQL database, this result instance holds information about the successfully inserted records such as the chosen column names, values of the those columns in each successfully inserted row, etc.

For PostgreSQL, by default, insert_all method appends RETURNING "id" clause to the SQL query where id is the primary key(s). This clause instructs the database to return the id of every successfully inserted record. By inspecting the result, especially the @columns=["id"], @rows=[[1], [2], [4], [6]] attributes of the result instance, we can see that the records having id attribute with values 1, 2, 4 and 6 were successfully inserted.

What if we want to see more attributes and not just the id attribute of the successfully inserted records in the result?

We should use the optional returning option, which accepts an array of attribute names, which should be returned for all successfully inserted records!

result = Article.insert_all(
  [
    { id: 1,
      title: 'Handling 1M Requests Per Second',
      author: 'John',
      slug: '1m-req-per-second' },
    #...snip...
  ],
  returning: %w[ id title ]
)
# Bulk Insert (2.3ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests  [...snip...] 'working-remotely') ON CONFLICT  DO NOTHING RETURNING "id","title"

puts result.inspect
#<ActiveRecord::Result:0x00007f902a1196f0 @columns=["id", "title"], @rows=[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]], @hash_rows=nil, @column_types={"id"=>#<ActiveModel::Type::Integer:0x00007f90290ca8d0 @precision=nil, @scale=nil, @limit=8, @range=-9223372036854775808...9223372036854775808>, "title"=>#<ActiveModel::Type::String:0x00007f9029978298 @precision=nil, @scale=nil, @limit=nil>}>

puts result.pluck("id", "title").inspect
#[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]]

Notice how the INSERT query appends RETURNING "id","title" clause and the result now holds the id and title attributes of the successfully inserted records.

1.2 Behavior with SQLite

Similar to PostgreSQL, the violating records are skipped during the bulk insert operation performed using insert_all when we run our example on a SQLite database.

result = Article.insert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    #...snip...
  ]
)
# Bulk Insert (1.6ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests [...snip...] 'working-remotely') ON CONFLICT  DO NOTHING

puts result.inspect
#<ActiveRecord::Result:0x00007fa9df448ff0 @columns=[], @rows=[], @hash_rows=nil, @column_types={}>

puts Article.pluck(:id, :title)
#[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]]

puts Article.count
# 4

Note that since SQLite does not support RETURING clause, it is not being added to the SQL query. Therefore, the returned ActiveRecord::Result instance does not contain any useful information.

If we try to explicitly use the returning option when the database being used is SQLite, then the insert_all method throws an error.

Article.insert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    #...snip...
  ],
  returning: %w[ id title ]
)
# ActiveRecord::ConnectionAdapters::SQLite3Adapter does not support :returning (ArgumentError)

1.3 Behavior with MySQL

The records that violate primary key, unique key constraints, or unique indexes are skipped during bulk insert operation performed using insert_all on a MySQL database.

result = Article.insert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    #...snip...
  ]
)
# Bulk Insert (20.3ms)  INSERT INTO `articles`(`id`,`title`,`author`,`slug`) VALUES (1, 'Handling 1M Requests [...snip...] 'working-remotely') ON DUPLICATE KEY UPDATE `id`=`id`

puts result.inspect
#<ActiveRecord::Result:0x000055d6cfea7580 @columns=[], @rows=[], @hash_rows=nil, @column_types={}>

puts Article.pluck(:id, :title)
#[[1, "Handling 1M Requests Per Second"], [2, "Authentication with Devise - Part 1"], [4, "Dockerizing and Deploying Rails App to Kubernetes"], [6, "Working Remotely"]]

puts Article.count
# 4

Here, the ON DUPLICATE KEY UPDATE 'id'='id' clause in the INSERT query is essentially doing the same thing as the ON CONFLICT DO NOTHING clause supported by PostgreSQL and SQLite.

Since MySQL does not support RETURING clause, it is not being included in the SQL query and therefore, the result doesn't contain any useful information.

Explicitly trying to use returning option with insert_all method on a MySQL database throws ActiveRecord::ConnectionAdapters::Mysql2Adapter does not support :returning error.

2. Performing bulk inserts by skipping duplicates on a specified unique constraint but raising exception if records violate other unique constraints

In the previous case, we were skipping the records that were violating any unique constraints. In some case, we may want to skip duplicates caused by only a specific unique index but abort transaction if the other records violate any other unique constraints.

The optional unique_by option of the insert_all method allows to define such a unique constraint.

2.1 Behavior with PostgreSQL and SQLite

Let's see an example to skip duplicate records that violate only the specified unique index :index_articles_on_title_and_author using unique_by option. The duplicate records that do not violate index_articles_on_title_and_author index are not skipped, and therefore throw an error.

result = Article.insert_all(
  [
    { .... },
    { .... }, # duplicate 'id' here
    { .... },
    { .... }, # duplicate 'title' and 'author' here
    { .... },
    { .... }, # duplicate 'slug' here
    { .... }
  ],
  unique_by: :index_articles_on_title_and_author
)
# PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "articles_pkey" (ActiveRecord::RecordNotUnique)
# DETAIL:  Key (id)=(1) already exists.

In case of SQLite, the error appears as shown below.

# SQLite3::ConstraintException: UNIQUE constraint failed: articles.id (ActiveRecord::RecordNotUnique)

In this case we get ActiveRecord::RecordNotUnique error which is caused by the violation of primary key constraint on the id column.

It didn't skip the second record in the example above which violated the unique index on primary key id since the unique_by option was specified with a different unique index.

When an exception occurs, no record persists to the database since insert_all executes just a single SQL query.

The unique_by option can be identified by columns or a unique index name.

~~~ruby unique_by: :index_articles_on_title_and_author

is same as

unique_by: %i[ title author ]

Also,

unique_by: :slug

is same as

unique_by: %i[ :slug ]

and also same as

unique_by: :index_articles_on_slug

Let's remove (or fix) the record that has duplicate primary key
and
re-run the above example.

~~~ruby
result = Article.insert_all(
  [
    { .... },
    { .... },
    { .... },
    { .... }, # duplicate 'title' and 'author' here
    { .... },
    { .... }, # duplicate 'slug' here
    { .... }
  ],
  unique_by: :index_articles_on_title_and_author
)
# PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_articles_on_slug" (ActiveRecord::RecordNotUnique)
# DETAIL:  Key (slug)=(1m-req-per-second) already exists.

In case of SQLite, the error looks appears as shown below.

# SQLite3::ConstraintException: UNIQUE constraint failed: articles.slug (ActiveRecord::RecordNotUnique)

The ActiveRecord::RecordNotUnique error in the example above now says the index_articles_on_slug unique constraint is violated. Note how it intentionally didn't raise an error for the unique constraint violated on the title and author columns by the fourth record in the examples above.

Now we will remove (or fix) the record that has same slug.

result = Article.insert_all(
  [
    { .... },
    { .... },
    { .... },
    { .... }, # duplicate 'title' and 'author' here
    { .... },
    { .... },
    { .... }
  ],
  unique_by: :index_articles_on_title_and_author
)
# Bulk Insert (2.5ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests Per Second', [...snip...] 'working-remotely') ON CONFLICT ("title","author") DO NOTHING RETURNING "id"

puts result.inspect
#<ActiveRecord::Result:0x00007fada2069828 @columns=["id"], @rows=[[1], [7], [2], [4], [5], [6]], @hash_rows=nil, @column_types={"id"=>#<ActiveModel::Type::Integer:0x00007fad9fdb9df0 @precision=nil, @scale=nil, @limit=8, @range=-9223372036854775808...9223372036854775808>}>

Here, the fourth record was skipped since that record violates the unique index index_articles_on_title_and_author specified by the unique_by option.

Similarly, we can specify a different unique index using the unique_by option. For example, if we specify unique_by: :slug option then the records containing duplicate slug columns will be skipped, but would raise ActiveRecord::RecordNotUnique exception if any record violates other unique constraints.

2.2 Behavior with MySQL

The unique_by option is not supported when the database is MySQL.

3. Raising exception if any of the records being bulk inserted violate any unique constraints

The insert_all! method (with bang version) never skips a duplicate record. If a record violates any unique constraints, then insert_all! method would simply throw an ActiveRecord::RecordNotUnique error.

When database is PostgreSQL, insert_all! method can accept optional returning option, which we discussed in depth in 1.1 section above.

The unique_by option is not supported by the insert_all! method.

4. Performing bulk upserts (updates or inserts)

So far, in the sections 1, 2 and 3 above, we discussed either skipping the duplicates or raising an exception if a duplicate is encountered during bulk inserts. Sometimes, we want to update the existing record when a duplicate occurs otherwise insert a new record. This operation is called upsert because either it tries to update the record, or if there is no record to update, then it tries to insert.

The upsert_all method in Rails 6 allows performing bulk upserts.

Let's see it's usage and behavior with different database systems.

4.1 upsert_all in MySQL

Let's try to bulk upsert multiple articles containing some duplicates.

result = Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, .... }, # duplicate 'id' here
    { id: 2, .... },
    { id: 3, .... }, # duplicate 'title' and 'author' here
    { id: 4, .... },
    { id: 5, .... }, # duplicate 'slug' here
    { id: 6, .... }
  ]
)
# Bulk Insert (26.3ms)  INSERT INTO `articles`(`id`,`title`,`author`,`slug`) VALUES (1, 'Handling 1M Requests Per Second', 'John', [...snip...] 'working-remotely') ON DUPLICATE KEY UPDATE `title`=VALUES(`title`),`author`=VALUES(`author`),`slug`=VALUES(`slug`)

puts result.inspect
#<ActiveRecord::Result:0x000055a43c1fae10 @columns=[], @rows=[], @hash_rows=nil, @column_types={}>

puts Article.count
# 5

puts Article.all
#<ActiveRecord::Relation [#<Article id: 1, title: "Type Safety in Elm", slug: "elm-type-safety", author: "George", description: nil>, #<Article id: 2, title: "Authentication with Devise - Part 1", slug: "devise-auth-2", author: "Laura", description: nil>, #<Article id: 4, title: "Dockerizing and Deploying Rails App to Kubernetes", slug: "rails-on-k8s", author: "Paul", description: nil>, #<Article id: 5, title: "Elm on Rails", slug: "1m-req-per-second", author: "Amanda", description: nil>, #<Article id: 6, title: "Working Remotely", slug: "working-remotely", author: "Greg", description: nil>]>

The persisted records in the database look exactly as intended. Let's discuss it in detail.

The second row in the input array that has the id: 1 attribute replaced the first row, which also had the duplicate id: 1 attribute.

The fourth row that has id: 3 replaced the attributes of the third row since both had duplicate "title" and "author" attributes.

The rest of the rows were not duplicates or no longer became duplicates, and therefore were inserted without any issues.

Note that the returning and unique_by options are not supported in the upsert_all method when the database is MySQL.

4.2 upsert_all in SQLite

Let's try to execute the same example from the above section 4.1 when database is SQLite.

result = Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
    { id: 4, title: 'Dockerizing and Deploying Rails App to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ]
)
# Bulk Insert (4.0ms)  INSERT INTO "articles"("id","title","author","slug") VALUES (1, 'Handling 1M Requests Per Second', [...snip...] 'working-remotely') ON CONFLICT ("id") DO UPDATE SET "title"=excluded."title","author"=excluded."author","slug"=excluded."slug"

# SQLite3::ConstraintException: UNIQUE constraint failed: articles.title, articles.author (ActiveRecord::RecordNotUnique)

The bulk upsert operation failed in the above example due to ActiveRecord::RecordNotUnique exception.

Why it didn't work similar to MySQL?

As per the documentation of MySQL, an upsert operation takes place if a new record violates any unique constraint.

Whereas, in case of SQLite, by default, new record replaces existing record when both the existing and new record have the same primary key. If a record violates any other unique constraints other than the primary key, it then raises ActiveRecord::RecordNotUnique exception.

The ON CONFLICT ("id") DO UPDATE clause in the SQL query above conveys the same intent.

Therefore, upsert_all in SQLite doesn't behave exactly same as in MySQL.

As a workaround, we will need to upsert records with the help of multiple upsert_all calls with the usage of unique_by option.

If a duplicate record is encountered during the upsert operation, which violates the unique index specified using unique_by option then it will replace the attributes of the existing matching record.

Let's try to understand this workaround with another example.

Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
  ],
  unique_by: :id
)

Article.upsert_all(
  [
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
    { id: 4, title: 'Dockerizing and Deploying Rails App to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ],
  unique_by: %i[ title author ]
)

puts Article.count
# 5

puts Article.all
#<ActiveRecord::Relation [#<Article id: 1, title: "Type Safety in Elm", slug: "elm-type-safety", author: "George", description: nil>, #<Article id: 2, title: "Authentication with Devise - Part 1", slug: "devise-auth-2", author: "Laura", description: nil>, #<Article id: 4, title: "Dockerizing and Deploying Rails App to Kubernetes", slug: "rails-on-k8s", author: "Paul", description: nil>, #<Article id: 5, title: "Elm on Rails", slug: "1m-req-per-second", author: "Amanda", description: nil>, #<Article id: 6, title: "Working Remotely", slug: "working-remotely", author: "Greg", description: nil>]>

Here, we first tried to upsert all the records which violated the unique primary key index on id column. Later, we upsert successfully all the remaining records, which violated the unique index on the title and author columns.

Note that since the first record's slug attribute was already replaced with the second record's slug attribute; the last second record having id: 5 didn't raise an exception because of duplicate slug column.

4.3 upsert_all in PostgreSQL

We will run the same example in the 4.1 section above with PostgreSQL database.

result = Article.upsert_all(
  [
    { id: 1, title: 'Handling 1M Requests Per Second', author: 'John', slug: '1m-req-per-second' },
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
    { id: 4, title: 'Dockerizing and Deploying Rails App to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ]
)
# PG::CardinalityViolation: ERROR:  ON CONFLICT DO UPDATE command cannot affect row a second time (ActiveRecord::StatementInvalid)
# HINT:  Ensure that no rows proposed for insertion within the same command have duplicate constrained values.

The bulk upsert operation failed in the above example due to ActiveRecord::StatementInvalid exception which was caused by another PG::CardinalityViolation exception.

The PG::CardinalityViolation exception originates from here.

The PG::CardinalityViolation exception occurs when a row cannot be updated a second time in the same ON CONFLICT DO UPDATE SQL query. PostgreSQL assumes this behavior would lead the same row to updated a second time in the same SQL query, in unspecified order, non-deterministically.

PostgreSQL recommends it is the developer's responsibility to prevent this situation from occurring.

Here's more discussion about this issue - rails/rails#35519.

Therefore, the upsert_all method doesn't work as intended due to the above limitation in PostgreSQL.

As a workaround, we can divide the single upsert_all query into multiple upsert_all queries with the use of unique_by option similar to how we did in case of SQLite in the 4.2 section above.

Article.insert_all(
  [
    { id: 1, title: 'Handling 1M requests per second', author: 'John', slug: '1m-req-per-second' },
    { id: 2, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-1' },
    { id: 4, title: 'Dockerizing and deploy Rails app to Kubernetes', author: 'Paul', slug: 'rails-on-k8s' },
    { id: 6, title: 'Working Remotely', author: 'Greg', slug: 'working-remotely' }
  ]
)

Article.upsert_all(
  [
    { id: 1, title: 'Type Safety in Elm', author: 'George', slug: 'elm-type-safety' }, # duplicate 'id' here
  ]
)

Article.upsert_all(
  [
    { id: 3, title: 'Authentication with Devise - Part 1', author: 'Laura', slug: 'devise-auth-2' }, # duplicate 'title' and 'author' here
  ],
  unique_by: :index_articles_on_title_and_author
)

Article.upsert_all(
  [
    { id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, # duplicate 'slug' here
  ]
)

puts Article.count
# 5

puts Article.all
#<ActiveRecord::Relation [#<Article id: 1, title: "Type Safety in Elm", slug: "elm-type-safety", author: "George", description: nil>, #<Article id: 2, title: "Authentication with Devise - Part 1", slug: "devise-auth-2", author: "Laura", description: nil>, #<Article id: 4, title: "Dockerizing and deploy Rails app to Kubernetes", slug: "rails-on-k8s", author: "Paul", description: nil>, #<Article id: 5, title: "Elm on Rails", slug: "1m-req-per-second", author: "Amanda", description: nil>, #<Article id: 6, title: "Working Remotely", slug: "working-remotely", author: "Greg", description: nil>]>

For reference, note that the upsert_all method also accepts returning option for PostgreSQL which we have already discussed in the 1.1 section above.

5. insert, insert! and upsert

Rails 6 has also introduced three more additional methods namely insert, insert! and upsert for convenience.

The insert method inserts a single record into the database. If that record violates a uniqueness constrain, then the insert method will skip inserting record into the database without raising an exception.

Similarly, the insert! method also inserts a single record into the database, but will raise ActiveRecord::RecordNotUnique exception if that record violates a uniqueness constraint.

The upsert method inserts or updates a single record into the database similar to how upsert_all does.

The methods insert, insert! and upsert are wrappers around insert_all, insert_all! and upsert_all respectively.

Let's see some examples to understand the usage of these methods.

Article.insert({ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, unique_by: :slug)

# is same as

Article.insert_all([{ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }], unique_by: :slug)

Article.insert!({ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, returning: %w[ id title ])

# is same as

Article.insert_all!([{ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }], returning: %w[ id title ])

Article.upsert({ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }, unique_by: :slug, returning: %w[ id title ])

# is same as

Article.upsert_all([{ id: 5, title: 'Elm on Rails', author: 'Amanda', slug: '1m-req-per-second' }], unique_by: :slug, returning: %w[ id title ])

To learn more about the bulk insert feature and its implementation, please check rails/rails#35077, rails/rails#35546 and rails/rails#35854.

Your version of PostgreSQL (90224) is too old. Active Record supports PostgreSQL >= 9.3.

Travis CI uses PostgreSQL 9.2 by default in their images. So this error can occur while testing the app on Travis CI with Rails 6. It can be resolved by using an add-on for PostgreSQL.

Let's discuss what we need to know if we are dealing with Rails 6.

Ensuring a valid Ruby version is set while creating a new Rails 6 app

While creating a new Rails 6 app, we need to ensure that the current Ruby version in the shell is set to 2.5 or newer.

If it is set to an older version then the same version will be used by the rails new command to set the Ruby version in .ruby-version and in Gemfile respectively in the created Rails app.

\$ ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]

\$ rails new meme-wizard
create
create README.md
create Rakefile
create .ruby-version
create config.ru
create .gitignore
create Gemfile
[...] omitted the rest of the output

\$ cd meme-wizard && grep -C 2 -Rn -a "2.3.1" .
./.ruby-version:1:2.3.1
--
--
./Gemfile-2-git_source(:github) { |repo| "https://github.com/#{repo}.git" }
./Gemfile-3-
./Gemfile:4:ruby '2.3.1'
./Gemfile-5-
./Gemfile-6-# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'

An easy fix for this is to install a Ruby version 2.5 or newer and use that version prior to running the rails new command.

\$ ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]

\$ rbenv local 2.6.0

\$ ruby -v
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-darwin18]

\$ rails new meme-wizard

\$ cd meme-wizard && grep -C 2 -Rn -a "2.6.0" .
./.ruby-version:1:2.6.0
--
--
./Gemfile-2-git_source(:github) { |repo| "https://github.com/#{repo}.git" }
./Gemfile-3-
./Gemfile:4:ruby '2.6.0'
./Gemfile-5-
./Gemfile-6-# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'

Upgrading an older Rails app to Rails 6

While upgrading an older Rails app to Rails 6, we need to update the Ruby version to 2.5 or newer in .ruby-version and Gemfile files respectively.

What else do we need to know?

Since Ruby 2.5 has added Hash#slice method, the extension method with the same name defined by activesupport/lib/active_support/core_ext/hash/slice.rb has been removed from Rails 6.

Similarly, Rails 6 has also removed the extension methods Hash#transform_values and Hash#transform_values! from Active Support in favor of the native methods with the same names which exist in Ruby. These methods were introduced in Ruby 2.4 natively.

If we try to explicitly require active_support/core_ext/hash/transform_values then it would print a deprecation warning.

> > require "active_support/core_ext/hash/transform_values"

# DEPRECATION WARNING: Ruby 2.5+ (required by Rails 6) provides Hash#transform_values natively, so requiring active_support/core_ext/hash/transform_values is no longer necessary. Requiring it will raise LoadError in Rails 6.1. (called from irb_binding at (irb):1)

=> true

In development environment, Rails uses async adapter as the default Active Job adapter. The Async adapter runs jobs with an in-process thread pool.

This specific issue was happening because the seed task was trying to attach a file using Active Storage. Active Storage adds a job in the background during the attachment process. This task was not getting executed properly using the async adapter and it was causing the seed task to hang without exiting.

It was found out that by using the inline adapter in development environment, this issue goes away. But making a wholesale change of making the default adapter in development environment as inline adapter defeats the purpose of having the async adapter as default in the first place.

Instead a change is made to execute all the code related to seeding using inline adapter. The inline adapter makes sure that all the code will be executed immediately.

As the inline adapter does not allow queuing up the jobs in future, this can result into an error if the seeding code somehow triggers such jobs. This issue is already reported on Github.

Update

Active Job is optional framework and we can skip it completely. Now that seeding depends on presence of Active Job, it was throwing an error when Active Job was not part of the application. Also, executing the jobs inline automatically, when users has set the Active Job queue adapter to something of their choice was surprising for the users. So a change has been made to load the seeds inline only when Active Job is included in the application and the queue adapter is async. This makes it backward compatible as well it does not change user's choice of queue adapter automatically.

Before Rails 6, if you want to change the previously set select statement attributes to new attributes, it was done as follows.

>> Post.select(:title, :body).unscope(:select).select(:views)

   SELECT "posts"."views" FROM "posts" LIMIT ? ["LIMIT", 1]]

In Rails 6, ActiveRecord::Relation#reselect method is added.

The reselect method is similar to rewhere and reorder. reselect is a short-hand for unscope(:select).select(fields).

Here is how reselect method can be used.

>> Post.select(:title, :body).reselect(:views)

   SELECT "posts"."views" FROM "posts" LIMIT ? ["LIMIT", 1]]

Check out the pull request for more details on this.

of_kind? is same as ActiveModel::Errors#added? but, it doesn't take extra options as a parameter.

Let's checkout how it works.

Rails 6.0.0.beta2

>> class User < ApplicationRecord
>>   validates :name, presence: true
>> end

>> user = User.new

=> => #<User id: nil, name: nil, password: nil, created_at: nil, updated_at: nil>

>> user.valid?

=> false

>> user.errors

=> #<ActiveModel::Errors:0x00007fc462a1d140 @base=#<User id: nil, name: nil, password: nil, created_at: nil, updated_at: nil>, @messages={:name=>["can't be blank"]}, @details={:name=>[{:error=>:blank}]}>

>> user.errors.of_kind?(:name)

=> false

>> user.errors.of_kind?(:name, :blank)

=> true

>> user.errors.of_kind?(:name, "can't be blank")

=> true

>> user.errors.of_kind?(:name, "is blank")

=> false

Here is the relevant pull request.

$ rails routes
   Prefix Verb   URI Pattern               Controller#Action
    users GET    /users(.:format)          users#index
          POST   /users(.:format)          users#create
 new_user GET    /users/new(.:format)      users#new
edit_user GET    /users/:id/edit(.:format) users#edit
     user GET    /users/:id(.:format)      users#show
          PATCH  /users/:id(.:format)      users#update
          PUT    /users/:id(.:format)      users#update
          DELETE /users/:id(.:format)      users#destroy

If we have long route names, they don't fit on the terminal window as the output lines wrap with each other.

Rails 6 has added a way to display the routes in an expanded format.

We can pass --expanded switch to the rails routes command to see this in action.

$ rails routes --expanded

--[ Route 1 ]--------------------------------------------------------------
Prefix            | users
Verb              | GET
URI               | /users(.:format)
Controller#Action | users#index
--[ Route 2 ]--------------------------------------------------------------
Prefix            |
Verb              | POST
URI               | /users(.:format)
Controller#Action | users#create
--[ Route 3 ]--------------------------------------------------------------
Prefix            | new_user
Verb              | GET
URI               | /users/new(.:format)
Controller#Action | users#new
--[ Route 4 ]--------------------------------------------------------------
Prefix            | edit_user
Verb              | GET
URI               | /users/:id/edit(.:format)
Controller#Action | users#edit

This shows the output of the routes command in much more user friendly manner.

The --expanded switch can be used in conjunction with other switches for searching specific routes.

Let's checkout how it works.

Rails 5.2

>> user = User.new

=> #<User id: nil, email: nil, password: nil, created_at: nil, updated_at: nil>

>> user.valid?

=> false

>> user.errors

=> #<ActiveModel::Errors:0x00007fc46700df10 @base=#<User id: nil, email: nil, password: nil, created_at: nil, updated_at: nil>, @messages={:email=>["can't be blank"], :password=>["can't be blank"]}, @details={:email=>[{:error=>:blank}], :password=>[{:error=>:blank}]}>

>> user.errors.slice!

=> Traceback (most recent call last):
        1: from (irb):16
NoMethodError (undefined method 'slice!' for #<ActiveModel::Errors:0x00007fa1f0e46eb8>)
Did you mean?  slice_when

>> errors = user.errors.to_h
>> errors.slice!(:email)

=> {:password=>["can't be blank"]}

>> errors

=> {:email=>["can't be blank"]}

Rails 6.0.0.beta2

>> user = User.new

=> #<User id: nil, email: nil, password: nil, created_at: nil, updated_at: nil>

>> user.valid?

=> false

>> user.errors

=> #<ActiveModel::Errors:0x00007fc46700df10 @base=#<User id: nil, email: nil, password: nil, created_at: nil, updated_at: nil>, @messages={:email=>["can't be blank"], :password=>["can't be blank"]}, @details={:email=>[{:error=>:blank}], :password=>[{:error=>:blank}]}>

>> user.errors.slice!(:email)

=> {:password=>["can't be blank"]}

>> user.errors

=> #<ActiveModel::Errors:0x00007fc46700df10 @base=#<User id: nil, email: nil, password: nil, created_at: nil, updated_at: nil>, @messages={:email=>["can't be blank"]}, @details={:email=>[{:error=>:blank}]}>

Here is the relevant pull request.

create_or_find_by is an improvement over find_or_create_by because find_or_create_by first queries for the record, and then inserts it if none is found. This could lead to a race condition.

As mentioned by DHH in the pull request, create_or_find_by has a few cons too:

• The table must have unique constraints on the relevant columns.
• This method relies on exception handling, which is generally slower.

create_or_find_by! raises an exception when creation fails because of the validations.

Let's see how both methods work in Rails 6.0.0.beta2.

Rails 6.0.0.beta2

> > class CreateUsers < ActiveRecord::Migration[6.0]
> > def change
> > create_table :users do |t|
> > t.string :name, index: { unique: true }
> >
> >       t.timestamps
> >     end
> >
> > end
> > end

> > class User < ApplicationRecord
> > validates :name, presence: true
> > end

> > User.create_or_find_by(name: 'Amit')
> > BEGIN
> > INSERT INTO "users" ("name", "created_at", "updated_at") VALUES ($1, $2, \$3) RETURNING "id" [["name", "Amit"], ["created_at", "2019-03-07 09:33:23.391719"], ["updated_at", "2019-03-07 09:33:23.391719"]]
> > COMMIT

=> #<User id: 1, name: "Amit", created_at: "2019-03-07 09:33:23", updated_at: "2019-03-07 09:33:23">

> > User.create_or_find_by(name: 'Amit')
> > BEGIN
> > INSERT INTO "users" ("name", "created_at", "updated_at") VALUES ($1, $2, \$3) RETURNING "id" [["name", "Amit"], ["created_at", "2019-03-07 09:46:37.189068"], ["updated_at", "2019-03-07 09:46:37.189068"]]
> > ROLLBACK

=> #<User id: 1, name: "Amit", created_at: "2019-03-07 09:33:23", updated_at: "2019-03-07 09:33:23">

> > User.create_or_find_by(name: nil)
> > BEGIN
> > COMMIT

=> #<User id: nil, name: nil, created_at: nil, updated_at: nil>

> > User.create_or_find_by!(name: nil)

=> Traceback (most recent call last):
1: from (irb):2
ActiveRecord::RecordInvalid (Validation failed: Name can't be blank)

Here is the relevant pull request.

Also note, create_or_find_by can lead to primary keys running out, if the type of primary key is int. This happens because each time create_or_find_by hits ActiveRecord::RecordNotUnique, it does not rollback auto-increment of the primary key. The problem is discussed in this pull request.

Rails 5.2

> > User.first.update_columns(email: 'amit@bigbinary.com')
> > SELECT "users".\* FROM "users" ORDER BY "users"."id" ASC LIMIT $1  [["LIMIT", 1]]
UPDATE "users" SET "email" = $1 WHERE "users"."id" = \$2 [["email", "amit@bigbinary.com"], ["id", 1]]

=> Traceback (most recent call last):
1: from (irb):8
ActiveRecord::StatementInvalid (PG::UndefinedColumn: ERROR: column "email" of relation "users" does not exist)
LINE 1: UPDATE "users" SET "email" = $1 WHERE "users"."id" = $2
^
: UPDATE "users" SET "email" = $1 WHERE "users"."id" = $2

Rails 6.0.0.beta2

> > User.first.update_columns(email: 'amit@bigbinary.com')
> > SELECT "users".\* FROM "users" ORDER BY "users"."id" ASC LIMIT ? [["LIMIT", 1]]

Traceback (most recent call last):
1: from (irb):1
ActiveModel::MissingAttributeError (can't write unknown attribute `email`)

Here is the relevant commit.

A method named configs_for has also been added on to fetch configurations for a particular environment.

Rails 5.2

>> ActiveRecord::Base.configurations

=> {"development"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/development.sqlite3"}, "test"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/test.sqlite3"}, "production"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/production.sqlite3"}}

Rails 6.0.0.beta2

>> ActiveRecord::Base.configurations

=> #<ActiveRecord::DatabaseConfigurations:0x00007fc18274f9f0 @configurations=[#<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fc18274f680 @env_name="development", @spec_name="primary", @config={"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/development.sqlite3"}>, #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fc18274f608 @env_name="test", @spec_name="primary", @config={"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/test.sqlite3"}>, #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fc18274f590 @env_name="production", @spec_name="primary", @config={"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/production.sqlite3"}>]>

>> ActiveRecord::Base.configurations.to_h

=> {"development"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/development.sqlite3"}, "test"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/test.sqlite3"}, "production"=>{"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/production.sqlite3"}}

>> ActiveRecord::Base.configurations['development']

=> {"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/development.sqlite3"}

>> ActiveRecord::Base.configurations.configs_for(env_name: "development")

=> [#<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fc18274f680 @env_name="development", @spec_name="primary", @config={"adapter"=>"sqlite3", "pool"=>5, "timeout"=>5000, "database"=>"db/development.sqlite3"}>]

Here is the relevant pull request.

It is easy to miss this message in the flurry of other messages.

Rails 6 has added a change to show these params in red color for better visibility.