Configure Sidekiq in Heroku

Search icon
Search Book

In the previous chapter, we have deployed our application to Heroku. However, we also need to set up Sidekiq on Heroku to process background jobs.

Credit card details

  • First, we need to have the heroku-redis add-on for our project.
  • To use an add-on, we need to verify our account using a credit card.
  • Ultimately the credit card is used to verify that the user is not bogus.
  • Since it's a free add-on, the credit card won't be charged for usage.
  • If we don't have a verified account, then we can't add the Redis add-on and thus won't be able to directly make sidekiq work in Heroku!

Note: If you don't own a credit card and is a part of the BigBinary Internship/TB-3 program, then please ask your mentor for the credit card details. Using that credit card, you can verify yourself as a non-bogus user and thereby add the free heroku-redis add-on.

Verifying account

To verify your Heroku account:

  • Go to your Account Settings in the Heroku Dashboard.
  • Click on the "Billing" tab.
  • Click on "Add Credit Card" and then enter the credit card details.

Note: Several people have faced issues when verifying their credit cards in Heroku. As per the response from Heroku Support "Cards can occasionally get stuck in the verification process which requires manual intervention to resolve".

If you face this issue you can get further information about it from the official support docs. You can also contact Heroku Support to try and resolve the issue.

Adding the heroku-redis add on

Assuming we have a verified account, the next step is to add heroku-redis to our project. Two ways to set it up.

Using the Heroku Dashboard

Go to the resources tab, and in the Add-ons section type Heroku Redis in the search field and add it.

Heroku Redis Add-on

Select the Hobby Dev - Free plan and click Submit Order Form.

Heroku Redis Add-on Plan

Or using the Heroku CLI

Run the following from the root of the project, if you haven't already added the heroku-redis add-on:

1heroku addons:create heroku-redis:hobby-dev

A brief introduction to Procfile and Heroku dynos

Before setting up the Procfile, let us understand what it is.

The Procfile specifies the commands that are executed by the app on startup. You can use a Procfile to declare a variety of process types like your app's web server, worker processes etc.

Also, let us understand what Heroku dynos are.

All Heroku applications run in a collection of lightweight Linux containers called dynos. There are three types of dyno configurations:

  • Web: Web dynos are dynos of the “web” process type that is defined in your Procfile. This is where our server usually runs.

  • Worker: Worker dynos can be of any process type declared in your Procfile, other than “web”. Usually, it's denoted using the worker. Worker dynos are typically used for background jobs, queueing systems, and timed jobs. Our Sidekiq worker will use this type of dyno configuration.

  • One-off: One-off dynos are temporary dynos that can run detached, or with their input/output attached to your local terminal. For example, when you use your terminal to access the Rails console of your Heroku app, you make use of this type of dyno configuration.

Now, since we have got an idea of what a Procfile is, let us create a file named Procfile (Note - no file extension needed) in our granite app's root directory and add the following to it:

1web: bundle exec puma -C config/puma.rb
2worker: bundle exec sidekiq -e production -C config/sidekiq.yml
3release: bundle exec rake db:migrate

The release resource in the Procfile is used to ensure that upon pushing changes to Heroku it will run the latest migrations.

The rake setup task helps in setting up the project on a new machine for development. This rake setup task helps in setting up the project by creating the database, running all the latest migrations, and then seeding the database with sample data.

We should not be seeding data in the production or staging environment unless it's communicated otherwise. We seed the database in the development environment so that we can quickly setup the project and test the working of the project. If we execute the rake setup task via the release process of Procfile then the database will be created and all migrations will be migrated but the seeding of the database will not take place because we have added a condition in the task file to populate data only if the environment is not production and staging, like so:

1task populate_with_sample_data: [:environment] do
2  if Rails.env.production?
3    puts "Skipping deleting and populating sample data in production"
4  elsif Rails.env.staging?
5    puts "Skipping deleting and populating sample data in staging"
6  else
7    create_sample_data!
8    puts "sample data has been added."
9  end

Although we can override the conditions to seed the database in a production or staging environment, we shouldn't override these conditions unless we consult with the tech lead of the project.

To know more about the importance of ./bin/setup file, please checkout this section from the "Setting up Rake tasks" chapter.

Now let's push our application to Heroku, by running the following:

1git push heroku main

Activating Heroku dynos

Dynos are not activated by default in Heroku. We have to scale our app to use the required dynos.

Let us scale our application to use both the web and worker dynos via the Heroku CLI.

Run the following from the root of the granite application:

1heroku ps:scale web=1
2heroku ps:scale worker+1

We don't need to scale the release resource since it will be automatically run once by Heroku upon pushing latest changes.

If you're confused about what's happening with the above command, then you can make use of the Heroku dashboard UI too, to do the same job.

You can also use the Heroku dashboard to activate/de-activate a dyno by doing the following:

  • Navigate to the Resources tab.
  • In the app's list of dynos, click the Edit button and use the toggle switch to activate/de-activate a particular dyno.

Activate Heroku dynos using dashboard

Gotchas with Sidekiq in Heroku

  • Sometimes our active jobs/background jobs get enqueued and not get processed in Heroku. To avoid that, make sure you have completed the above steps, and then add the following under your production key in config/database.yml:
1url: <%= ENV['DATABASE_URL'] %>
  • [Optional][rare case] Sometimes the jobs even after doing the above steps, gets enqueued in Heroku without processing. Weird right? Then we can invoke the job from our Task model as mentioned below, which hopefully should fix it:
1  after_commit :log_task_details, on: :create

Viewing the Sidekiq Web UI

This is an optional step, which you can execute if you want to monitor Sidekiq processes via browser.

Configuring Heroku Config Variables for Sidekiq

In the production environment, we need to authenticate the user before giving access to the Sidekiq Web UI. So we need to set up two env or config variables in our Heroku project:

1SIDEKIQ_USERNAME: <some_username>
2SIDEKIQ_PASSWORD: <some_password>

Here are the detailed steps to do so:

  • Navigate to the Settings tab.
  • Go to the Config Vars section and click on the Reveal Config Vars button.
  • Add the following env variables whose values would be a username and a password of your choice.

Heroku Config Vars

Now that we have setup the Heroku config vars, let's configure the routes.

Adding authentication to Sidekiq Web UI

We need to add the following into the routes.rb file to authenticate a user before giving access to the Sidekiq Web UI:

1require 'sidekiq/web'
2require 'sidekiq/cron/web'
4Rails.application.routes.draw do
5  Sidekiq::Web.use Rack::Auth::Basic do |username, password|
6    ActiveSupport::SecurityUtils.secure_compare(::Digest::SHA256.hexdigest(username), ::Digest::SHA256.hexdigest(ENV["SIDEKIQ_USERNAME"])) &
7      ActiveSupport::SecurityUtils.secure_compare(::Digest::SHA256.hexdigest(password), ::Digest::SHA256.hexdigest(ENV["SIDEKIQ_PASSWORD"]))
8  end
9  mount Sidekiq::Web, at: "/sidekiq"
11  constraints(lambda { |req| req.format == :json }) do
12    resources :tasks, except: %i[new edit], param: :slug
13    resources :users, only: %i[create index]
14    resource :session, only: %i[create destroy]
15    resources :comments, only: :create
16    resource :preference, only: %i[show update] do
17      patch :mail, on: :collection
18    end
19  end
21  root "home#index"
22  get '*path', to: 'home#index', via: :all

Even with such a barebones routes file, it's already looking very cluttered and pretty unreadable.

When writing code the key point to focus on is to make code readable. Someone else might be working on the same codebase in the future. Thus naming and properly modularizing go a long way.

Let's see how we can make the routes.rb file look cleaner.

Modularizing routes file

Therefore, to enhance readability, we can break the routes.rb file into multiple small units:

1require 'sidekiq/web'
2require 'sidekiq/cron/web'
4Rails.application.routes.draw do
5  def draw(routes_name)
6    instance_eval("config/routes/#{routes_name}.rb")))
7  end
9  draw :sidekiq
11  #previous routes as is

Here, we have defined a custom macro named draw. As we had discussed in previous chapters, a macro is a method, which instead of returning a Ruby datatype, returns more Ruby code.

This draw macro allows us to load smaller route files from the config/routes folder. Now let's configure a route file for sidekiq.

Run the following command:

1touch config/routes/sidekiq.rb

Open the file and add the following:

1# frozen_string_literal: true
2require "sidekiq/web"
4def sha256_digest(value)
5  ::Digest::SHA256.hexdigest(value)
8def secure_compare(string, key)
9  sidekiq_web_credentials = Rails.application.secrets.sidekiq
10  expected_credential = sidekiq_web_credentials && sidekiq_web_credentials[key]
11  return false if [string, expected_credential].any?(&:nil?)
13  ActiveSupport::SecurityUtils.secure_compare(
14    sha256_digest(string),
15    sha256_digest(expected_credential)
16  )
19mount Sidekiq::Web => "/sidekiq"
21if %w[staging production].include?(Rails.env)
22  Sidekiq::Web.use Rack::Auth::Basic do |username, password|
23    secure_compare(username, :username) & secure_compare(password, :password)
24  end

Let us make use of the secrets.yml file to access the env/config vars we have configured in Heroku into our Rails application.

Fully replace the config/secrets.yml file with the following content:

1default: &default
2  redis_url:
3    <%= ENV['REDISTOGO_URL'] || ENV['REDIS_URL'] || 'redis://localhost:6379/1'%>
4  sidekiq:
5    username: <%= ENV['SIDEKIQ_USERNAME'] %>
6    password: <%= ENV['SIDEKIQ_PASSWORD'] %>
9  <<: *default
12  <<: *default
15  <<: *default
18  <<: *default

Now, let us understand the various methods defined in config/routes/sidekiq.rb. We have split the code into these methods for enhancing readability and reusability.

  • The secure_compare method uses the built-in ActiveSupport::SecurityUtils.secure_compare method for string comparison. We are making use of this method to prevent timing attacks.

A timing attack uses statistical analysis of how long it takes for your application to do something to learn something about the data it’s operating on.

While an attacker cannot use a timing attack to find out the secrets, it is possible to determine the secret length. Now here's where our sha256_digest method will come to the rescue.

The [string, expected_credential].any?(&:nil?) part checks whether either the configured credential or supplied credential is nil.

If any of those values are nil, then we have to fail our authentication.

To mark the process as failure, we can return false value.

But what does &:nil? in [string, expected_credential].any?(&:nil?) denote?

By prepending & to a symbol we are creating a lambda function that will call method with a name of that symbol(.nil? here) on the object you pass into this function.

So the above statement is equivalent to the following:

1[string, expected_credential].any? { |item| item.nil? }
  • The sha256_digest(value) method computes and returns a 256-bit hash or message digest of the value parameter supplied.

We are computing the message digest because, doing so will prevent leaking of the length information of value.

Irrespective of the length of value, this method will always return a 256-bit hash.

Also, the hashing function used is one-way i.e, one cannot compute the original value from the digest.

Now you can view the Sidekiq Web UI by navigating to the /sidekiq endpoint.

Note that we have enabled authentication only in staging and production environments.

In the development environment we can directly access the Sidekiq Web UI without providing any credentials.

Heroku Sidekiq Auth

A sample preview of the Sidekiq Web UI:

Sidekiq Web UI

Now commit the changes that were made in this chapter:

1git add -A
2git commit -m  "Added Heroku Sidekiq configs"