Please note that Solid Queue currently doesn't support cron-style or recurring jobs. There is a PR open regarding this issue. We have only partially migrated to Solid Queue. For recurring jobs, we are still using Sidekiq. Once the PR is merged, we will migrate completely to Solid Queue.

Migrating to Solid Queue from Sidekiq

Here is a step-by-step migration guide you can use to migrate your Rails application from Sidekiq to Solid Queue.

1. Installation

• Add gem "solid_queue" to your Rails application's Gemfile and run bundle install.
• Run bin/rails generate solid_queue:install which copies the config file and the required migrations.
• Run the migrations using bin/rails db:migrate.

2. Configuration

The installation step should have created a config/solid_queue.yml file. Uncomment the file and modify it as per your needs. Here is how the file looks for our application.

default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "auth"
      threads: 3
      processes: 1
      polling_interval: 0.1
    - queues: "urgent"
      threads: 3
      processes: 1
      polling_interval: 0.1
    - queues: "low"
      threads: 3
      processes: 1
      polling_interval: 2
    - queues: "*"
      threads: 3
      processes: 1
      polling_interval: 1

development:
  <<: *default

staging:
  <<: *default

heroku:
  <<: *default

test:
  <<: *default

production:
  <<: *default

3. Starting Solid Queue

On your development machine, you can start Solid Queue by running the following command.

bundle exec rake solid_queue:start

This will start Solid Queue's supervisor process and will start processing any enqueued jobs. The supervisor process forks workers and dispatchers according to the configuration provided in the config/solid_queue.yml file. The supervisor process also controls the heartbeats of workers and dispatchers, and sends signals to stop and start them when needed.

Since we use foreman, we added the above command to our Procfile.

# Procfile
web:  bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -C config/sidekiq.yml
solidqueueworker: bundle exec rake solid_queue:start
release: bundle exec rake db:migrate

4. Setting the Active Job queue adapter

You can set the Active Job queue adapter to :solid_queue by adding the following line in your application.rb file.

# application.rb
config.active_job.queue_adapter = :solid_queue

The above change sets the queue adapter at the application level for all the jobs. However, since we wanted to use Solid Queue for our regular jobs and continue using Sidekiq for cron jobs, we didn't make the above change in application.rb.

Instead, we created a new base class that inherited from ApplicationJob and set the queue adapter to :solid_queue inside that.

# sq_base_job.rb
class SqBaseJob < ApplicationJob
  self.queue_adapter = :solid_queue
end

Then we made all the classes implementing regular jobs inherit from this new class SqBaseJob instead of ApplicationJob.

# send_email_job.rb
- class SendEmailJob < ApplicationJob
+ class SendEmailJob < SqBaseJob
  # ...
end

By making the above change, all our regular jobs got enqueued via Solid Queue instead of Sidekiq.

But we realized later that emails were still being sent via Sidekiq. On debugging and looking into Rails internals, we found that ActionMailer uses ActionMailer::MailDeliveryJob for enqueuing or sending emails.

ActionMailer::MailDeliveryJob inherits from ActiveJob::Base rather than the application's ApplicationJob. So even if we set the queue_adapter in application_job.rb, it won't work. ActionMailer::MailDeliveryJob fallbacks to using the adapter defined in application.rb or environment-specific (production.rb / staging.rb / development.rb) config files. But we can't do that because we still want to use Sidekiq for cron jobs.

To use Solid Queue for mailers, we needed to override the queue_adapter for mailers. We can do that in application_mailer.rb.

# application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  # ...
  ActionMailer::MailDeliveryJob.queue_adapter = :solid_queue
end

This change is only until we use both Sidekiq and Solid Queue. Once cron style jobs feature lands in Solid Queue, we can remove this override and set the queue_adapter directly in application.rb, which will enforce the setting globally.

5. Code changes

For migrating from Sidekiq to Solid Queue, we had to make the following changes to the syntax for enqueuing a job.

• Replaced .perform_async with .perform_later.
• Replaced .perform_at with .set(...).perform_later(...).

- SendMailJob.perform_async
+ SendMailJob.perform_later

- SendMailJob.perform_at(1.minute.from_now)
+ SendMailJob.set(wait: 1.minute).perform_later

At some places we were storing the Job ID on a record, for querying the job's status or for cancelling the job. For such cases, we made the following change.

def disable_form_at_deadline
- job_id = DisableFormJob.perform_at(deadline, self.id)
- self.disable_job_id = job_id
+ job_id = DisableFormJob.set(wait_until: deadline).perform_later(self.id)
+ self.disable_job_id = job.job_id
end

def cancel_form_deadline
- Sidekiq::Status.cancel(self.disable_job_id)
+ SolidQueue::Job.find_by(active_job_id: self.disable_job_id).destroy!
  self.disable_job_id = nil
end

6. Error handling and retries

Initially, we thought the on_thread_error configuration provided by Solid Queue can be used for error handling. However, during the development phase, we noticed that it wasn't capturing errors. We raised an issue with Solid Queue as we thought it was a bug.

Rosa Gutiérrez responded on the issue and clarified the following.

on_thread_error wasn't intended for errors on the job itself, but rather errors in the thread that's executing the job, but around the job itself. For example, if you had an Active Record's thread pool too small for your number of threads and you got an error when trying to check out a new connection, on_thread_error would be called with that.

For errors in the job itself, you could try to hook into Active Job's itself.

Based on the above information, we modified our SqBaseJob base class to handle the exceptions and report it to Honeybadger.

# sq_base_job.rb
class SqBaseJob < ApplicationJob
  self.queue_adapter = :solid_queue

  rescue_from(Exception) do |exception|
    context = {
      error_class: self.class.name,
      args: self.arguments,
      scheduled_at: self.scheduled_at,
      job_id: self.job_id
    }
    Honeybadger.notify(exception, context:)
    raise exception
  end
end

Remember we mentioned that ActionMailer doesn't inherit from ApplicationJob. So similarly, we would have to handle exceptions for Mailers separately.

# application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  # ...
  ActionMailer::MailDeliveryJob.rescue_from(Exception) do |exception|
    context = {
      error_class: self.class.name,
      args: self.arguments,
      scheduled_at: self.scheduled_at,
      job_id: self.job_id
    }
    Honeybadger.notify(exception, context:)
    raise exception
  end
end

For retries, unlike Sidekiq, Solid Queue doesn't include any automatic retry mechanism, it relies on Active Job for this. We wanted our application to retry sending emails in case of any errors. So we added the retry logic in the ApplicationMailer.

# application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  # ...
  ActionMailer::MailDeliveryJob.retry_on StandardError, attempts: 3
end

Note that, although the queue adapter configuration can be removed from application_mailer.rb once the entire application migrates to Solid Queue, error handling and retry override cannot be removed because of the way ActionMailer::MailDeliveryJob inherits from ActiveJob::Base rather than application's ApplicationJob.

7. Testing

Once all the above changes were done, it was obvious that a lot of tests were failing. Apart from fixing the usual failures related to the syntax changes, some of the tests were failing inconsistently. On debugging, we found that the affected tests were all related to controllers, specifically tests inheriting from ActionDispatch::IntegrationTest.

We tried debugging and searched for solutions when we stumbled upon Ben Sheldon's comment on one of Good Job's issues. Ben points out that this is actually an issue in Rails where Rails sometimes inconsistently overrides ActiveJob's queue_adapter setting with TestAdapter. A PR is already open for the fix. Thankfully, Ben, in the same comment, also mentioned a workaround for it until the fix has been added to Rails.

We added the workaround in our test helper_methods.rb and called the method in each of our controller tests which were failing.

# test/support/helper_methods.rb
def ensure_consistent_test_adapter_is_used
  # This is a hack mentioned here: https://github.com/bensheldon/good_job/issues/846#issuecomment-1432375562
  # The actual issue is in Rails for which a PR is pending merge
  # https://github.com/rails/rails/pull/48585
  (ActiveJob::Base.descendants + [ActiveJob::Base]).each(&:disable_test_adapter)
end

# test/controllers/exports_controller_test.rb
class ExportsControllerTest < ActionDispatch::IntegrationTest
  def setup
    ensure_consistent_test_adapter_is_used
    # ...
  end

  # ...
end

8. Monitoring

Basecamp has released mission_control-jobs, which can be used to monitor background jobs. It is generic, so it can be used with any compatible ActiveJob adapter.

Add gem "mission_control-jobs" to your Gemfile and run bundle install.

Mount the mission control route in your routes.rb file.

# routes.rb
Rails.application.routes.draw do
  # ...
  mount MissionControl::Jobs::Engine, at: "/jobs"

By default, mission control would try to load the adapter specified in your application.rb or individual environment-specific files. Currently, Sidekiq isn't compatible with mission control, so you will face an error while loading the dashboard at /jobs. The fix is to explicitly specify solid_queue to the list of mission control adapters.

# application.rb
# ...
config.mission_control.jobs.adapters = [:solid_queue]

Now, visiting /jobs on your site should load a dashboard where you can monitor your Solid Queue jobs.

But that isn't enough. There is no authentication. For development environments, it is fine, but the /jobs route would be exposed on production too. By default, Mission Control's controllers will extend the host app's ApplicationController. If no authentication is enforced, /jobs will be available to everyone.

To implement some kind of authentication, we can specify a different controller as the base class for Mission Control's controllers and add the authentication there.

# application.rb
# ...
MissionControl::Jobs.base_controller_class = "MissionControlController"

# app/controllers/mission_control_controller.rb
class MissionControlController < ApplicationController
  before_action :authenticate!, if: :restricted_env?

  private

    def authenticate!
      authenticate_or_request_with_http_basic do |username, password|
        username == "solidqueue" && password == Rails.application.secrets.mission_control_password
      end
    end

    def restricted_env?
      Rails.env.staging? || Rails.env.production?
    end
end

Here, we have specified that MissionControlController would be our base controller for mission control related controllers. Then in MissionControlController we implemented basic authentication for staging and production environments.

Observations

We haven't had any complaints so far. Solid Queue offers simplicity, requires no additional infrastructure and provides visibility for managing jobs since they are stored in the database.

In the coming days, we will migrate all of our 22 Neeto products to Solid Queue. And once cron-style job support lands in Solid Queue, we will completely migrate from Sidekiq.

Recently, 37signals open sourced Solid Queue.

Solid Queue is database based queuing backend for Active Job. In contrast Sidekiq and Resque are Redis-based queuing backends.

In her blog Rosa Gutiérrez mentioned following lines which captured our attention.

In our case, one feature that PostgreSQL has had for quite some time and that was finally introduced in MySQL 8, has been crucial to our implementation:

SELECT ... FOR UPDATE SKIP LOCKED

This allows Solid Queue’s workers to fetch and lock jobs without locking other workers.

As per her, this feature had been in PostgreSQL for a while, and now this feature has landed in MySQL making it possible to build Solid Queue.

We had never heard of UPDATE SKIP LOCKED feature either in PostgreSQL or in MySQL. We were wondering what is this UPDATE SKIP LOCKED without which it was not possible to build Solid Queue. So we decided to look into it.

Processing jobs from a queue

Consider a case where we need to build a system where a bunch of jobs need to be processed in the background.

There are a bunch of workers waiting to grab a job and start processing the moment a job becomes available. The challenge is when multiple workers attempt to claim the same job simultaneously how do we ensure that only one of the workers claims the job for processing. At any point of time, a worker should claim only the "unclaimed" job, and an "unclaimed" job should be claimed by one and only one worker.

Here is how one might go about it implementing it.

START TRANSACTION
SELECT * FROM JOBS WHERE processed='no' LIMIT 1;
-- Process the job
COMMIT;

With the above code, it's possible that two workers might claim the same job.

One way to resolve this issue is by marking a particular row as locked for update.

START TRANSACTION;
SELECT * FROM JOBS WHERE processed='no' FOR UPDATE LIMIT 1;
-- Process the job
COMMIT

SELECT ... FOR UPDATE locks a particular row, and hence no one else can lock that record.

As soon as a new job comes in, multiple workers will execute the above query and will try to take a lock on that record. The database will ensure that only one of the workers gets the lock.

The first worker will take the lock on the record using FOR UPDATE. When other workers come to that record and they see that there is a lock FOR UPDATE, they will wait for the lock to be lifted. Yes, these workers will wait until the lock is released.

The lock will only be released when the transaction is committed. When the transaction is committed and the lock is released, then other workers will get hold of the record only to find that the job has already been processed. As you can see, this is a highly inefficient process.

That is where FOR UPDATE SKIP LOCKED comes in.

SKIP LOCKED skips locked rows

START TRANSACTION;
SELECT * FROM jobs_table FOR UPDATE SKIP LOCKED LIMIT 1;
-- Process the job
COMMIT;

Imagine the same scenario here. A job comes in. Multiple workers compete to claim the job. The database ensures that only one worker gets the lock. However in this case the other workers will move on to the next record. They will not wait. That's what SKIP LOCKED does.

MySQL has detailed documentation on how SKIP LOCKED works If you want to read in more detail.

Solid Queue uses FOR UPDATE SKIP LOCKED feature to ensure that a job is claimed by only one worker.

How GoodJob manages job processing without SKIP LOCKED

GoodJob burst into the scene around July, 2020. GoodJobs supports only PostgreSQL database because it uses advisory locks to guarantee that no two workers claim the same job.

PostgreSQL folks understand that the lock mechanism provided by the database would not satisfy all the variety of cases that might arise in an application. Advisory locks are a mechanism that allows applications to establish a communication channel to coordinate actions between different sessions or transactions. Unlike regular row-level locks enforced by the database system, advisory locks are implemented as a set of low-level functions that applications can use to acquire and release locks based on their requirements. We can read more about it here.

pg_advisory_lock function will lock the given resource. However, if another session already holds a lock on the same resourc,e then this function will wait. This is similar to the FOR UPDATE case we saw above.

However pg_try_advisory_lock function will either obtain the lock immediately and return true, or return false if the lock cannot be acquired immediately. As you can see, the name has the word try. This function attempts to acquire a lock. If it can't get the lock, then it won't wait. Now this function can be utilized to build a queuing system.

Any usage of an advisory lock means the application needs to coordinate action. It gives more power to the application but it also means more work by the application. In contrast, FOR UPDATE SKIP RECORD is natively supported by PostgreSQL.

Based on the discussions here and here, it seems GoodJob is evaluating the possibility of migrating from advisory locks to using FOR UPDATE SKIP LOCKED for better performance. Going through these issues was quite revealing and I got to learn a lot about things I was unaware of.

Delayed job implementation

DelayedJob has been there since 2009, long before Sidekiq. It doesn't use SKIP LOCK. Instead it uses a row level locking system by updating a field in the job record to indicate that the job is being processed. In short DelayedJob ensures that no two workers take the same job at the application level without taking any help in this direction from the database.

What about SQLite

So far, we discussed PostgreSQL and MySQL. What about SQLite? Does it support SKIP LOCK. No, it doesn't support it, but it's ok. As per the documentation, it supports only writer at any instant in time.

High Concurrency

SQLite supports an unlimited number of simultaneous readers, but it will only allow one writer at any instant in time. For many situations, this is not a problem. Writers queue up. Each application does its database work quickly and moves on, and no lock lasts for more than a few dozen milliseconds. But there are some applications that require more concurrency, and those applications may need to seek a different solution.

NOWAIT

For completeness, let's discuss the NOWAIT feature. We saw earlier that if we take a lock on a row using FOR UPDATE, then other workers will wait until the lock is released.

START TRANSACTION;
SELECT * FROM JOBS WHERE processed='no' FOR UPDATE NOWAIT LIMIT 1;
-- Process the job
COMMIT

NOWAIT feature allows other transactions to not wait for the lock to be released. In this case if a transaction is not able to get a lock on the given row then it will raise an error and the application needs to handle the error.

In contrast, SKIP LOCKED will allow the transaction to move on to the next row if a lock is already taken.

Redis backed queue vs database backed queue

Now that we looked at how FOR UPDATE SKIP LOCK helps build queuing system using database itself, let's see some pros and cons of each type of queuing system.

Simplicity and familiarity

Database-backed queues are often simpler to set up and manage, especially if your application is already using a relational database. There's no need for an additional dependency like Redis.

No Additional Infrastructure

Since the job information is stored in the same database as your application data, you don't need to set up and maintain a separate infrastructure like a Redis server.

Transactionality

Database-backed queues can leverage database transactions, ensuring that both the job creation and any related database operations are committed or rolled back together. This can be important in scenarios where data consistency is critical.

Modifiability

It is easier to modify the jobs stored in the database than Redis, but doing so requires caution, and it's generally not recommended. In Redis, jobs are often stored as serialized data, and modifying them directly is not as straightforward or common. Redis provides commands to interact with data, but modifying job data directly is not a standard practice and could result in data corruption.