Search
⌘K
    to navigateEnterto select Escto close

    Handling idempotency when sending emails using Sidekiq

    Now that we have covered unit testing and background jobs with Sidekiq, let's add a new feature to send reminder emails to users with outstanding tasks.

    In this chapter we will discuss concepts like idempotency, mail delivery windows, setting user preferences, and more.

    Features

    These are the basic requirements of the feature :

    • Send reminder emails to users with pending tasks.
    • Users should have an option to select their timezone while scheduling mail.
    • User can set preferred hour i.e delivery time to receive mail with all pending tasks, if any.

    Mail delivery feature

    Technical design

    To implement this feature, we need to introduce the following changes:

    On the backend

    • For scheduling periodic Sidekiq jobs, we can use the sidekiq-cron gem, which helps us to schedule the worker, which will be sending the mail.

    • In the development environment, instead of sending an email to the user's email address, we can preview the email in our browser using the letter_opener gem.

    • Mail will be sent to each user at 10 AM by default, or at their preferred time, which we will store in a new table called preference. This preferences table consists of notification_delivery_hour,receive_email and user columns.

    • Create user_notifications table with last_notification_sent_date and user columns.

    • In user_notification model, add validations to ensure that the last_notification_sent_date attribute is present, has a valid date format, and the date is not in the past.

    • We will be creating a default preference for the user whenever a new user is created.

    • We will be saving the default preference value in a constant DEFAULT_NOTIFICATION_DELIVERY_HOUR, which will be loaded automatically from config/initializers/constants.rb, by Rails, after it loads the framework plus any gems and plugins in our application.

    • Add PreferencesController which will add the ability for the user to update their preference for when to receive the email.

    • Add PreferencePolicy to enforce authorization so that only authorized users can access the preferences and update them.

    • Mail sending can often fail due to network errors, among others. When that happens, Sidekiq will retry based on rules we set in the initializer.

    On the frontend

    • Define preference related axios connectors.

    • We will add a MyPreferences component which will handle the preferences of the user.

    • Delegate the job of taking in the preferred time for mail delivery to a PreferenceForm component within the MyPreferences component.

    • Add a function fetchPreferenceDetails in the MyPreferences container component which will fetch preference details like notification_delivery_hour and receive_email.

    • Similarly, add a function updatePreference in the MyPreferences container component which will update notification_delivery_hour.

    Sidekiq worker

    To send each batch of reminder emails, we will schedule a TodoNotificationsWorker to run every hour and delegate the job of finding the recipients to a TodoNotificationsService.

    This service will ultimately invoke separate UserNotificationWorker's for each recipient it finds.

    Invoking separate workers helps to avoid having a single point of failure in the batch, which may bring the whole delivery system down.

    We will also create a scheduler file to store cron values that will help schedule our TodoNotificationsWorker at the desired time interval for each environment.

    The parent Sidekiq worker will then invoke TodoNotificationsService, which will find all the users with pending tasks but no entries in a new table called user_notifications.

    What is idempotency?

    We need our service to be idempotent, ie: duplicate mail shouldn't be sent every time we retry.

    From a RESTful service standpoint, an operation or service call is idempotent when clients can perform a duplicate request harmlessly. In other words, making multiple identical requests has the same effect as making a single request.

    Note that while idempotent operations produce the same result on the server, the response itself may not be the same; e.g. a resource's state may change between requests.

    To ensure this, we will use the user_notifications table from earlier to associate each user with a last_notification_sent_date.

    If a user has an entry for today in user_notifications, it means there's no need to try again.

    Action Mailer

    Action Mailer allows you to send emails from your application using mailer classes and views. They inherit from ActionMailer::Base and live in app/mailers.

    To make sure that the email has truly been sent to the user, we will make use of Action Mailer's after_action callback to safely set the last_notification_sent_date value only after the mail has been sent successfully.

    Mailers have methods called "actions" and they use views to structure their content.

    A controller generates content like say HTML, which will be sent back to the client, whereas a Mailer creates a message to be delivered via email.

    We will be creating the TaskMailer mailer and task_mailer view.

    We are now ready to start coding. Let us dive in.

    Setting up base

    Open the Gemfile.rb and add the following lines:

    1# For periodic sidekiq jobs
    2gem "sidekiq-cron"
    3
    4# For opening mails in development env
    5gem "letter_opener", :group => :development

    Install the gems:

    1bundle install

    In order for the letter_opener gem to open the mails automatically in our browser, we need to set the following in config/environments/development.rb:

    1config.action_mailer.delivery_method = :letter_opener
    2config.action_mailer.perform_deliveries = true

    Now any email delivery will pop up in our browser instead of being sent via the internet. The messages will be stored in ./tmp/letter_opener.

    Creating models

    1bundle exec rails g model user_notification
    2bundle exec rails g model preference

    Go to db/migrate folder and paste the following lines of code for the CreateUserNotifications migration:

    1class CreateUserNotifications < ActiveRecord::Migration[6.1]
    2  def change
    3    create_table :user_notifications do |t|
    4      t.date :last_notification_sent_date, null: false
    5      t.references :user, foreign_key: true
    6      t.index [:user_id, :last_notification_sent_date],
    7        name: :index_user_preferences_on_user_id_and_notification_sent_date,
    8        unique: true
    9      t.timestamps
    10    end
    11  end
    12end

    Also, paste the following lines of code for the CreatePreferences migration:

    1class CreatePreferences < ActiveRecord::Migration[6.1]
    2  def change
    3    create_table :preferences do |t|
    4      t.integer :notification_delivery_hour
    5      t.boolean :receive_email, default: true, null: false
    6      t.references :user, foreign_key: true
    7      t.timestamps
    8    end
    9  end
    10end

    Run the migrations:

    1bundle exec rails db:migrate

    Adding validations

    Now let's setup UserNotification model and add validations for last_notification_sent_date.

    Open app/models/user_notification.rb and add the following lines to it:

    1class UserNotification < ApplicationRecord
    2  belongs_to :user
    3
    4  validates :last_notification_sent_date, presence: true
    5  validate :last_notification_sent_date_is_valid_date
    6  validate :last_notification_sent_date_cannot_be_in_the_past
    7
    8  private
    9
    10  def last_notification_sent_date_is_valid_date
    11    if last_notification_sent_date.present?
    12      Date.parse(last_notification_sent_date.to_s)
    13    end
    14  rescue ArgumentError
    15    errors.add(:last_notification_sent_date, "must be a valid date")
    16  end
    17
    18  def last_notification_sent_date_cannot_be_in_the_past
    19    if last_notification_sent_date.present? &&
    20        last_notification_sent_date < Time.zone.today
    21      errors.add(:last_notification_sent_date, "can't be in the past")
    22    end
    23  end
    24end

    Now let's setup our Preference model along with validations in app/models/preference.rb:

    1class Preference < ApplicationRecord
    2  belongs_to :user
    3
    4  validates :notification_delivery_hour, presence: true,
    5    numericality: { only_integer: true },
    6    inclusion: {
    7      in: 0..23,
    8      message: "value should be between 0 and 23"
    9    }
    10end

    Currently, we are only providing the ability to set the preference for when to receive an email notification.

    Thus in the above code, we have added the validations for the notification_delivery_hour attribute which makes sure that it's present, and it only accepts integers ranging from 0-23.

    Fully replace config/initializers/constants.rb with the following lines of code:

    1module Constants
    2  is_sqlite_db = ActiveRecord::Base.connection_db_config.configuration_hash[:adapter] == "sqlite3"
    3  DB_REGEX_OPERATOR = is_sqlite_db ? "REGEXP" : "~*"
    4
    5  DEFAULT_NOTIFICATION_DELIVERY_HOUR = 10
    6end

    Now let's set the associations correctly in our User model too:

    1class User < ApplicationRecord
    2  # previous code if any
    3  has_many :user_notifications, dependent: :destroy, foreign_key: :user_id
    4  has_one  :preference, dependent: :destroy, foreign_key: :user_id
    5
    6  # previous code if any
    7  before_save :to_lowercase
    8  before_create :build_default_preference
    9
    10  private
    11
    12    # previous code if any
    13
    14    def to_lowercase
    15      email.downcase!
    16    end
    17
    18    def build_default_preference
    19      self.build_preference(notification_delivery_hour: Constants::DEFAULT_NOTIFICATION_DELIVERY_HOUR)
    20    end
    21end

    The private method, build_default_preference, will create a default preference for the user whenever a new user is created.

    Inside the build_default_preference method we have used the build_preference method. Let's take a look at where it came from.

    When we declare a belongs_to or has_one association, the declaring class automatically will have access to a build_association method related to the association. In this case, because of the has_one :preference association in User model, Rails adds the build_preference method to the User class.

    In case of has_many or has_and_belongs_to_many associations, we should use the association.build method instead of the build_association method. For example, if we were to declare a has_many :preferences association in the User model, we would have to use the preferences.build method given that Rails won't add the build_preferences method for an association that is not has_one.

    Adding preferences for existing users

    The build_preference method we had added in the previous section will only add a default preference for new users. The preference for all existing users will remain nil. There are two ways by which we can fix this issue:

    1. Deleting all users from the console and creating new users. Newly added users will have a default preference but we shouldn't take this approach because:

      • It's considered a bad practice.

      • Once the application is deployed, we shouldn't delete users from console in production environment.

    2. A better way to do this would be to add a migration which will add default preference for all existing users where preference is nil.

    If you recall, we have done something similar in the Adding slug to task chapter where we had used a migration to add slugs to existing task records.

    Before moving on towards adding a migration to fix the issue, you should attempt to do this by yourself. If you need help you can refer to the Adding slug to task chapter.

    Now, let's us add a migration to fix this issue. Run the following command to generate a migration:

    1bundle exec rails g migration AddDefaultPreferenceToExistingUsers

    The above command with generate a migration file. Inside the migration file add the following lines of code:

    1class AddDefaultPreferenceToExistingUsers < ActiveRecord::Migration[6.1]
    2 def up
    3    users_with_nil_preference = User.where.missing(:preference)
    4    users_with_nil_preference.each do |user|
    5      user.send(:build_default_preference)
    6      user.save!
    7    end
    8  end
    9
    10  def down
    11    User.find_each do |user|
    12      user.preference.delete
    13    end
    14  end
    15end

    We have used the missing method which performs a left outer join operation on parent and child tables then uses a where clause to identify the missing relations. In this case we are querying all user records which have a missing preference relation and then adding a default preference for all those user records.

    You can read more about the missing method from official documentation.

    Run the following command to execute the migration:

    1bundle exec rails db:migrate

    Creating the controller

    Create a new controller for the preference model by running the following command:

    1bundle exec rails generate controller Preferences

    Open app/controllers/preferences_controller.rb and add the following lines to it:

    1class PreferencesController < ApplicationController
    2  before_action :authenticate_user_using_x_auth_token
    3  before_action :load_preference, only: %i[update]
    4
    5  def show
    6    preference = Preference.find_by(user_id: params[:id])
    7    render status: :ok, json: {preference: preference}
    8  end
    9
    10  def update
    11    if @preference.update(preference_params)
    12      render status: :ok, json: {
    13        notice: t('successfully_updated', entity: 'Preference')
    14      }
    15    else
    16      errors = @preference.errors.full_messages.to_sentence
    17      render status: :unprocessable_entity, json: { error: errors }
    18    end
    19  end
    20
    21  private
    22
    23    def preference_params
    24      params.require(:preference).permit(:notification_delivery_hour, :receive_email)
    25    end
    26
    27    def load_preference
    28      @preference = Preference.find_by(id: params[:id])
    29      unless @preference
    30        render status: :not_found, json: { error: "Preference not found" }
    31      end
    32    end
    33end

    In the above code, you can see that we have used errors, full_messages, and to_sentence helpers.

    The errors helper method provides us all the errors that might've occurred while performing any ActiveRecord operation on a particular model.

    errors is an instance of ActiveRecord::Errors. Thus we can make use of the public instance methods which come as part of this class to format the errors in our desired manner.

    There is a method called full_messages, which will provide us the full error messages in an array.

    We used the to_sentence helper to convert the array of errors returned from errors.full_messages to a comma-separated sentence where the last element is joined by the connector word, which by default is and.

    Now, let's create the route for the preference. Open the routes.rb and append the following line:

    1Rails.application.routes.draw do
    2  defaults format: :json do
    3    # ---Previous Routes---
    4    resources :preferences, only: %i[show update]
    5  end
    6
    7  root "home#index"
    8  get '*path', to: 'home#index', via: :all
    9end

    Adding Preference policy

    We should add a mechanism which will forbid a user from updating another user's preferences.

    We have seen in the Authorization chapter that we can use policies to enforce authorization by using the Pundit gem.

    So similarly, let's create the preference_policy file using the following command:

    1touch app/policies/preference_policy.rb

    Open app/policies/preference_policy.rb and add the following lines of code into it:

    1class PreferencePolicy
    2  attr_reader :user, :preference
    3
    4  def initialize(user, preference)
    5    @user = user
    6    @preference = preference
    7  end
    8
    9  def show?
    10    preference.user_id == user.id
    11  end
    12
    13  def update?
    14    show?
    15  end
    16end

    The condition for the update policy is the same as that of the show policy. Hence, we can call show? inside the update? policy here. It's the same as reusing a method in Ruby.

    Let's also update the PreferencesController and add authorization to the respective controller actions:

    1class PreferencesController < ApplicationController
    2  # previous code if any
    3
    4  def show
    5    preference = Preference.find_by(user_id: params[:id])
    6    authorize preference
    7    render status: :ok, json: { preference: preference }
    8  end
    9
    10  def update
    11    authorize @preference
    12    if @preference.update(preference_params)
    13      render status: :ok, json: { notice: t("successfully_updated", entity: "Preference") }
    14    else
    15      errors = @preference.errors.full_messages.to_sentence
    16      render status: :unprocessable_entity, json: { error: errors }
    17    end
    18  end
    19
    20  # previous code if any
    21end

    Building the UI for user preference

    Since we have already added APIs for handling user preference, let's now build the views for it.

    Let's create a new file to define all the preference related APIs:

    1touch app/javascript/src/apis/preferences.js

    Open the apis/preferences.js and paste the following lines:

    1import axios from "axios";
    2
    3const show = userId => axios.get(`/preferences/${userId}`);
    4
    5const update = ({ id, payload }) => axios.put(`/preferences/${id}`, payload);
    6
    7const preferencesApi = {
    8  show,
    9  update
    10};
    11
    12export default preferencesApi;

    Now, let's create the preference component:

    1mkdir -p app/javascript/src/components/MyPreferences
    2touch app/javascript/src/components/MyPreferences/index.jsx

    Paste the following lines into index.jsx:

    1import React, { useState, useEffect } from "react";
    2
    3import preferencesApi from "apis/preferences";
    4import Container from "components/Container";
    5import PageLoader from "components/PageLoader";
    6import { getFromLocalStorage } from "helpers/storage";
    7
    8import PreferenceForm from "./Form";
    9
    10const MyPreferences = () => {
    11  const [notificationDeliveryHour, setNotificationDeliveryHour] = useState("");
    12  const [receiveEmail, setReceiveEmail] = useState(true);
    13  const [loading, setLoading] = useState(false);
    14  const [pageLoading, setPageLoading] = useState(true);
    15  const [preferenceId, setPreferenceId] = useState("");
    16  const userId = getFromLocalStorage("authUserId");
    17
    18  const updatePreference = async () => {
    19    try {
    20      setLoading(true);
    21      await preferencesApi.update({
    22        id: preferenceId,
    23        payload: {
    24          notification_delivery_hour: notificationDeliveryHour,
    25          receive_email: receiveEmail
    26        }
    27      });
    28    } catch (error) {
    29      logger.error(error);
    30    } finally {
    31      setLoading(false);
    32    }
    33  };
    34
    35  const fetchPreferenceDetails = async () => {
    36    try {
    37      const { data } = await preferencesApi.show(userId);
    38
    39      setNotificationDeliveryHour(data.preference.notification_delivery_hour);
    40      setReceiveEmail(data.preference.receive_email);
    41      setPreferenceId(data.preference.id);
    42    } catch (error) {
    43      logger.error(error);
    44    } finally {
    45      setPageLoading(false);
    46    }
    47  };
    48
    49  useEffect(() => {
    50    fetchPreferenceDetails();
    51  }, []);
    52
    53  if (pageLoading || !userId || !preferenceId) {
    54    return (
    55      <div className="w-screen h-screen">
    56        <PageLoader />
    57      </div>
    58    );
    59  }
    60
    61  return (
    62    <Container>
    63      <PreferenceForm
    64        notificationDeliveryHour={notificationDeliveryHour}
    65        setNotificationDeliveryHour={setNotificationDeliveryHour}
    66        receiveEmail={receiveEmail}
    67        setReceiveEmail={setReceiveEmail}
    68        loading={loading}
    69        updatePreference={updatePreference}
    70      />
    71    </Container>
    72  );
    73};
    74
    75export default MyPreferences;

    Here, fetchPreferenceDetails function uses the show axios API call from preference related axios APIs. Similarly, the updatePreference function will be using the update axios API.

    As you can see we have used a PreferenceForm component which is not yet created. Let's create it:

    1mkdir -p app/javascript/src/components/MyPreferences/Form
    2touch app/javascript/src/components/MyPreferences/Form/index.jsx

    Paste the following lines in the MyPreferences/Form/index.jsx:

    1import React from "react";
    2
    3import Button from "components/Button";
    4import Input from "components/Input";
    5import Select from "react-select";
    6
    7const defaultTimezone = "UTC";
    8
    9const PreferenceForm = ({
    10  notificationDeliveryHour,
    11  setNotificationDeliveryHour,
    12  receiveEmail,
    13  setReceiveEmail,
    14  loading,
    15  updatePreference
    16}) => {
    17  const onHandleDeliveryHourChange = event => {
    18    const regex = /^[0-9\b]*$/;
    19    const deliveryHour = event.target.value;
    20    if (!regex.test(deliveryHour)) return null;
    21
    22    return setNotificationDeliveryHour(deliveryHour);
    23  };
    24
    25  const handleSubmit = event => {
    26    event.preventDefault();
    27    updatePreference();
    28  };
    29
    30  return (
    31    <form className="max-w-lg mx-auto" onSubmit={handleSubmit}>
    32      <div className="flex justify-between text-bb-gray-600 mt-10">
    33        <h1 className="pb-3 mt-5 text-2xl leading-5 font-bold">
    34          Delivery Hour For Mail
    35        </h1>
    36      </div>
    37
    38      <div className="flex justify-between text-gray-700 mb-5">
    39        Set your preferred hour to receive mail with your pending tasks, if any
    40      </div>
    41
    42      <div>
    43        <p className="leading-5 font-medium text-bb-gray-700 text-sm my-3">
    44          Delivery Timezone
    45        </p>
    46        <div className="w-full">
    47          <Select
    48            options={defaultTimezone}
    49            defaultValue={defaultTimezone}
    50            placeholder={defaultTimezone}
    51            isDisabled={true}
    52          />
    53        </div>
    54      </div>
    55
    56      <div className="w-2/6">
    57        <Input
    58          type="number"
    59          label="Delivery Time (Hours)"
    60          placeholder="Enter hour"
    61          min={0}
    62          max={23}
    63          value={notificationDeliveryHour}
    64          onChange={onHandleDeliveryHourChange}
    65        />
    66      </div>
    67
    68      <div className="flex items-center w-2/6 mt-5 ">
    69        <input
    70          type="checkbox"
    71          id="receiveEmail"
    72          checked={receiveEmail}
    73          className="w-4 h-4 text-bb-purple border-gray-300 rounded form-checkbox focus:ring-bb-purple cursor-pointer"
    74          onChange={e => setReceiveEmail(e.target.checked)}
    75        />
    76        <label className="ml-2">Receive Mail</label>
    77      </div>
    78
    79      <div className="w-2/6">
    80        <Button
    81          type="submit"
    82          buttonText="Update Preferences"
    83          loading={loading}
    84        />
    85      </div>
    86    </form>
    87  );
    88};
    89
    90export default PreferenceForm;

    We need to provide an Input component from where the user can set their preferred time for mail delivery.

    We have to constraint the values that this component can accept, when it's type is set to be number.

    Thus let's update our reusable Input component to provide this functionality:

    1import React from "react";
    2import PropTypes from "prop-types";
    3
    4const Input = ({
    5  type = "text",
    6  label,
    7  value,
    8  onChange,
    9  placeholder,
    10  min,
    11  max,
    12  required = true
    13}) => {
    14  return (
    15    <div className="mt-6">
    16      {label && (
    17        <label className="block text-sm font-medium leading-5 text-bb-gray-700">
    18          {label}
    19        </label>
    20      )}
    21      <div className="mt-1 rounded-md shadow-sm">
    22        <input
    23          type={type}
    24          required={required}
    25          value={value}
    26          onChange={onChange}
    27          placeholder={placeholder}
    28          min={min}
    29          max={max}
    30          className="block w-full px-3 py-2 placeholder-gray-400 transition duration-150
    31          ease-in-out border border-gray-300 rounded-md appearance-none focus:outline-none
    32          focus:shadow-outline-blue focus:border-blue-300 sm:text-sm sm:leading-5"
    33        />
    34      </div>
    35    </div>
    36  );
    37};
    38
    39Input.propTypes = {
    40  type: PropTypes.string,
    41  label: PropTypes.string,
    42  value: PropTypes.node,
    43  placeholder: PropTypes.string,
    44  onChange: PropTypes.func,
    45  required: PropTypes.bool,
    46  min: PropTypes.number,
    47  max: PropTypes.number
    48};
    49
    50export default Input;

    Finally, our Preference component is ready to use. Let's add a route for this component in App.jsx and let's also create a link to it in our navbar:

    1//  previous code if any
    2import MyPreferences from "components/MyPreferences";
    3
    4const App = () => {
    5  //  previous code if any
    6  return (
    7    <Router>
    8      <ToastContainer />
    9      <Switch>
    10        {/* previous code if any */}
    11        <Route exact path="/login" component={Login} />
    12        <Route exact path="/my/preferences" component={MyPreferences} />
    13        {/* previous code if any */}
    14      </Switch>
    15    </Router>
    16  );
    17};
    18
    19export default App;

    Now open the Navbar component from components/NavBar/index.jsx and add the following line in order to create a Preferences item next to Create item in navbar:

    1// previous code if any
    2const NavBar = () => {
    3  // previous code if any
    4  <NavItem name="Todos" path="/" />
    5  <NavItem
    6      name="Create"
    7      iconClass="ri-add-fill"
    8      path="/tasks/create"
    9  />
    10  <NavItem name="Preferences" path="/my/preferences" />

    Implementing task mailer

    Let's create the task mailer by running the following Rails command:

    1bundle exec rails g mailer TaskMailer

    Open the application_mailer.rb from app/mailers/application_mailer.rb and, provide the default email from which the mail will be send:

    1class ApplicationMailer < ActionMailer::Base
    2  default from: 'no-reply@granite.com'
    3  layout 'mailer'
    4end

    Mailers have methods called "actions" and they use views to structure their content. A controller generates content like say HTML, which will be sent back to the client, whereas a Mailer creates a message to be delivered via email.

    Now, let's edit the task_mailer.rb which we have created.

    Paste the following lines in task_mailer.rb:

    1class TaskMailer < ApplicationMailer
    2  after_action :create_user_notification, if: -> { @receiver }
    3
    4  def pending_tasks_email(receiver_id)
    5    @receiver = User.find_by(id: receiver_id)
    6    return unless @receiver
    7
    8    @tasks = @receiver.assigned_tasks.pending
    9    mail(to: @receiver.email, subject: "Pending Tasks")
    10  end
    11
    12  private
    13
    14    def create_user_notification
    15      @receiver.user_notifications.create(last_notification_sent_date: Time.zone.today)
    16    end
    17end

    As you can see in the above code we have used after_action which is an Action Mailer Callback. Using an after_action callback enables you to perform operations after successfully sending the mail.

    Once we ensure that mail is successfully sent, we will create an entry in user_notifications with today's date, for the current user.

    Our TaskMailer is ready. Let's create a mailer view, which will contain the content and styling of the email.

    Open app/views and you should be able to see the task_mailer directory. Let's create a pending_tasks_email.html.erb file into it:

    1touch app/views/task_mailer/pending_tasks_email.html.erb

    Paste the following lines into it:

    1<html>
    2  <head>
    3    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type" />
    4  </head>
    5  <body>
    6    <h1>Hi! <%= @receiver.name %></h1>
    7    <p>Here is a list of your pending tasks for today.<br /></p>
    8    <ul>
    9      <% @tasks.each do |task| %>
    10      <li><%= task.title %></li>
    11      <% end %>
    12    </ul>
    13  </body>
    14</html>

    Use case of secrets.yml file

    Rails supports the secrets.yml file to store your application's secrets. It is used to store secrets such as access keys for external APIs or our Redis URL etc.

    This will provide reusability and also will help provide us easier access.

    Let's create this file:

    1touch config/secrets.yml

    Open the file and paste the following lines:

    1default: &default
    2  redis_url: <%= ENV['REDISTOGO_URL'] || ENV['REDIS_URL'] || 'redis://localhost:6379/1' %>

    Creating schedule.yml

    We will create a scheduler file containing the cron syntax for different environments and workers.

    Cron is a standard Unix utility that is used to schedule commands for automatic execution at specific intervals.

    Here we are using cron to send an email notification to the users every day at the default time, i.e., at 10 AM or preferred time, if set by the user.

    Let's create the scheduler file:

    1touch config/schedule.yml

    Open the file and paste the following lines into it:

    1default: &default
    2  todo_notifications_worker:
    3    cron: "0 * * * *"
    4    class: "TodoNotificationsWorker"
    5    queue: "default"
    6
    7development:
    8  todo_notifications_worker:
    9    cron: "* * * * *"
    10    class: "TodoNotificationsWorker"
    11    queue: "default"
    12
    13test:
    14  <<: *default
    15
    16staging:
    17  <<: *default
    18
    19production:
    20  <<: *default

    The default and development are two important keys in the YAML file. Each of those keys invoke a worker.

    In the above code, you can see that the cron value for default and development are different. The reason is that in development we don't need to wait. We need the cron to get triggered as soon as possible.

    The cron value for development executes a cron job every minute, whereas for default the same is executed every hour.

    Here in our YAML file, the default key is the main key because its value will be used in other environments like test, staging, and production.

    We create an alias for the default key using an anchor marked with the ampersand & and then reference that alias using the asterisk *. This is purely a "YAML" way of inheriting existing keys.

    Adding logic to schedule cron jobs

    We have already written down when and what needs to be enqueued, in our schedule.yml file.

    Thus in the Sidekiq initializer, we have to add logic that will read the content from our scheduler file and will enqueue based on the cron defined for the specific environment.

    Also, we have to update the REDIS_URL with the URL we have specified in our secrets file.

    Open sidekiq.rb from config/initializers.

    Add the following lines to this file:

    1# frozen_string_literal: true
    2
    3if Rails.env.test? || Rails.env.heroku?
    4  require 'sidekiq/testing'
    5  Sidekiq::Testing.inline!
    6end
    7
    8Sidekiq::Extensions.enable_delay!
    9
    10Sidekiq.configure_server do |config|
    11  config.redis = { url:  Rails.application.secrets.redis_url, size: 9 }
    12  unless Rails.env.test? || Rails.env.production?
    13    schedule_file = "config/schedule.yml"
    14
    15    if File.exists?(schedule_file)
    16      Sidekiq::Cron::Job.load_from_hash! YAML.load_file(schedule_file)[Rails.env]
    17    end
    18  end
    19end
    20
    21Sidekiq.configure_client do |config|
    22  config.redis = { url: Rails.application.secrets.redis_url, size: 1 }
    23end

    We have currently not enabled mail sending in production environment.

    If possible, we will be handling this case in the upcoming chapters.

    Adding TodoNotifications functionality

    For Sidekiq workers, there already exists a generator for generating a worker file.

    Let's create the TodoNotifications worker:

    1rails generate sidekiq:worker TodoNotificationsWorker

    The sole purpose of the TodoNotificationsWorker is process a TodoNotificationsService, which will take care of other core functionalities.

    Open app/workers/todo_notifications_worker.rb and paste the following:

    1# frozen_string_literal: true
    2
    3class TodoNotificationsWorker
    4  include Sidekiq::Worker
    5
    6  def perform
    7    todo_notification_service = TodoNotificationService.new
    8    todo_notification_service.process
    9  end
    10end

    Let's create our TodoNotificationService to handle the heavy work.

    Rails do not provide any inbuilt generators to create services.

    Currently, we can only manually create the service as well as its test file.

    This service will take care of finding users with pending tasks, whose delivery time falls with the current execution time range, and invoke workers which will send the mail to each user.

    Let's create the service:

    1mkdir -p app/services
    2touch app/services/todo_notification_service.rb

    Open the todo_notification_service.rb file and paste the following:

    1# frozen_string_literal: true
    2
    3class TodoNotificationService
    4  attr_reader :users_to_notify
    5
    6  def initialize
    7    @users_to_notify = get_users_to_notify
    8  end
    9
    10  def process
    11    notify_users
    12  end
    13
    14  private
    15
    16    def get_users_to_notify
    17      users = get_users_with_pending_tasks
    18      users_with_pending_notifications(users)
    19    end
    20
    21    def notify_users
    22      users_to_notify.each do |user|
    23        UserNotificationsWorker.perform_async(user.id)
    24      end
    25    end
    26
    27    def get_users_with_pending_tasks
    28      User.includes(:assigned_tasks, :preference).where(
    29        tasks: { progress: 'pending' },
    30        preferences: {
    31          receive_email: true,
    32          notification_delivery_hour: hours_till_now
    33        }
    34      )
    35    end
    36
    37    def users_with_pending_notifications(users)
    38      no_mail_sent = "user_notifications.last_notification_sent_date <>"
    39      first_time_mailing = "user_notifications.id is NULL"
    40      delivery_condition = "#{no_mail_sent} ? OR #{first_time_mailing}"
    41
    42      users.left_outer_joins(:user_notifications)
    43        .where(delivery_condition, Time.zone.today).distinct
    44    end
    45
    46    def hours_till_now
    47      current_hour = Time.now.utc.hour
    48      (0..current_hour)
    49    end
    50end

    Let's also create a separate worker, named UserNotificationsWorker.

    As per our technical design, we need to ensure that mail delivery for one single user, doesn't break the whole delivery system.

    The first line of defense against this case, is to invoke a separate worker, which is UserNotificationsWorker, for each of the user.

    This way failure of one worker won't affect the other workers, given that Sidekiq runs them in separate threads.

    Let's create UserNotificationsWorker:

    1rails generate sidekiq:worker UserNotificationsWorker

    Open app/workers/user_notifications_worker.rb and paste the following:

    1class UserNotificationsWorker
    2  include Sidekiq::Worker
    3
    4  def perform(user_id)
    5    TaskMailer.delay.pending_tasks_email(user_id)
    6  end
    7end

    Phew! That was a lot of content with loads of good stuff.

    That completes our feature and mail delivery should be working in a development environment.

    We will be writing tests for the same, in the next chapter.

    Moving messages to i18n en.locales

    Recall that we already have a Task not found. translation definition inside en.yml file and it is very similar to Preference not found. which is one of the error translation definitions we have to add.

    Rather than having two similar translations for messages which look like record not found, we can use the variable interpolation feature provided by i18n to pass the name of entity which is not found. This will prevent code duplication.

    Fully replace en.yml with the following lines of code:

    1en:
    2  authorization:
    3    denied: "Access denied. You are not authorized to perform this action."
    4  session:
    5    could_not_auth: "Could not authenticate with the provided credentials."
    6    incorrect_credentials: "Incorrect credentials, try again."
    7  successfully_created: "%{entity} was successfully created!"
    8  successfully_updated: "%{entity} was successfully updated!"
    9  not_found: "%{entity} not found."
    10  task:
    11    slug:
    12      immutable: "is immutable!"
    13  preference:
    14    notification_delivery_hour:
    15      range: "value should be between 0 and 23"
    16  date:
    17    invalid: "must be a valid date"
    18    cant_be_in_past: "can't be in the past"

    Now, the not_found translation in en.yml, will expect a value to be passed for the entity variable. So let's go through our JSON responses and pass that variable with appropriate entity name, wherever we are using the not_found translation key.

    Update the following highlighted line in the load_preference method in PreferencesController:

    1def load_preference
    2  @preference = Preference.find_by(id: params[:id])
    3  unless @preference
    4    render status: :not_found, json: { error: t("not_found", entity:"Preference") }
    5  end
    6end

    Also, update the following highlighted line in the load_task method part of the TasksController:

    1def load_task
    2  @task = Task.find_by(slug: params[:slug])
    3  unless @task
    4    render status: :not_found, json: { error: t("not_found", entity: "Task") }
    5  end
    6end

    We have also used a Task not found translation inside test_not_found_error_rendered_for_invalid_task_slug in TasksControllerTest where we are testing if the correct error is being rendered for an invalid task slug. Let's update the test case to use the updated i18n translation string.

    1def test_not_found_error_rendered_for_invalid_task_slug
    2  invalid_slug = "invalid-slug"
    3
    4  get task_path(invalid_slug), headers: @creator_headers
    5  assert_response :not_found
    6  assert_equal response.parsed_body["error"], t("not_found", entity: "Task")
    7end

    Let's also add the error translations in UserNotification and Preference models.

    Update UserNotification model with the following lines of code:

    1class UserNotification < ApplicationRecord
    2  belongs_to :user
    3
    4  validates :last_notification_sent_date, presence: true
    5  validate :last_notification_sent_date_is_valid_date
    6  validate :last_notification_sent_date_cannot_be_in_the_past
    7
    8  private
    9
    10    def last_notification_sent_date_is_valid_date
    11      if last_notification_sent_date.present?
    12        Date.parse(last_notification_sent_date.to_s)
    13      end
    14    rescue ArgumentError
    15      errors.add(:last_notification_sent_date, t("date.invalid"))
    16    end
    17
    18    def last_notification_sent_date_cannot_be_in_the_past
    19      if last_notification_sent_date.present? && last_notification_sent_date < Time.zone.today
    20        errors.add(:last_notification_sent_date, t("date.cant_be_in_past"))
    21      end
    22    end
    23end

    Update the following highlighted line in the Preference model:

    1class Preference < ApplicationRecord
    2  belongs_to :user
    3
    4  validates :notification_delivery_hour,
    5    presence: true,
    6    numericality: { only_integer: true },
    7    inclusion: {
    8      in: 0..23,
    9      message: I18n.t("preference.notification_delivery_hour.range")
    10    }
    11end

    Testing PreferencesController

    To test the preferences controller, we do not need to create a preference factory since a preference association is created automatically for each user when that new user is created.

    Now, create a new file test/controllers/preferences_controller_test.rb and add the following content:

    1require "test_helper"
    2
    3class PreferencesControllerTest < ActionDispatch::IntegrationTest
    4  def setup
    5    @user = create(:user)
    6    @preference = @user.preference
    7    @headers = headers(@user)
    8  end
    9end

    Now, let us test if the correct preferences are rendered for a valid user. Append the following test case to the PreferencesControllerTest:

    1def test_show_preference_for_a_valid_user
    2  get preference_path(@user.id), headers: @headers
    3  assert_response :ok
    4  assert_equal response.parsed_body["preference"]["id"], @preference.id
    5end

    Let us test if the update action is working as intended. Add the following test cases to the PreferencesControllerTest:

    1def test_update_success
    2  preference_params = { preference: { receive_email: false } }
    3
    4  put preference_path(@preference.id), params: preference_params, headers: @headers
    5  @preference.reload
    6  assert_response :ok
    7  refute @preference.receive_email
    8end
    9
    10def test_update_failure_for_invalid_notification_delivery_hour
    11  preference_params = { preference: { notification_delivery_hour: 24 } }
    12
    13  put preference_path(@preference.id), params: preference_params, headers: @headers
    14  assert_response :unprocessable_entity
    15  assert_equal response.parsed_body["error"], "Notification delivery hour #{t("preference.notification_delivery_hour.range")}"
    16end

    Let's test if the update operation fails with appropriate error response when a request is received to update a preference record with preference_id. Add the following test case to PreferencesControllerTest:

    1def test_not_found_error_rendered_when_updating_preference_with_invalid_preference_id
    2  patch preference_path("invalid_id"), params: { notification_delivery_hour: 14 }, headers: @headers
    3  assert_response :not_found
    4  assert_equal response.parsed_body["error"], t("not_found", entity: "Preference")
    5end

    Testing authorization of preferences

    Let's us test if the authorization is working as expected. Add the following test cases to PreferencesControllerTest:

    1def test_unauthorized_user_cannot_access_preference_of_other_users
    2  unauthorized_user = create(:user)
    3  headers = headers(unauthorized_user)
    4
    5  get preference_path(@preference.id), headers: headers
    6  assert_response :forbidden
    7  assert_equal response.parsed_body["error"], t("authorization.denied")
    8end
    9
    10def test_unauthorized_user_cannot_update_preferences_of_other_users
    11  unauthorized_user = create(:user)
    12  headers = headers(unauthorized_user)
    13  preference_params = { preference: { receive_email: false } }
    14
    15  put preference_path(@preference.id), params: preference_params,headers: headers
    16  assert_response :forbidden
    17  assert_equal response.parsed_body["error"], t("authorization.denied")
    18end

    Now you can try running all the preference test cases and they should be passing:

    1bundle exec rails test -v test/controllers/preferences_controller_test.rb

    Now let's commit these changes:

    1git add -A
    2git commit -m "Added ability to send email to each user with pending tasks"
    Previous
    Next