Handling idempotency when sending emails using Sidekiq

Now that we have covered unit testing and background jobs with Sidekiq, let's add a new feature to send reminder emails to users with outstanding tasks.

In this chapter we will discuss concepts like idempotency, mail delivery windows, setting user preferences, and more.

Features

These are the basic requirements of the feature :

• Send reminder emails to users who have pending tasks.
• Users should have an option to enable/disable the email notifications.
• User can set preferred time to receive this pending tasks notification email.
• If a user has disabled the email notification then that user should not be able to set preferred time to receive email.

Technical design

To implement this feature, we need to introduce the following changes:

On the backend

• For scheduling periodic Sidekiq jobs, we can use the sidekiq-cron gem, which helps us to schedule the worker, which will be sending the mail.

• In the development environment, instead of sending an email to the user's email address, we can preview the email in our browser using the letter_opener gem.

• By default, if the user has set a preferred time then the mail will be sent at that time else mail will be sent at 10 AM.

• We store this preferred time in a table called preferences. This preferences table consists of notification_delivery_hour,should_receive_email, and user columns.

• Create user_notifications table with last_notification_sent_date and user columns.

• In UserNotification model, add validations to ensure that the last_notification_sent_date attribute is present, has a valid date format, and the date is not in the past.

• We will be creating a default preference for the user whenever a new user is created.

• We will be saving the default preference value in a constant DEFAULT_NOTIFICATION_DELIVERY_HOUR, which will be loaded automatically from config/initializers/constants.rb, by Rails, after it loads the framework plus any gems and plugins in our application.

• Add PreferencesController which will add the ability for the user to update their preference for if and when to receive the email.

  Inside the PreferencesController, we will add a custom action called mail which will handle enabling and disabling the email notification.

• Mail sending can often fail due to network errors, among others. When that happens, Sidekiq will retry based on rules we set in the initializer.

On the frontend

• Define preference related axios connectors.

• We will add a MyPreferences component which will handle the preferences of the user.

• Delegate the job of taking in the preferred time for mail delivery to a Form component within the MyPreferences folder.

• Add a function fetchPreferenceDetails in the MyPreferences container component which will fetch preference details like notification_delivery_hour and should_receive_email.

• Add a function updateEmailNotification in the MyPreferences component which will handle enabling/disabling email notifications.

• Similarly, add a function updatePreference in the MyPreferences container component which will update notification_delivery_hour.

Sidekiq worker

To send each batch of reminder emails, we will schedule a TodoNotificationsWorker to run every hour and delegate the job of finding the recipients to a TodoNotificationsService.

This service will ultimately invoke separate UserNotificationWorker's for each recipient it finds.

Let's say we use a single worker for sending all the emails. If this worker fails in the middle of sending the emails the whole email delivery system will be down. The rest of the users will not get any email. This is known as the single point of failure. To avoid this single point of failure we are using separate workers for each recipient. If a separate worker fails then it does not affect the rest of the workers and they can continue sending emails. Thus we can use separate workers and avoid a single point of failure and make our system more reliable.

We will also create a scheduler file to store cron values that will help schedule our TodoNotificationsWorker at the desired time interval for each environment.

The parent Sidekiq worker will then invoke TodoNotificationsService, which will find all the users with pending tasks but no entries in a new table called user_notifications.

What is idempotency?

We need our service to be idempotent, ie: duplicate mail shouldn't be sent every time we retry.

Let's say our email delivery service started and successfully sends 200 emails. But on the delivery of the 201st email, it failed. Now, if we retry sending the failed emails, then it's possible that those 200 emails which were already delivered, will be sent again to the recipient. This only occurs if our service is not idempotent. We can say our service is idempotent only when no duplicate emails are sent each and every time we retry/rerun the service.

To make our service idempotent we need to add a mechanism to keep track of who has received the email and who has not. If we have this data, then we can figure out to whom all we have already sent the emails successfully. This ensures that even if the service fails in between and is restarted, no user should will get the same email twice.

From a RESTful service standpoint, an operation or service call is idempotent when clients can perform a duplicate request harmlessly. In other words, making multiple identical requests has the same effect as making a single request.

Note that while idempotent operations produce the same result on the server, the response itself may not be the same; e.g. a resource's state may change between requests.

To keep track of sent emails we will use the user_notifications table from earlier to associate each user with a last_notification_sent_date.

If a user has an entry for today in user_notifications, it means there's no need to try again.

Action Mailer

Action Mailer allows you to send emails from your application using mailer classes and views. They inherit from ActionMailer::Base and live in app/mailers.

To make sure that the email has truly been sent to the user, we will make use of Action Mailer's after_action callback to safely set the last_notification_sent_date value only after the mail has been sent successfully.

Mailers have methods called "actions" and they use views to structure their content.

A controller generates content like say HTML, which will be sent back to the client, whereas a Mailer creates a message to be delivered via email.

We will be creating the TaskMailer mailer and task_mailer view.

We are now ready to start coding. Let us dive in.

Setting up base

Open the Gemfile.rb and add the following lines:

1# For periodic sidekiq jobs
2gem "sidekiq-cron"
3
4# For opening mails in development env
5gem "letter_opener", :group => :development
Copy

Install the gems:

1bundle install
Copy

For the letter_opener gem to open the mails automatically in our browser, we need to set the following in config/environments/development.rb:

1config.action_mailer.delivery_method = :letter_opener
2config.action_mailer.perform_deliveries = true
Copy

Now any email delivery will pop up in our browser instead of being sent via the internet. The messages will be stored in ./tmp/letter_opener.

Creating models

1bundle exec rails g model user_notification
2bundle exec rails g model preference
Copy

Go to db/migrate folder and paste the following lines of code for the CreateUserNotifications migration:

1class CreateUserNotifications < ActiveRecord::Migration[7.0]
2  def change
3    create_table :user_notifications do |t|
4      t.date :last_notification_sent_date, null: false
5      t.references :user, foreign_key: true
6      t.index [:user_id, :last_notification_sent_date],
7        name: :index_user_preferences_on_user_id_and_notification_sent_date,
8        unique: true
9      t.timestamps
10    end
11  end
12end
Copy

Also, paste the following lines of code for the CreatePreferences migration:

1class CreatePreferences < ActiveRecord::Migration[7.0]
2  def change
3    create_table :preferences do |t|
4      t.integer :notification_delivery_hour
5      t.boolean :should_receive_email, default: true, null: false
6      t.references :user, foreign_key: true
7      t.timestamps
8    end
9  end
10end
Copy

Run the migrations:

1bundle exec rails db:migrate
Copy

Adding validations

Now let's setup UserNotification model and add validations for last_notification_sent_date.

Open app/models/user_notification.rb and add the following lines to it:

1class UserNotification < ApplicationRecord
2  belongs_to :user
3
4  validates :last_notification_sent_date, presence: true
5  validate :last_notification_sent_date_is_valid_date
6  validate :last_notification_sent_date_cannot_be_in_the_past
7
8  private
9
10  def last_notification_sent_date_is_valid_date
11    if last_notification_sent_date.present?
12      Date.parse(last_notification_sent_date.to_s)
13    end
14  rescue ArgumentError
15    errors.add(:last_notification_sent_date, "must be a valid date")
16  end
17
18  def last_notification_sent_date_cannot_be_in_the_past
19    if last_notification_sent_date.present? &&
20        last_notification_sent_date < Time.zone.today
21      errors.add(:last_notification_sent_date, "can't be in the past")
22    end
23  end
24end
Copy

Now let's setup our Preference model along with validations in app/models/preference.rb:

1class Preference < ApplicationRecord
2  belongs_to :user
3
4  validates :notification_delivery_hour, presence: true,
5    numericality: { only_integer: true },
6    inclusion: {
7      in: 0..23,
8      message: "value should be between 0 and 23"
9    }
10end
Copy

Currently, we are only providing the ability to set the preference for when to receive an email notification.

Thus in the above code, we have added the validations for the notification_delivery_hour attribute which makes sure that it's present, and it only accepts integers ranging from 0-23.

Add the following line in config/initializers/constants.rb file::

1module Constants
2  is_sqlite_db = ActiveRecord::Base.connection_db_config.configuration_hash[:adapter] == "sqlite3"
3  DB_REGEX_OPERATOR = is_sqlite_db ? "REGEXP" : "~*"
4  DEFAULT_NOTIFICATION_DELIVERY_HOUR = 10
5end
Copy

We are setting the DEFAULT_NOTIFICATION_DELIVERY_HOUR in our constants.rb file instead of a specific model. This is because the constant is not limited to a specific context and will be used across different files. Thus it makes sense to add it to a global Constants context instead.

Now let's set the associations correctly in our User model too:

1class User < ApplicationRecord
2  # previous code if any
3  has_many :user_notifications, dependent: :destroy, foreign_key: :user_id
4  has_one  :preference, dependent: :destroy, foreign_key: :user_id
5
6  # previous code if any
7  before_save :to_lowercase
8  before_create :build_default_preference
9
10  private
11
12    # previous code if any
13
14    def to_lowercase
15      email.downcase!
16    end
17
18    def build_default_preference
19      self.build_preference(notification_delivery_hour: Constants::DEFAULT_NOTIFICATION_DELIVERY_HOUR)
20    end
21end
Copy

The private method, build_default_preference, will create a default preference for the user whenever a new user is created.

Inside the build_default_preference method we have used the build_preference method. Let's take a look at where it came from.

When we declare a belongs_to or has_one association, the declaring class automatically will have access to a build_association method related to the association. In this case, because of the has_one :preference association in User model, Rails adds the build_preference method to the User class.

In case of has_many or has_and_belongs_to_many associations, we should use the association.build method instead of the build_association method. For example, if we were to declare a has_many :preferences association in the User model, we would have to use the preferences.build method given that Rails won't add the build_preferences method for an association that is not has_one.

Adding preferences for existing users

The build_preference method we had added in the previous section will only add a default preference for new users. The preference for all existing users will remain nil. There are two ways by which we can fix this issue:

1. Deleting all users from the console and creating new users. Newly added users will have a default preference but we shouldn't take this approach because:

   • It's considered a bad practice.

   • Once the application is deployed, we shouldn't delete users from console in production environment.

2. A better way to do this would be to add a migration which will add default preference for all existing users where preference is nil.

If you recall, we have done something similar in the Adding slug to task chapter where we had used a migration to add slugs to existing task records.

Before moving on towards adding a migration to fix the issue, you should attempt to do this by yourself. If you need help you can refer to the Adding slug to task chapter.

Now, let's us add a migration to fix this issue. Run the following command to generate a migration:

1bundle exec rails g migration AddDefaultPreferenceToExistingUsers
Copy

The above command with generate a migration file. Inside the migration file add the following lines of code:

1class AddDefaultPreferenceToExistingUsers < ActiveRecord::Migration[7.0]
2 def up
3    users_with_nil_preference = User.where.missing(:preference)
4    users_with_nil_preference.each do |user|
5      user.send(:build_default_preference)
6      user.save!
7    end
8  end
9
10  def down
11    User.find_each do |user|
12      user.preference.delete
13    end
14  end
15end
Copy

We have used the missing method which performs a left outer join operation on parent and child tables then uses a where clause to identify the missing relations. In this case we are querying all user records which have a missing preference relation and then adding a default preference for all those user records.

You can read more about the missing method from official documentation.

Run the following command to execute the migration:

1bundle exec rails db:migrate
Copy

Creating the controller

Create a new controller for the preference model by running the following command:

1bundle exec rails generate controller Preferences
Copy

Open app/controllers/preferences_controller.rb and add the following lines to it:

1class PreferencesController < ApplicationController
2  before_action :load_preference
3
4  def show
5    respond_with_json({ preference: @preference })
6  end
7
8  def update
9    @preference.update!(preference_params)
10    respond_with_success(t("successfully_updated", entity: "Preference"))
11  end
12
13  def mail
14    @preference.update!(should_receive_email: preference_params[:should_receive_email])
15    respond_with_success(
16      t(
17        "preference.mail.notification_status",
18        status: @preference.should_receive_email ? "enabled" : "disabled"
19      )
20    )
21  end
22
23  private
24
25    def preference_params
26      params.require(:preference).permit(:notification_delivery_hour, :should_receive_email)
27    end
28
29    def load_preference
30      @preference = current_user.preference
31      unless @preference
32        respond_with_error("Preference not found", :not_found)
33      end
34    end
35end
Copy

We have added a mail action in the PreferencesController which will handle the logic to enable and disable the email notifications. Now, the question arises that why we need another action when updating a record should be handled by the update action. We want to do so because the mail action is only concerned with updating the should_receive_email attribute of a preference record and the response for that will also be different from the update action. Therefore, it is better to add a custom action.

We have created a load_preference method to load the preference before the show, update and mail actions are called. Since each user has one associated preference record we can query it using current_user.preference inside the load_preference method. Hence appending the preference id to the preference routes or sending it as a request param while making a request is not required in this case.

Now, let's create the route for the preference. We should declare preference as a singular resource because we do not need a resource identifier in the routes. Open the routes.rb and append the following line:

1Rails.application.routes.draw do
2  constraints(lambda { |req| req.format == :json }) do
3    # ---Previous Routes---
4    resource :preference, only: %i[show update] do
5      patch :mail, on: :collection
6    end
7  end
8
9  root "home#index"
10  get '*path', to: 'home#index', via: :all
11end
Copy

We have added a collection route called mail for the preference resource. Any patch request on preference/mail will invoke the mail action inside the PreferencesController.

Note that we have used a patch verb for the mail route over the put verb. This is so because a put verb is used when the entire entity needs to be updated whereas a patch request is used when the entity needs to be updated partially. In our case, we are only updating the should_receive_email attribute of a preference record thus the use of patch verb.

Why we do not need a preference policy?

Policies prevent any unauthorized access of database records. In case of preferences, a user should not be able to access the preferences of another user. We do not need a policy to ensure this because we are using the preferences association of the User model to load the preference inside the PreferencesController.

Doing so will only load the preference of the current_user thus eliminating the need for authorization using a policy.

Building the UI for user preference

Since we have already added APIs for handling user preference, let's now build the views for it.

Let's create a new file to define all the preference related APIs:

1touch app/javascript/src/apis/preferences.js
Copy

Open the apis/preferences.js and paste the following lines:

1import axios from "axios";
2
3const show = () => axios.get("/preference");
4
5const mail = ({ payload }) => axios.patch(`/preference//mail`, payload);
6
7const update = ({ payload }) => axios.put("/preference", payload);
8
9const preferencesApi = {
10  show,
11  update,
12  mail,
13};
14
15export default preferencesApi;
Copy

Now, let's create the preference component:

1mkdir -p app/javascript/src/components/MyPreferences
2touch app/javascript/src/components/MyPreferences/index.jsx
Copy

Paste the following lines into index.jsx:

1import React, { useState, useEffect } from "react";
2
3import preferencesApi from "apis/preferences";
4import Container from "components/Container";
5import PageLoader from "components/PageLoader";
6import { getFromLocalStorage } from "utils/storage";
7
8import Form from "./Form";
9
10const MyPreferences = () => {
11  const [notificationDeliveryHour, setNotificationDeliveryHour] = useState("");
12  const [shouldReceiveEmail, setShouldReceiveEmail] = useState(true);
13  const [loading, setLoading] = useState(false);
14  const [pageLoading, setPageLoading] = useState(true);
15  const [preferenceId, setPreferenceId] = useState("");
16  const userId = getFromLocalStorage("authUserId");
17
18  const updatePreference = async () => {
19    setLoading(true);
20    try {
21      await preferencesApi.update({
22        payload: {
23          notification_delivery_hour: notificationDeliveryHour,
24          should_receive_email: shouldReceiveEmail,
25        },
26      });
27    } catch (error) {
28      logger.error(error);
29    } finally {
30      setLoading(false);
31    }
32  };
33
34  const updateEmailNotification = async emailNotificationStatus => {
35    setLoading(true);
36    try {
37      await preferencesApi.mail({
38        id: preferenceId,
39        payload: {
40          should_receive_email: emailNotificationStatus,
41        },
42      });
43    } catch (error) {
44      logger.error(error);
45    } finally {
46      setLoading(false);
47    }
48  };
49
50  const fetchPreferenceDetails = async () => {
51    try {
52      const {
53        data: {
54          preference: { notification_delivery_hour, should_receive_email, id },
55        },
56      } = await preferencesApi.show();
57      setNotificationDeliveryHour(notification_delivery_hour);
58      setShouldReceiveEmail(should_receive_email);
59      setPreferenceId(id);
60    } catch (error) {
61      logger.error(error);
62    } finally {
63      setPageLoading(false);
64    }
65  };
66
67  useEffect(() => {
68    fetchPreferenceDetails();
69  }, []);
70
71  if (pageLoading || !userId || !preferenceId) {
72    return (
73      <div className="w-screen h-screen">
74        <PageLoader />
75      </div>
76    );
77  }
78
79  return (
80    <Container>
81      <Form
82        notificationDeliveryHour={notificationDeliveryHour}
83        setNotificationDeliveryHour={setNotificationDeliveryHour}
84        shouldReceiveEmail={shouldReceiveEmail}
85        setShouldReceiveEmail={setShouldReceiveEmail}
86        loading={loading}
87        updatePreference={updatePreference}
88        updateEmailNotification={updateEmailNotification}
89      />
90    </Container>
91  );
92};
93
94export default MyPreferences;
Copy

Here, fetchPreferenceDetails function uses the show axios API call from preference related axios APIs. Similarly, the updatePreference function will be using the update axios API.

As you can see we have used a Form component which is not yet created. Let's create it:

1mkdir -p app/javascript/src/components/MyPreferences/Form
2touch app/javascript/src/components/MyPreferences/Form.jsx
Copy

Paste the following lines in the MyPreferences/Form.jsx:

1import React from "react";
2
3import classnames from "classnames";
4import Button from "components/Button";
5import Input from "components/Input";
6import Select from "react-select";
7
8const defaultTimezone = "UTC";
9
10const Form = ({
11  notificationDeliveryHour,
12  setNotificationDeliveryHour,
13  shouldReceiveEmail,
14  setShouldReceiveEmail,
15  loading,
16  updatePreference,
17  updateEmailNotification,
18}) => {
19  const onHandleDeliveryHourChange = event => {
20    const regex = /^[0-9\b]*$/;
21    const deliveryHour = event.target.value;
22    if (!regex.test(deliveryHour)) return null;
23
24    return setNotificationDeliveryHour(deliveryHour);
25  };
26
27  const handleSubmit = event => {
28    event.preventDefault();
29    if (shouldReceiveEmail) {
30      updatePreference();
31    }
32  };
33
34  const handleEmailNotificationChange = e => {
35    setShouldReceiveEmail(e.target.checked);
36    return updateEmailNotification(e.target.checked);
37  };
38
39  return (
40    <form className="max-w-lg mx-auto" onSubmit={handleSubmit}>
41      <div className="flex justify-between text-bb-gray-600 mt-10 mb-2">
42        <h1 className="pb-3 mt-5 text-2xl leading-5 font-bold">
43          Pending Tasks in Email
44        </h1>
45      </div>
46
47      <div
48        className={classnames("flex  items-baseline space-x-1", {
49          "text-bb-gray-700": shouldReceiveEmail,
50          "text-bb-gray-600": !shouldReceiveEmail,
51        })}
52      >
53        <input
54          type="checkbox"
55          checked={shouldReceiveEmail}
56          id="shouldReceiveEmail"
57          onChange={handleEmailNotificationChange}
58        />
59        <span>
60          Send me a daily email of the pending tasks assigned to me.
61          <br /> No email will be sent if there are no pending tasks.
62        </span>
63      </div>
64
65      <div
66        className={classnames("flex space-x-4 items-center", {
67          "text-bb-gray-700": shouldReceiveEmail,
68          "text-bb-gray-600": !shouldReceiveEmail,
69        })}
70      >
71        <p className="text-sm font-medium mt-6 leading-5 ">
72          Delivery Time (Hours)
73        </p>
74        <Input
75          type="number"
76          placeholder="Enter hour"
77          disabled={!shouldReceiveEmail}
78          min={0}
79          max={23}
80          value={notificationDeliveryHour}
81          onChange={onHandleDeliveryHourChange}
82        />
83        <p className="mt-6 font-extrabold">(UTC)</p>
84      </div>
85
86      <div className="w-2/6">
87        <Button
88          type="submit"
89          buttonText="Schedule Email"
90          className={classnames({
91            "cursor-not-allowed bg-opacity-60": !shouldReceiveEmail,
92          })}
93          loading={loading}
94        />
95      </div>
96    </form>
97  );
98};
99
100export default Form;
Copy

The Form component contains a checkbox for enabling and disabling the email notifications. When the checkbox value changes, updateEmailNotification function defined inside the MyPreferences component will be called and it will in turn make an API for mail action in the PreferencesController.

There is a updatePreference function as well which will be called upon submitting the form and it will make an API request to update the logged in user's preference.

We need to provide an Input component from where the user can set their preferred time for mail delivery.

We have to constraint the values that this component can accept, when it's type is set to be number.

Thus let's update our reusable Input component to provide this functionality:

1import React from "react";
2
3import classnames from "classnames";
4import PropTypes from "prop-types";
5
6const Input = ({
7  type = "text",
8  label,
9  value,
10  onChange,
11  placeholder,
12  min,
13  max,
14  required = true,
15}) => {
16  return (
17    <div className="mt-6">
18      {label && (
19        <label className="block text-sm font-medium leading-5 text-bb-gray-700">
20          {label}
21        </label>
22      )}
23      <div className="mt-1 rounded-md shadow-sm">
24        <input
25          type={type}
26          required={required}
27          value={value}
28          onChange={onChange}
29          placeholder={placeholder}
30          min={min}
31          max={max}
32          className={classnames(
33            "block w-full px-3 py-2 placeholder-gray-400 transition duration-150 ease-in-out border border-gray-300 rounded-md appearance-none focus:outline-none focus:shadow-outline-blue focus:border-blue-300 sm:text-sm sm:leading-5",
34            [className]
35          )}
36        />
37      </div>
38    </div>
39  );
40};
41
42Input.propTypes = {
43  type: PropTypes.string,
44  label: PropTypes.string,
45  value: PropTypes.node,
46  placeholder: PropTypes.string,
47  onChange: PropTypes.func,
48  required: PropTypes.bool,
49  min: PropTypes.number,
50  max: PropTypes.number,
51};
52
53export default Input;
Copy

Finally, our Preference component is ready to use. Let's add a route for this component in App.jsx and let's also create a link to it in our navbar:

1//  previous code if any
2import MyPreferences from "components/MyPreferences";
3
4const App = () => {
5  //  previous code if any
6  return (
7    <Router>
8      <ToastContainer />
9      <Switch>
10        {/* previous code if any */}
11        <Route exact path="/login" component={Login} />
12        <Route exact path="/my/preferences" component={MyPreferences} />
13        {/* previous code if any */}
14      </Switch>
15    </Router>
16  );
17};
18
19export default App;
Copy

Now open the Navbar component from components/NavBar/index.jsx and add the following line to create a Preferences item in navbar:

1// previous code if any
2const NavBar = () => {
3  // previous code if any
4  return (
5    <nav className="bg-white shadow">
6      <div className="px-2 mx-auto max-w-7xl sm:px-4 lg:px-8">
7        <div className="flex justify-between h-16">
8          <div className="flex px-2 lg:px-0">
9            <div className="hidden lg:flex">
10              <NavItem name="Todos" path="/dashboard" />
11              <NavItem
12                name="Add"
13                iconClass="ri-add-fill"
14                path="/tasks/create"
15              />
16            </div>
17          </div>
18          <div className="flex items-center justify-end gap-x-4">
19            <span
20              className="inline-flex items-center px-2 pt-1 text-sm
21              font-regular leading-5 text-bb-gray-600 text-opacity-50
22              transition duration-150 ease-in-out border-b-2
23              border-transparent focus:outline-none
24              focus:text-bb-gray-700"
25            >
26              <Link to="/my/preferences">Preferences </Link>
27            </span>
28            <span
29              className="inline-flex items-center px-2 pt-1 text-sm
30              font-regular leading-5 text-bb-gray-600 text-opacity-50
31              transition duration-150 ease-in-out border-b-2
32              border-transparent focus:outline-none
33              focus:text-bb-gray-700"
34            >
35              {userName}
36            </span>
37            <a
38              onClick={handleLogout}
39              className="inline-flex items-center px-1 pt-1 text-sm
40              font-semibold leading-5 text-bb-gray-600 text-opacity-50
41              transition duration-150 ease-in-out border-b-2
42              border-transparent hover:text-bb-gray-600 focus:outline-none
43              focus:text-bb-gray-700 cursor-pointer"
44            >
45              LogOut
46            </a>
47          </div>
48        </div>
49      </div>
50    </nav>
51  );
52};
Copy

Implementing task mailer

Let's create the task mailer by running the following Rails command:

1bundle exec rails g mailer TaskMailer
Copy

Open the application_mailer.rb from app/mailers/application_mailer.rb and, provide the default email from which the mail will be send:

1class ApplicationMailer < ActionMailer::Base
2  default from: 'no-reply@granite.com'
3  layout 'mailer'
4end
Copy

Mailers have methods called "actions" and they use views to structure their content. A controller generates content like say HTML, which will be sent back to the client, whereas a Mailer creates a message to be delivered via email.

Now, let's edit the task_mailer.rb which we have created.

Paste the following lines in task_mailer.rb:

1class TaskMailer < ApplicationMailer
2  after_action :create_user_notification, if: -> { @receiver }
3
4  def pending_tasks_email(receiver_id)
5    @receiver = User.find_by(id: receiver_id)
6    return unless @receiver
7
8    @tasks = @receiver.assigned_tasks.pending
9    mail(to: @receiver.email, subject: "Pending Tasks")
10  end
11
12  private
13
14    def create_user_notification
15      @receiver.user_notifications.create(last_notification_sent_date: Time.zone.today)
16    end
17end
Copy

As you can see in the above code we have used after_action which is an Action Mailer Callback. Using an after_action callback enables you to perform operations after successfully sending the mail.

Once we ensure that mail is successfully sent, we will create an entry in user_notifications with today's date, for the current user.

Our TaskMailer is ready. Let's create a mailer view, which will contain the content and styling of the email.

Open app/views and you should be able to see the task_mailer directory. Let's create a pending_tasks_email.html.erb file into it:

1touch app/views/task_mailer/pending_tasks_email.html.erb
Copy

Paste the following lines into it:

1<html>
2  <head>
3    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type" />
4  </head>
5  <body>
6    <h1>Hi! <%= @receiver.name %></h1>
7    <p>Here is a list of your pending tasks for today.<br /></p>
8    <ul>
9      <% @tasks.each do |task| %>
10      <li><%= task.title %></li>
11      <% end %>
12    </ul>
13  </body>
14</html>
Copy

Use case of secrets.yml file

Rails supports the secrets.yml file to store your application's secrets. It is used to store secrets such as access keys for external APIs or our Redis URL etc.

This will provide reusability and also will help provide us easier access.

Let's create this file:

1touch config/secrets.yml
Copy

Open the file and paste the following lines:

1default: &default
2  redis_url: <%= ENV['REDISTOGO_URL'] || ENV['REDIS_URL'] || 'redis://localhost:6379/1' %>
Copy

Creating schedule.yml

We will create a scheduler file containing the cron syntax for different environments and workers.

Cron is a standard Unix utility that is used to schedule commands for automatic execution at specific intervals.

Here we are using cron to send an email notification to the users every day at the default time, i.e., at 10 AM or preferred time, if set by the user.

Let's create the scheduler file:

1touch config/schedule.yml
Copy

Open the file and paste the following lines into it:

1default: &default
2  todo_notifications_worker:
3    cron: "0 * * * *"
4    class: "TodoNotificationsWorker"
5    queue: "default"
6
7development:
8  todo_notifications_worker:
9    cron: "* * * * *"
10    class: "TodoNotificationsWorker"
11    queue: "default"
12
13test:
14  <<: *default
15
16staging:
17  <<: *default
18
19production:
20  <<: *default
Copy

The default and development are two important keys in the YAML file. Each of those keys invoke a worker.

In the above code, you can see that the cron value for default and development are different. The reason is that in development we don't need to wait. We need the cron to get triggered as soon as possible.

The cron value for development executes a cron job every minute, whereas for default the same is executed every hour.

Here in our YAML file, the default key is the main key because its value will be used in other environments like test, staging, and production.

We create an alias for the default key using an anchor marked with the ampersand & and then reference that alias using the asterisk *. This is purely a "YAML" way of inheriting existing keys.

Adding logic to schedule cron jobs

We have already written down when and what needs to be enqueued, in our schedule.yml file.

Thus in the Sidekiq initializer, we have to add logic that will read the content from our scheduler file and will enqueue based on the cron defined for the specific environment.

Also, we have to update the REDIS_URL with the URL we have specified in our secrets file.

Open sidekiq.rb from config/initializers.

Add the following lines to this file:

1# frozen_string_literal: true
2
3if Rails.env.test? || Rails.env.heroku?
4  require 'sidekiq/testing'
5  Sidekiq::Testing.inline!
6end
7
8Sidekiq::Extensions.enable_delay!
9
10Sidekiq.configure_server do |config|
11  config.redis = { url:  Rails.application.secrets.redis_url, size: 9 }
12  unless Rails.env.test? || Rails.env.production?
13    schedule_file = "config/schedule.yml"
14
15    if File.exists?(schedule_file)
16      Sidekiq::Cron::Job.load_from_hash! YAML.load_file(schedule_file)[Rails.env]
17    end
18  end
19end
20
21Sidekiq.configure_client do |config|
22  config.redis = { url: Rails.application.secrets.redis_url, size: 1 }
23end
Copy

We have currently not enabled mail sending in production environment.

If possible, we will be handling this case in the upcoming chapters.

Adding TodoNotifications functionality

Let's create the TodoNotifications worker:

1mkdir -p app/workers
2touch app/workers/todo_notifications_worker.rb
Copy

The sole purpose of the TodoNotificationsWorker is process a TodoNotificationsService, which will take care of other core functionalities.

Open app/workers/todo_notifications_worker.rb and paste the following:

1# frozen_string_literal: true
2
3class TodoNotificationsWorker
4  include Sidekiq::Worker
5
6  def perform
7    todo_notification_service = TodoNotificationService.new
8    todo_notification_service.process
9  end
10end
Copy

Let's create our TodoNotificationService to handle the heavy work.

Rails do not provide any inbuilt generators to create services.

Currently, we can only manually create the service as well as its test file.

This service will take care of finding users with pending tasks, whose delivery time falls with the current execution time range, and invoke workers which will send the mail to each user.

Let's create the service:

1mkdir -p app/services
2touch app/services/todo_notification_service.rb
Copy

Open the todo_notification_service.rb file and paste the following:

1# frozen_string_literal: true
2
3class TodoNotificationService
4  attr_reader :users_to_notify
5
6  def initialize
7    @users_to_notify = get_users_to_notify
8  end
9
10  def process
11    notify_users
12  end
13
14  private
15
16    def get_users_to_notify
17      users = get_users_with_pending_tasks
18      users_with_pending_notifications(users)
19    end
20
21    def notify_users
22      users_to_notify.each do |user|
23        UserNotificationsWorker.perform_async(user.id)
24      end
25    end
26
27    def get_users_with_pending_tasks
28      User.includes(:assigned_tasks, :preference).where(
29        tasks: { progress: 'pending' },
30        preferences: {
31          should_receive_email: true,
32          notification_delivery_hour: hours_till_now
33        }
34      )
35    end
36
37    def users_with_pending_notifications(users)
38      no_mail_sent = "user_notifications.last_notification_sent_date <>"
39      first_time_mailing = "user_notifications.id is NULL"
40      delivery_condition = "#{no_mail_sent} ? OR #{first_time_mailing}"
41
42      users.left_outer_joins(:user_notifications)
43        .where(delivery_condition, Time.zone.today).distinct
44    end
45
46    def hours_till_now
47      current_hour = Time.now.utc.hour
48      (0..current_hour)
49    end
50end
Copy

Let's discuss the initialize method before moving ahead. initialize method in Ruby is basically a constructor which we use to initialize instance variables during object creation. The initialize method gets invoked implicitly even if no instance variables are initialized.

If there is an attribute which you need to access in multiple methods defined inside the class then you should declare it as an instance variable and initialize it in the initialize method. At BigBinary we use the attr_accessor macro to access the instance methods inside the class.

Let's also create a separate worker, named UserNotificationsWorker.

As per our technical design, we need to ensure that mail delivery for one single user, doesn't break the whole delivery system.

The first line of defense against this case, is to invoke a separate worker, which is UserNotificationsWorker, for each of the user.

This way failure of one worker won't affect the other workers, given that Sidekiq runs them in separate threads.

Let's create UserNotificationsWorker:

1touch app/workers/user_notifications_worker.rb
Copy

Open app/workers/user_notifications_worker.rb and paste the following:

1class UserNotificationsWorker
2  include Sidekiq::Worker
3
4  def perform(user_id)
5    TaskMailer.delay.pending_tasks_email(user_id)
6  end
7end
Copy

Phew! That was a lot of content with loads of good stuff.

That completes our feature and mail delivery should be working in a development environment.

We will be writing tests for the same, in the next chapter.

Moving messages to i18n en.locales

Recall that we already have a Task not found. translation definition inside en.yml file and it is very similar to Preference not found. which is one of the error translation definitions we have to add.

Rather than having two similar translations for messages which look like record not found, we can use the variable interpolation feature provided by i18n to pass the name of entity which is not found. This will prevent code duplication.

Fully replace en.yml with the following lines of code:

1en:
2  generic_error: "Something went wrong. Please try again later."
3  authorization:
4    denied: "Access denied. You are not authorized to perform this action."
5  session:
6    could_not_auth: "Could not authenticate with the provided credentials."
7    incorrect_credentials: "Incorrect credentials, try again."
8  successfully_created: "%{entity} was successfully created!"
9  successfully_updated: "%{entity} was successfully updated!"
10  successfully_deleted: "%{entity} was successfully deleted!"
11  not_found: "Couldn't find %{entity}"
12  task:
13    slug:
14      immutable: "is immutable!"
15  preference:
16    notification_delivery_hour:
17      range: "value should be between 0 and 23"
18    mail:
19      notification_status: "Email notifications %{status}."
20  date:
21    invalid: "must be a valid date"
22    cant_be_in_past: "can't be in the past"
Copy

Now, the not_found translation in en.yml, will expect a value to be passed for the entity variable. So let's go through our JSON responses and pass that variable with appropriate entity name, wherever we are using the not_found translation key.

Update the following highlighted line in the load_preference method in PreferencesController:

1def load_preference
2  @preference = current_user.preference
3  unless @preference
4    respond_with_error(t("not_found", entity:"Preference"), :not_found)
5  end
6end
Copy

We have also used a Task not found translation inside test_not_found_error_rendered_for_invalid_task_slug in TasksControllerTest where we are testing if the correct error is being rendered for an invalid task slug. Let's update the test case to use the updated i18n translation string.

1def test_not_found_error_rendered_for_invalid_task_slug
2  invalid_slug = "invalid-slug"
3
4  get task_path(invalid_slug), headers: @creator_headers
5  assert_response :not_found
6  response_json = response.parsed_body
7  assert_equal t("not_found", entity: "Task"), response_json["error"]
8end
Copy

Update the test_shouldnt_create_comment_without_task inside the CommentsControllerTest like so:

1def test_shouldnt_create_comment_without_task
2  post comments_path, params: { comment: { content: "This is a comment", task_id: "" } }, headers: @headers
3  assert_response :not_found
4  response_json = response.parsed_body
5  assert_equal t("not_found", entity: "Task"), response_json["error"]
6end
Copy

Update the test_should_respond_with_not_found_error_if_user_is_not_present inside the SessionsControllerTest like so:

1def test_should_respond_with_not_found_error_if_user_is_not_present
2  non_existent_email = "this_email_does_not_exist_in_db@example.email"
3  post session_path, params: { login: { email: non_existent_email, password: "welcome" } }, as: :json
4  assert_response :not_found
5  response_json = response.parsed_body
6  assert_equal t("not_found", entity: "User"), response_json["error"]
7end
Copy

Update the handle_api_exception method inside the ApplicationController like so:

1def handle_api_exception(exception)
2  case exception
3  # rest of the exception cases
4
5  when ActiveRecord::RecordNotFound
6    respond_with_error(t("not_found", entity: exception.model), :not_found)
7end
Copy

Let's also add the error translations in UserNotification and Preference models.

Update UserNotification model with the following lines of code:

1class UserNotification < ApplicationRecord
2  belongs_to :user
3
4  validates :last_notification_sent_date, presence: true
5  validate :last_notification_sent_date_is_valid_date
6  validate :last_notification_sent_date_cannot_be_in_the_past
7
8  private
9
10    def last_notification_sent_date_is_valid_date
11      if last_notification_sent_date.present?
12        Date.parse(last_notification_sent_date.to_s)
13      end
14    rescue ArgumentError
15      errors.add(:last_notification_sent_date, t("date.invalid"))
16    end
17
18    def last_notification_sent_date_cannot_be_in_the_past
19      if last_notification_sent_date.present? && last_notification_sent_date < Time.zone.today
20        errors.add(:last_notification_sent_date, t("date.cant_be_in_past"))
21      end
22    end
23end
Copy

Update the following highlighted line in the Preference model:

1class Preference < ApplicationRecord
2  belongs_to :user
3
4  validates :notification_delivery_hour,
5    presence: true,
6    numericality: { only_integer: true },
7    inclusion: {
8      in: 0..23,
9      message: t("preference.notification_delivery_hour.range")
10    }
11end
Copy

Testing PreferencesController

To test the preferences controller, we do not need to create a preference factory since a preference association is created automatically for each user when that new user is created.

Now, create a new file test/controllers/preferences_controller_test.rb and add the following content:

1require "test_helper"
2
3class PreferencesControllerTest < ActionDispatch::IntegrationTest
4  def setup
5    @user = create(:user)
6    @preference = @user.preference
7    @headers = headers(@user)
8  end
9end
Copy

Now, let us test if the correct preferences are rendered for a valid user. Append the following test case to the PreferencesControllerTest:

1def test_show_preference_for_a_valid_user
2  get preference_path, headers: @headers
3  assert_response :ok
4  assert_equal @preference.id, response.parsed_body["preference"]["id"]
5end
Copy

We should also add a test case to check if correct response is being rendered if the preference is not found. Add the following test case to the PreferencesControllerTest:

1def test_not_found_error_rendered_if_preference_is_not_present
2  @user.preference = nil
3  get preference_path, headers: @headers
4  assert_response :not_found
5end
Copy

Let us test if the update action is working as intended. Add the following test cases to the PreferencesControllerTest:

1def test_update_success
2  preference_params = { preference: { should_receive_email: false } }
3
4  put preference_path, params: preference_params, headers: @headers
5  @preference.reload
6  assert_response :ok
7  refute @preference.should_receive_email
8end
9
10def test_update_failure_for_invalid_notification_delivery_hour
11  preference_params = { preference: { notification_delivery_hour: 24 } }
12
13  put preference_path, params: preference_params, headers: @headers
14  assert_response :unprocessable_entity
15  assert_equal "Notification delivery hour #{t("preference.notification_delivery_hour.range")}", response.parsed_body["error"]
16end
Copy

Let us test the mail action. Add the following test case to the PreferencesControllerTest:

1def test_update_success_mail
2    preference_params = { preference: { should_receive_email: false } }
3
4    patch mail_preference_path, params: preference_params, headers: @headers
5    @preference.reload
6    assert_response :ok
7    refute @preference.should_receive_email
8  end
Copy

Now you can try running all the preference test cases and they should be passing:

1bundle exec rails test -v test/controllers/preferences_controller_test.rb
Copy

Now let's commit these changes:

1git add -A
2git commit -m "Added ability to send email to each user with pending tasks"
Copy