Sidekiq, systemd and Capistrano

October 10, 2019

In this blog post I am going to write about my approach on deploying a Ruby On Rails application and dealing with Sidekiq (job scheduler written in Ruby). This approach is probably suitable for a small/medium application like Todor Grudev - Outmuscle Me. I would not recommend following this for a project with a lot of usage, Outmuscle Me at the time of that post has 262 Users, so you can imagine the infrastructure is not really demanding. Heavy operations like syncing activities with Garmin and Fitbit were actually the reasons I decided to rollout Sidekiq in the first place.

The current setup consists of a single server that takes care of serving the Rails content, storing the database and running the Sidekiq. Yeah, I am aware that this is not a good idea, but yet again I am running a start up here :). Deploying the app with Capistrano (a tool for running scripts) and in order to run Sidekiq I am using systemd. If you are not familiar with systemd, here is how Wiki describes it:

The systemd software suite provides fundamental building blocks for a Linux operating system. It includes the systemd “System and Service Manager”, an init system used to bootstrap user space and manage user processes.

Check A brief overview and history of systemd — the Linux process manager - By for more information, the important part of this sentence is that it manages user processes. In my case it will take care of keeping Sidekiq alive. It will also take care of stoping it gracefully. My server is running Ubuntu, things may vary based on the operating system (ex. the folder you put the service script in). I am also using rbenv to control the Ruby versions so you may need to change some paths in the configurations below. Another prerequisite is having a deploy user that Capistrano would use. I believe that if you are searching the internet and have reached this article you probably already know all those stuff, so let’s jump into the necessesary steps needed to do the magic:

Step 1 - Create a service

Add a new file in /lib/systemd/system called sidekiq.service, content:

[Unit]
Description=Outmuscle Me Production Sidekiq
After=syslog.target network.target

[Service]
WorkingDirectory=/data/apps/outmuscleme/current

User=deploy
Group=deploy

Environment=RBENV_ROOT=/usr/local/rbenv
EnvironmentFile=/data/apps/outmuscleme/shared/.env

ExecStart=/usr/local/rbenv/bin/rbenv exec bundle exec sidekiq -e production

RestartSec=1
Restart=on-failure

SyslogIdentifier=sidekiq

[Install]
WantedBy=multi-user.target

Step 2 - Enable the service

/bin/systemctl enable sidekiq

Here systemctl is a command, which is the central management tool for controlling the init system.

Step 3 - Allow the deploy user to restart / kill

As a root user type visudo it will open up the vi editor loaded with the current /etc/sudoers file, at the end of the file you can add the following (or preferably you can create a new file at /etc/sudoers.d/sidekiq-users and add it there)

%deploy ALL= NOPASSWD: /bin/systemctl restart sidekiq
%deploy ALL= NOPASSWD: /bin/systemctl kill -s TSTP sidekiq

Step 4 - Modify Capistrano’s deploy file (config/deploy.rb)

set :sidekiq_service_name, "sidekiq"

namespace :sidekiq do
  desc "Quiet sidekiq (stop fetching new tasks from Redis)"
  task :quiet do
    on roles(:app) do
      execute :sudo, :systemctl, :kill, "-s", "TSTP", fetch(:sidekiq_service_name)
    end
  end

  desc "Restart sidekiq service"
  task :restart do
    on roles(:app) do
      execute :sudo, :systemctl, :restart, fetch(:sidekiq_service_name)
    end
  end
end

after "deploy:starting", "sidekiq:quiet"
after "deploy:published", "sidekiq:restart"

According to Sidekiq ‘s documentation - TSTP tells Sidekiq to “quiet” as it will be shutting down at some point in the near future. It will stop fetching new jobs but continue working on current jobs. TSTP + TERM guarantee shutdown within a time period. Best practice is to send TSTP at the start of a deploy and TERM at the end of a deploy. So as you can see we are doing the quiet operation at the starting of the deployment and the terminating / restarting at the publishing state. That’s it, as a final step verify that everything is running correctly.

I would really appreciate some feedback and I would love to see how you’ve setup the background processing of you application, leave me a comment ;)

Additional sources:
Sidekiq in Production | playbook
Sidekiq’s signals
Sidekiq’s service example