Regular Rake tasks with systemd timers

16.9.2020 by Premysl Donat

Sooner or later you will run into a situation in which you need to run some server task regularly. I often need this for some maintenance or data synchronization.

For Rails apps there are multiple ways how to do this - ActiveJob, cron, gems like clockwork or others. However, if you run on Linux with systemd, you might not need these. In such case you can use systemd's own feature called timers. Let's look at some basics.

The code

We need three separate files. First, the Rake task:

# /my_rails_app/lib/tasks/some_task.rake
namespace :some_group do
  desc 'Does some regular thing'

  task do_it: :environment do
    Rails.log('running some regular code')
    # ..some other code..
  end
end

Then we need systemd service unit. This is group of directives in .INI syntax placed in a file with .service suffix in /etc/systemd/system/ directory. Below is a very basic example but it can be adjusted to a great extent. 

# /etc/systemd/system/my_task.service
[Unit]
Description=Do some regular thing

[Service]
# This directive lets you setup needed environment variables
#   they can be separated by space
Environment="RAILS_ENV=production" "OTHER=one two three"

# This has to be set to our Rails app root otherwise Rails
#   wouldn't find the Rake task
WorkingDirectory=/my_rails_app

# The command to start the Rake task itself with absolute path
ExecStart=/my_rails_app/bin/rails some_group:do_it

# User under which the command will run. Root if not specified
User=fred

And last we need timer unit. This is another type of systemd's units which is used for scheduling something to run at specific time events:

# /etc/systemd/system/my_task.timer
[Unit]
Description=Run my regular task every day

[Timer]
# When the task should run. It has neat aliases like 'daily'
#   but also powerful syntax like cron
OnCalendar=daily

By default this will run invoke service with same name but of course it can configured differently.

The commands

We have the code, but how to get it moving? For that we will use systemd's command line utility systemctl.

To start the service, type systemctl start my_task.service in your terminal. You won't see any output as it will run the service in separate process. This way you can start the service manually as often as you want.

Now in order to run our service automatically we need to start the timer. For that, type systemctl start my_task.timer. Now the timer is running and it will invoke our service as configured. But we need to do one more thing. If we want our timer to run after we reboot the machine we have to enable it: systemctl enable my_task.timer. Now systemd will take care of starting the timer after reboots.

In order to check what state the timer(or other systemd unit) is in, run systemctl status my_task.timer. You should see output like this:

terminal output of systemctl status my_task.timer

Notice how the next run time is also showed.

When something doesn't work

Most common problems you might encounter are wrong service directives or missing environment configuration.

When debugging, ensure especially that:

  • the Rake task runs ok when manually invoked
  • you prepared the environment our Rake task needs through service directives properly
  • environment for our Rake task is properly prepared through service directives
  • the service runs ok when invoked manually with systemctl start my_task.service
  • the timer is active and/or enabled(check with systemctl status my_task.timer)

Then there are logs. systemd will log it's events, errors and also standard output into system journal by default. To query these logs use command line utility journalctl.

Also when you make changes in existing unit files in /etc/systemd/system/, you have to let it know to systemd via systemctl daemon-reload.

Environment and configuration

Providing environment variables through Environment directive is straighforward. But there are also other handy directives like EnvironmentFile which you might point to your .env file in Rails root. This directive and other possibilities are described nicely here.

Then there is security. The very basic thing is to run the service under some other user than root. But then the services can be restricted in a lot of different ways through many other directives. Search for "securing systemd services" if you need to.

For the timer interval configuration consult this. Timers can also run in intervals relative to their's or system's events.

Logging and exceptions

By default the service will log standard output and error to system journal. But it that can be changed for example with StandardOutput=append:/my_rails_app/log/production.log and StandardError=append:/my_rails_app/log/production.log. Another useful directive migth be SyslogIdentifier=taxbox_recurring_tasks, which sets label showed in journal(normally it would be name of the executed command which is 'rails').

Beware with application exceptions. Maybe you only catch and notify about app exceptions with some king of Rack middleware meaning it won't work in Rake tasks. That might be the case if you use exception_notification. If that's the case you have to catch and and notify about exceptions manually.

In conclusion and resources

I like three things about systemd timers. First, if i run systemd Linux, the timers are simply there and i don't need to install and learn anything else. Second, when i work with services and units and timers i get more familiar with systemd itself. Definitelly a benefit since my whole system depends on it. Three, debugging is easier than with cron.

A couple of useful resources:

  1. other great tutorial for systemd timers: https://opensource.com/article/20/7/systemd-timers
  2. systemd project homepage: https://systemd.io/
  3. systemd man pages: https://www.man7.org/linux/man-pages/man1/systemd.1.html
  4. list of all systemd unit directiveshttps://www.man7.org/linux/man-pages/man7/systemd.directives.7.html
  5. systemd command line cheat sheet: https://access.redhat.com/sites/default/files/attachments/12052018_systemd_6.pdf
  6. journalctl command tutorial: https://linuxhandbook.com/journalctl-command/