Elixir by Example

Testing Processes

Every command ran when the site was built; the output is real Download project

Testing with ExUnit tested plain functions: call, compare, done. A process is harder to test. It has to be started and stopped, it answers with messages that arrive later, it may run on a timer, and a process with a name is shared by every test. This page tests a small GenServer and the application around it:

  1. A fresh process for each test with start_supervised!/1.
  2. Checking messages a process sends, without sleeping.
  3. Driving a timer by sending its message yourself.
  4. Testing a crash.
  5. Testing the processes the application starts by name.

The project

quota has one GenServer, Quota.Meter. It counts hits per key, forgets the counts every interval milliseconds, and tells a notify process when a key goes over its limit.

mix.exs
defmodule Quota.MixProject do
  use Mix.Project

  def project do
    [
      app: :quota,
      version: "0.1.0",
      elixir: "~> 1.20",
      elixirc_options: [warnings_as_errors: true],
      deps: []
    ]
  end

  def application do
    [
      mod: {Quota.Application, []},
      extra_applications: [:logger]
    ]
  end
end
.formatter.exs
[
  inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"]
]

Everything the meter needs comes in as options, including the interval and the process to notify. That's what makes it testable: a test can pick a short limit, a reset that never fires on its own, and itself as the process to notify. Keyword.pop/2 takes the optional name out of the options, and the meter is registered under it only if there is one. GenServer.server() and GenServer.on_start() are types GenServer defines for "a pid or a name" and "what start_link returns".

The notification is sent before hit/2 replies. When hit/2 returns, the message is already in the notified process's mailbox.

lib/quota/meter.ex
defmodule Quota.Meter do
  @moduledoc """
  Counts hits per key and forgets them every `interval` milliseconds. When a
  key goes over `limit`, sends `{:over_limit, key}` to the `notify` process.
  """
  use GenServer

  @doc """
  Options: `limit` and `interval` (required), `notify` (a pid, optional) and
  `name` (optional).
  """
  @spec start_link(keyword()) :: GenServer.on_start()
  def start_link(opts) do
    {name, opts} = Keyword.pop(opts, :name)
    GenServer.start_link(__MODULE__, opts, if(name, do: [name: name], else: []))
  end

  @doc "Records a hit and returns the key's count since the last reset."
  @spec hit(GenServer.server(), term()) :: pos_integer()
  def hit(meter, key), do: GenServer.call(meter, {:hit, key})

  @doc "The key's count since the last reset."
  @spec count(GenServer.server(), term()) :: non_neg_integer()
  def count(meter, key), do: GenServer.call(meter, {:count, key})

  @impl true
  def init(opts) do
    state = %{
      limit: Keyword.fetch!(opts, :limit),
      interval: Keyword.fetch!(opts, :interval),
      notify: Keyword.get(opts, :notify),
      counts: %{}
    }

    schedule_reset(state.interval)
    {:ok, state}
  end

  @impl true
  def handle_call({:hit, key}, _from, state) do
    count = Map.get(state.counts, key, 0) + 1
    if count == state.limit + 1 and state.notify, do: send(state.notify, {:over_limit, key})
    {:reply, count, put_in(state.counts[key], count)}
  end

  def handle_call({:count, key}, _from, state) do
    {:reply, Map.get(state.counts, key, 0), state}
  end

  @impl true
  def handle_info(:reset, state) do
    schedule_reset(state.interval)
    {:noreply, %{state | counts: %{}}}
  end

  defp schedule_reset(interval), do: Process.send_after(self(), :reset, interval)
end

The application starts one meter, registered as Quota.Meter, with a limit of 100 hits per minute.

lib/quota/application.ex
defmodule Quota.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      {Quota.Meter, name: Quota.Meter, limit: 100, interval: :timer.minutes(1)}
    ]

    Supervisor.start_link(children, strategy: :one_for_one, name: Quota.Supervisor)
  end
end

A process per test

mix test starts the application before any test runs, so the named meter is already up. The tests of the meter itself don't use it: each test starts its own, so no test sees another's counts.

start_supervised!/1 takes a child spec, as a supervisor's children list does (see Supervisors), and starts the process under a supervisor that ExUnit runs for each test. It returns the pid, and stops the process when the test ends, passed or failed, before the next test starts. A process started with start_link in the test is linked to it and dies when the test ends too, but asynchronously: it may still be running, holding its name or its files, when the next test starts.

The supervised process isn't linked to the test, so its crash doesn't fail the test by itself, and the test supervisor restarts it like any supervisor would. start_link_supervised!/1 links it as well, for tests where any crash should fail the test.

These meters have no name, so the module can be async: true: tests running at the same time each talk to their own meter by pid.

test/test_helper.exs
ExUnit.start(exclude: [:broken])
test/quota/meter_test.exs
defmodule Quota.MeterTest do
  use ExUnit.Case, async: true
  alias Quota.Meter

  # Each test gets its own meter, with a reset an hour away: it never fires
  # during a test unless the test sends `:reset` itself.
  setup do
    meter = start_supervised!({Meter, limit: 2, interval: :timer.hours(1), notify: self()})
    %{meter: meter}
  end

  test "counts hits per key", %{meter: meter} do
    assert Meter.hit(meter, "a") == 1
    assert Meter.hit(meter, "a") == 2
    assert Meter.count(meter, "b") == 0
  end

  test "tells the subscriber once a key is over the limit", %{meter: meter} do
    Meter.hit(meter, "a")
    Meter.hit(meter, "a")
    refute_received {:over_limit, _}

    Meter.hit(meter, "a")
    assert_received {:over_limit, "a"}

    Meter.hit(meter, "a")
    refute_received {:over_limit, _}
  end

  test "forgets every count on reset", %{meter: meter} do
    Meter.hit(meter, "a")
    send(meter, :reset)
    assert Meter.count(meter, "a") == 0
  end

  @tag :capture_log
  test "a bad call crashes the meter and its caller", %{meter: meter} do
    ref = Process.monitor(meter)
    assert {{:function_clause, _stack}, _call} = catch_exit(GenServer.call(meter, :oops))
    assert_receive {:DOWN, ^ref, :process, ^meter, {:function_clause, _stack}}
  end
end

Messages, in order of the tests:

  • Already sent. The meter notifies before it replies, so after hit/2 returns, the message is either in the test's mailbox or was never sent. assert_received checks the mailbox now, without waiting, and refute_received checks that no matching message is there. assert_received removes the message it matches, so the last refute_received proves the notification came only once.
  • The timer. The reset is an ordinary :reset message the meter sends itself with Process.send_after/3. The test sends :reset itself instead of waiting a minute. Messages from one process to another arrive in the order they were sent, so the meter handles :reset before the count/2 call that follows it.
  • Arriving later. A process's :DOWN message comes from the runtime when the process dies, at some point after the call fails. assert_receive waits for a matching message, 100 ms by default (a second argument changes it), and fails if none comes. ^ref and ^meter pin the values already bound, so only this meter's :DOWN matches.

The crash test calls the meter with a message it has no clause for, so handle_call/3 raises and the meter dies. A process that dies makes its callers' GenServer.call/3 exit too, and catch_exit/1 turns that exit into a value the test can match. The reason is the raw Erlang error, {:function_clause, stacktrace}, not an exception struct. The dying GenServer logs an error report, with pids in it. @tag :capture_log captures the logs of that one test and prints them only if it fails.

The named process

A test of the meter the application started has to share it with every other test that might touch Quota.Meter. This module is async: false, and its setup resets the meter first, so earlier hits don't leak in.

The second test shows the other half of the problem: a meter started by a test can't take a name that's already taken. start_supervised/1 (no !) returns the error instead of raising.

test/quota/application_test.exs
defmodule Quota.ApplicationTest do
  # Uses the meter the application started under the name Quota.Meter, which
  # every test can reach: not async.
  use ExUnit.Case, async: false

  setup do
    send(Quota.Meter, :reset)
    :ok
  end

  test "the application starts a named meter" do
    assert [{Quota.Meter, pid, :worker, [Quota.Meter]}] =
             Supervisor.which_children(Quota.Supervisor)

    assert Process.whereis(Quota.Meter) == pid
    assert Quota.Meter.hit(Quota.Meter, "a") == 1
  end

  test "a second meter can't take the name" do
    spec = {Quota.Meter, name: Quota.Meter, limit: 1, interval: :timer.hours(1)}
    assert {:error, {:already_started, pid}} = start_supervised(spec)
    assert pid == Process.whereis(Quota.Meter)
  end
end

Run them

The same t helper as on Testing with ExUnit: a fixed seed, one test at a time, and no run-time line.

$ t() { s=0; mix test --seed 0 --max-cases 1 "$@" > test.log 2>&1 || s=$?; grep -v '^Finished in' test.log; return $s; }
$ t
Compiling 2 files (.ex)
Generated quota app
Running ExUnit with seed: 0, max_cases: 1
Excluding tags: [:broken]

......

Result: 6 passed, 1 excluded

A message that never comes

When assert_receive gives up, it shows what is in the mailbox. This test waits for a notification about the wrong key:

test/broken_test.exs
defmodule BrokenTest do
  # Wrong on purpose, to show what a missing message looks like. Excluded by
  # default.
  use ExUnit.Case, async: true
  @moduletag :broken

  test "waits for a message that never comes" do
    meter = start_supervised!({Quota.Meter, limit: 1, interval: :timer.hours(1), notify: self()})
    Quota.Meter.hit(meter, "a")
    Quota.Meter.hit(meter, "a")
    assert_receive {:over_limit, "b"}
  end
end
$ t --only broken
Running ExUnit with seed: 0, max_cases: 1
Excluding tags: [:test]
Including tags: [:broken]



  1) test waits for a message that never comes (BrokenTest)
     test/broken_test.exs:7
     Assertion failed, no matching message after 100ms
     Showing 1 of 1 message in the mailbox
     code: assert_receive {:over_limit, "b"}
     mailbox:
       pattern: {:over_limit, "b"}
       value:   {:over_limit, "a"}
     stacktrace:
       test/broken_test.exs:11: (test)



Result: 0/1 passed, 6 excluded
Failed: 1 test

When the message did arrive but with a different shape than the pattern, the listing shows the difference at once.

Pitfall: a test that Process.sleep/1s until something has happened is slow and still fails sometimes on a busy machine. Wait for a message with assert_receive, or make a call to the process: it handles the call after every message the caller sent it before, so the reply means those have all been handled.

Pitfall: refute_receive waits its full timeout (100 ms by default) before passing, on every run. When the message would already be in the mailbox, as with the meter's notification, refute_received checks at once.

Pitfall: a process registered under a fixed name can't be started once per test, and async tests would share it. Make the name an option, as the meter does, and let tests start unnamed processes.

Improve this page on GitHub