Elixir by Example

Testing with ExUnit

Every command ran when the site was built; the output is real Download project

Mix Projects showed the basics of ExUnit, Elixir's test framework: a test module, test blocks of asserts, doctests and mix test. This page covers what a real test suite needs on top of that:

  1. Grouping tests with describe, and the different kinds of assertion.
  2. Shared setup, and the context each test receives.
  3. What a failing test prints.
  4. Which tests may run at the same time (async), and which may not.
  5. Testing printed output and log messages.
  6. Running some of the tests, and measuring how much code they run.

The project

span parses and formats durations such as 1h30m. It has no dependencies.

mix.exs
defmodule Span.MixProject do
  use Mix.Project

  def project do
    [
      app: :span,
      version: "0.1.0",
      elixir: "~> 1.20",
      elixirc_options: [warnings_as_errors: true],
      deps: []
    ]
  end

  def application do
    [extra_applications: [:logger]]
  end
end
.formatter.exs
[
  inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"]
]

mix test --cover (below) writes a cover/ directory, so it's ignored along with the build output.

.gitignore
/_build/
/cover/
/deps/

parse/1 returns {:ok, seconds} or {:error, reason}, and parse!/1 raises instead. A configurable maximum (a week unless the :max_seconds key of the application environment says otherwise) rejects durations that are too long. The iex> examples in the docs run as doctests.

lib/span.ex
defmodule Span do
  @moduledoc "Parses and formats durations such as `1h30m`."

  @units %{"d" => 86_400, "h" => 3_600, "m" => 60, "s" => 1}

  @doc """
  Parses a duration into seconds.

      iex> Span.parse("1h30m")
      {:ok, 5400}

      iex> Span.parse("90x")
      {:error, {:unknown_unit, "x"}}
  """
  @spec parse(String.t()) :: {:ok, non_neg_integer()} | {:error, term()}
  def parse(""), do: {:error, :empty}

  def parse(text) do
    with {:ok, seconds} <- parse(text, 0) do
      max = Application.get_env(:span, :max_seconds, 7 * 86_400)
      if seconds <= max, do: {:ok, seconds}, else: {:error, {:too_long, seconds}}
    end
  end

  defp parse("", total), do: {:ok, total}

  defp parse(text, total) do
    case Integer.parse(text) do
      {n, rest} when n >= 0 ->
        {unit, rest} = String.split_at(rest, 1)

        case Map.fetch(@units, unit) do
          {:ok, size} -> parse(rest, total + n * size)
          :error when unit == "" -> {:error, {:missing_unit, n}}
          :error -> {:error, {:unknown_unit, unit}}
        end

      _ ->
        {:error, {:not_a_number, text}}
    end
  end

  @doc "Like `parse/1`, but raises `ArgumentError` on bad input."
  @spec parse!(String.t()) :: non_neg_integer()
  def parse!(text) do
    case parse(text) do
      {:ok, seconds} -> seconds
      {:error, reason} -> raise ArgumentError, "bad duration #{inspect(text)}: #{inspect(reason)}"
    end
  end

  @doc """
  Formats seconds, biggest unit first, skipping zero parts.

      iex> Span.format(5400)
      "1h 30m"
  """
  @spec format(non_neg_integer()) :: String.t()
  def format(0), do: "0s"

  def format(seconds) do
    {parts, 0} =
      Enum.reduce(~w(d h m s), {[], seconds}, fn unit, {parts, left} ->
        size = @units[unit]
        count = div(left, size)
        parts = if count > 0, do: ["#{count}#{unit}" | parts], else: parts
        {parts, rem(left, size)}
      end)

    parts |> Enum.reverse() |> Enum.join(" ")
  end
end

Span.CLI.main/1 adds up the durations it's given and prints the total. A bad duration is logged with Logger.warning/1 and skipped. require Logger makes Logger's macros available.

lib/span/cli.ex
defmodule Span.CLI do
  @moduledoc "Adds up durations given on the command line."

  require Logger

  @doc "Prints the sum of the durations in `args`; logs and skips bad ones."
  @spec main([String.t()]) :: :ok
  def main(args) do
    total =
      Enum.reduce(args, 0, fn arg, total ->
        case Span.parse(arg) do
          {:ok, seconds} ->
            total + seconds

          {:error, reason} ->
            Logger.warning("skipping #{inspect(arg)}: #{inspect(reason)}")
            total
        end
      end)

    IO.puts("total: " <> Span.format(total))
  end
end

The tests

test_helper.exs runs before any test. exclude: [:broken] skips every test tagged :broken; the failures section below uses that tag for tests that are wrong on purpose.

test/test_helper.exs
ExUnit.start(exclude: [:broken])

async: true lets this module's tests run at the same time as tests from other async modules. Within one module, tests always run one after another.

describe groups tests under a name, which becomes part of each test's name (test parse/1 adds up every unit). Inside:

  • assert left == right compares two values.
  • assert pattern = value matches instead: it passes for any value that fits the pattern and binds the pattern's variables, so the next line can check unit on its own. A literal in the pattern (5 in {:missing_unit, 5}) must match exactly.
  • refute passes when its expression is falsy. match?/2 returns true or false, so refute match?(...) checks a value doesn't fit a pattern.
  • assert_raise passes only if the function raises that exception, here with a message matching a regular expression.

test is a macro, so a for comprehension can define a test per row of a table. The comprehension runs at compile time, and unquote/1 puts each row's values into the test's body. Each test needs a unique name, so the name includes the input.

setup runs before each test in its module (or, inside a describe, each test of that group) and returns a map that's merged into the test's context. A test receives the context as its second argument and pattern-matches what it needs. setup_all is the same, but runs once per module.

test/span_test.exs
defmodule SpanTest do
  use ExUnit.Case, async: true
  doctest Span

  describe "parse/1" do
    test "adds up every unit" do
      assert Span.parse("1d2h3m4s") == {:ok, 93_784}
    end

    test "returns the reason for bad input" do
      assert {:error, {:unknown_unit, unit}} = Span.parse("5w")
      assert unit == "w"
      assert {:error, {:missing_unit, 5}} = Span.parse("5")
      refute match?({:ok, _}, Span.parse("h"))
    end

    for {text, seconds} <- [{"0s", 0}, {"45s", 45}, {"2m", 120}, {"1h1s", 3_601}] do
      test "parses #{text}" do
        assert Span.parse(unquote(text)) == {:ok, unquote(seconds)}
      end
    end
  end

  describe "parse!/1" do
    test "raises on bad input" do
      assert_raise ArgumentError, ~r/bad duration "5w"/, fn -> Span.parse!("5w") end
    end
  end

  describe "format/1" do
    setup do
      %{week: 7 * 86_400}
    end

    test "skips zero parts", %{week: week} do
      assert Span.format(week + 5) == "7d 5s"
    end

    test "round-trips through parse!/1", %{week: week} do
      for seconds <- [0, 59, 3_600, week] do
        assert seconds |> Span.format() |> String.replace(" ", "") |> Span.parse!() == seconds
      end
    end
  end
end

The limit test changes the application environment, which is global: every test running at the same moment sees it. So this module is async: false, and ExUnit runs it on its own, after the async modules. on_exit/1 registers a function that runs after the test, whether it passed or failed, and puts the environment back.

test/span/limit_test.exs
defmodule Span.LimitTest do
  # Changes the application environment, which every test can see: not async.
  use ExUnit.Case, async: false

  setup do
    Application.put_env(:span, :max_seconds, 60)
    on_exit(fn -> Application.delete_env(:span, :max_seconds) end)
  end

  test "rejects durations over the configured limit" do
    assert Span.parse("1m") == {:ok, 60}
    assert Span.parse("61s") == {:error, {:too_long, 61}}
  end
end

capture_io/1 runs a function and returns everything it printed, instead of letting it reach the terminal. It captures the output of the test's own process only, so it's safe in async tests. with_log/1 returns the function's result and the log messages sent while it ran. Logs are not per process: in an async test they can include messages from other tests running at the same time, so the assertion uses =~ (does the string contain this?) rather than ==.

test/span/cli_test.exs
defmodule Span.CLITest do
  use ExUnit.Case, async: true
  import ExUnit.CaptureIO
  import ExUnit.CaptureLog

  test "prints the total" do
    assert capture_io(fn -> Span.CLI.main(["1h", "30m"]) end) == "total: 1h 30m\n"
  end

  test "logs and skips a bad duration" do
    {output, log} = with_log(fn -> capture_io(fn -> Span.CLI.main(["1h", "oops"]) end) end)
    assert output == "total: 1h\n"
    assert log =~ ~s(skipping "oops")
  end
end

Run the tests

As on Mix Projects, the output on this page uses a fixed seed and one test at a time, and drops the line with the run time, so it's the same on every run. t wraps that up. It keeps mix test's exit status, which is non-zero when a test fails. On your machine, plain mix test does the same job.

$ t() { s=0; mix test --seed 0 --max-cases 1 "$@" > test.log 2>&1 || s=$?; grep -v '^Finished in' test.log; return $s; }
$ t
Compiling 2 files (.ex)
Generated span app
Running ExUnit with seed: 0, max_cases: 1
Excluding tags: [:broken]

...............

Result: 15 passed (3 doctests, 12 tests), 3 excluded

Fifteen tests ran: three doctests and twelve tests, four of them from the for comprehension. The three :broken tests were excluded.

Failures

These tests are wrong on purpose. @moduletag :broken tags every test in the module.

test/broken_test.exs
defmodule BrokenTest do
  # Wrong on purpose, to show what failures look like. Excluded by default.
  use ExUnit.Case, async: true
  @moduletag :broken

  test "a comparison" do
    assert Span.format(3_700) == "1h 2m"
  end

  test "a pattern" do
    assert {:ok, _} = Span.parse("1h30")
  end

  test "an exception" do
    assert_raise ArgumentError, fn -> Span.parse!("1h") end
  end
end

--only broken runs just the tests with that tag. It works by excluding all tests (the :test tag every test has) and including :broken back.

$ t --only broken
Running ExUnit with seed: 0, max_cases: 1
Excluding tags: [:test]
Including tags: [:broken]



  1) test a comparison (BrokenTest)
     test/broken_test.exs:6
     Assertion with == failed
     code:  assert Span.format(3700) == "1h 2m"
     left:  "1h 1m 40s"
     right: "1h 2m"
     stacktrace:
       test/broken_test.exs:7: (test)



  2) test a pattern (BrokenTest)
     test/broken_test.exs:10
     match (=) failed
     code:  assert {:ok, _} = Span.parse("1h30")
     left:  {:ok, _}
     right: {:error, {:missing_unit, 30}}
     stacktrace:
       test/broken_test.exs:11: (test)



  3) test an exception (BrokenTest)
     test/broken_test.exs:14
     Expected exception ArgumentError but nothing was raised
     code: assert_raise ArgumentError, fn -> Span.parse!("1h") end
     stacktrace:
       test/broken_test.exs:15: (test)



Result: 0/3 passed, 15 excluded
Failed: 3 tests

Every failure names its test and its file and line, then shows the code, both sides, and the stack trace:

  • With ==, left is what the code returned and right what the test expected.
  • A failed match shows the pattern on the left and the value that didn't fit on the right.
  • assert_raise reports that nothing was raised: "1h" is a valid duration.

Pick the tests to run

A file name runs the tests in that file, and file:line runs the test at that line. ExUnit adds a location filter for it, so the other tests in the file are excluded:

$ t test/span_test.exs:11
Running ExUnit with seed: 0, max_cases: 1
Excluding tags: [:test]
Including tags: [location: {"test/span_test.exs", 11}]

.

Result: 1 passed, 11 excluded

The excluded tests are the ones in that file only; other files aren't loaded at all. Tags select tests too: --only tag, --exclude tag, and --include tag to bring back a tag that test_helper.exs excludes.

Coverage

--cover compiles the project with Erlang's cover tool, which counts every line each test runs, and prints the share of lines run per module:

$ t --cover
Cover compiling modules ...
Running ExUnit with seed: 0, max_cases: 1
Excluding tags: [:broken]

...............

Result: 15 passed (3 doctests, 12 tests), 3 excluded

Generating cover results ...

| Percentage | Module     |
|------------|------------|
|     95.24% | Span       |
|    100.00% | Span.CLI   |
|------------|------------|
|     96.30% | Total      |

Generated HTML coverage results in "cover" directory

cover/Elixir.Span.html marks the one line in Span that no test runs: def parse(""), do: {:error, :empty}. Nothing tests the empty string. High coverage doesn't prove the tests check the right things, but a line no test runs is certainly not checked.

Pitfall: a test that changes something global isn't safe in an async module: the application environment, a named process or ETS table, a file, or the Logger level. Another test running at the same time sees the change. Put such tests in an async: false module, as Span.LimitTest does.

Pitfall: assert {:ok, _} = value passes for any {:ok, ...}, while assert value == {:ok, 5} needs exactly that. Use a match when only the shape matters, and == when the value does.

Pitfall: don't wait for something with Process.sleep/1 in a test. It makes the suite slow, and still fails now and then on a busy machine. Testing Processes shows how to wait for messages instead.

Improve this page on GitHub