Elixir by Example

Transactions and Constraints

Every command ran when the site was built; the output is real Download project

Everything on the earlier pages kept its state in process memory, and everything there was lost on a restart. Data that must survive goes in a database, and the database brings a guarantee no process gives you on its own: a transaction. Several writes either all happen or none of them do, even if the program crashes halfway through.

This page uses Ecto with SQLite, so the whole database is one file and there's nothing to install. Everything shown works the same on Postgres.

On this page:

  1. A transfer between two accounts as one transaction with Ecto.Multi.
  2. Constraints in the database (no negative balance, one row per idempotency key) that make bad writes impossible, not just unlikely.
  3. A lost update, where two read-modify-write cycles overwrite each other, and optimistic locking, which catches it.

The project

mix.exs
defmodule Bank.MixProject do
  use Mix.Project

  def project do
    [app: :bank, version: "0.1.0", elixir: "~> 1.20", deps: deps()]
  end

  def application do
    [mod: {Bank.Application, []}]
  end

  defp deps do
    [{:ecto_sqlite3, "~> 0.26"}]
  end
end

Query logging is switched off so the output below shows results, not SQL with timings.

config/config.exs
import Config

config :bank, ecto_repos: [Bank.Repo]

# `log: false` keeps query logs (with their timings) out
# of the output.
config :bank, Bank.Repo, database: "bank.db", log: false
lib/bank/repo.ex
defmodule Bank.Repo do
  use Ecto.Repo, otp_app: :bank, adapter: Ecto.Adapters.SQLite3
end
lib/bank/application.ex
defmodule Bank.Application do
  use Application

  @impl true
  def start(_type, _args) do
    Supervisor.start_link([Bank.Repo], strategy: :one_for_one, name: Bank.Supervisor)
  end
end

The migration puts the rules in the database itself: a CHECK that a balance can't go below zero, and a unique index on the transfer's idempotency key. Rules in the database hold no matter which code writes, and no matter how many writers run at once.

priv/repo/migrations/20261108000000_create_accounts.exs
defmodule Bank.Repo.Migrations.CreateAccounts do
  use Ecto.Migration

  def change do
    create table(:accounts) do
      add :name, :string, null: false
      # The database itself refuses a negative balance.
      add :balance, :integer,
        null: false,
        check: %{name: "balance_not_negative", expr: "balance >= 0"}

      add :lock_version, :integer, null: false, default: 1
    end

    create unique_index(:accounts, [:name])

    create table(:transfers) do
      add :key, :string, null: false
      add :from_id, references(:accounts), null: false
      add :to_id, references(:accounts), null: false
      add :amount, :integer, null: false
    end

    # One row per idempotency key, enforced by the database.
    create unique_index(:transfers, [:key])
  end
end

check_constraint/3 and unique_constraint/3 don't check anything themselves. They tell Ecto that when the database refuses a write for that reason, it should return a changeset error instead of raising.

lib/bank/account.ex
defmodule Bank.Account do
  use Ecto.Schema
  import Ecto.Changeset

  schema "accounts" do
    field :name, :string
    field :balance, :integer
    field :lock_version, :integer, default: 1
  end

  def open(name, balance) do
    %Bank.Account{}
    |> change(name: name, balance: balance)
    |> unique_constraint(:name)
  end

  # `check_constraint/3` turns the database's refusal into
  # a changeset error instead of an exception.
  def adjust(account, delta) do
    account
    |> change(balance: account.balance + delta)
    |> check_constraint(:balance, name: :balance_not_negative, message: "would go negative")
  end

  # Same, plus optimistic locking: the update only
  # applies if `lock_version` is still what we read.
  def adjust_locked(account, delta) do
    account
    |> adjust(delta)
    |> optimistic_lock(:lock_version)
  end
end
lib/bank/transfer.ex
defmodule Bank.Transfer do
  use Ecto.Schema
  import Ecto.Changeset

  schema "transfers" do
    field :key, :string
    field :from_id, :integer
    field :to_id, :integer
    field :amount, :integer
  end

  def record(key, from, to, amount) do
    %Bank.Transfer{}
    |> change(key: key, from_id: from.id, to_id: to.id, amount: amount)
    |> unique_constraint(:key)
  end
end

Bank.transfer/4 builds an Ecto.Multi: a list of named steps run in one transaction. On failure it returns the step that failed, its errors, and the steps that had already run (which were rolled back).

lib/bank.ex
defmodule Bank do
  alias Bank.{Account, Repo, Transfer}
  alias Ecto.Multi

  def open(name, balance), do: Repo.insert(Account.open(name, balance))

  def balances do
    Account |> Repo.all() |> Map.new(&{&1.name, &1.balance})
  end

  def account(name), do: Repo.get_by!(Account, name: name)

  # A transfer is three writes: debit, credit, and a row
  # recording it under its idempotency key. `Ecto.Multi`
  # runs them in one transaction. If any step fails,
  # none of them happened.
  def transfer(key, from_name, to_name, amount) do
    from = account(from_name)
    to = account(to_name)

    Multi.new()
    |> Multi.update(:debit, Account.adjust(from, -amount))
    |> Multi.update(:credit, Account.adjust(to, amount))
    |> Multi.insert(:record, Transfer.record(key, from, to, amount))
    |> Repo.transaction()
    |> case do
      {:ok, _changes} -> :ok
      {:error, step, changeset, _done} -> {:error, step, errors(changeset)}
    end
  end

  # Read-modify-write with optimistic locking. If someone
  # else changed the row since `account` was read, the
  # update matches no row and raises instead of
  # overwriting their change.
  def deposit_locked(account, amount) do
    Repo.update(Account.adjust_locked(account, amount))
  rescue
    Ecto.StaleEntryError -> {:error, :stale}
  end

  defp errors(changeset) do
    Ecto.Changeset.traverse_errors(changeset, fn {message, _opts} -> message end)
  end
end

Set up

Fetching and compiling the dependencies prints download progress and timings, so that output is hidden. ecto.create makes the SQLite file and ecto.migrate creates the tables.

$ mix deps.get > /dev/null
$ mix deps.compile > /dev/null 2>&1
$ mix compile
$ mix ecto.create --quiet
$ mix ecto.migrate --quiet
$ mix run -e 'Bank.open("alice", 100); Bank.open("bob", 50); IO.inspect(Bank.balances())'
Compiling 5 files (.ex)
Generated bank app

14:47:54.065 [error] Exqlite.Connection (#PID<0.194.0> ("db_conn_1")) failed to connect: ** (Exqlite.Error) database is locked
%{"alice" => 100, "bob" => 50}

All or nothing

A transfer that works: three writes, one transaction.

$ mix run -e 'IO.inspect(Bank.transfer("t-1", "alice", "bob", 30)); IO.inspect(Bank.balances())'
:ok
%{"alice" => 70, "bob" => 80}

Bob tries to send more than he has. The CHECK refuses the debit, and the transaction stops at the first step.

$ mix run -e 'IO.inspect(Bank.transfer("t-2", "bob", "alice", 500))'
{:error, :debit, %{balance: ["would go negative"]}}

Now the same transfer as before is sent again, under the same idempotency key, as a client would after a lost reply. This time the debit and the credit both succeed, and the last step, recording the key, fails on the unique index. The transaction rolls back the two writes that had already happened. The balances are exactly as they were: the retry did nothing, which is what a retry of finished work should do.

$ mix run -e 'IO.inspect(Bank.transfer("t-1", "alice", "bob", 30)); IO.inspect(Bank.balances())'
{:error, :record, %{key: ["has already been taken"]}}
%{"alice" => 70, "bob" => 80}

Lost updates

A transaction protects writes that happen together. It doesn't protect a read-modify-write spread over time: read the row, compute a new value in Elixir, write it back. Two such cycles that read the same row both write "balance + 10" computed from the same old balance, and one deposit disappears.

$ mix run -e '
$ first = Bank.account("alice")
$ second = Bank.account("alice")
$ {:ok, _} = Bank.Repo.update(Bank.Account.adjust(first, 10))
$ {:ok, _} = Bank.Repo.update(Bank.Account.adjust(second, 10))
$ IO.inspect(Bank.balances(), label: "two deposits of 10")
$ '
two deposits of 10: %{"alice" => 80, "bob" => 80}

Alice had 70 and got two deposits, but has 80. Optimistic locking fixes it. Every update also checks and bumps lock_version (UPDATE ... WHERE id = ? AND lock_version = ?). The second writer's version is out of date, its update matches no row, and Ecto raises Ecto.StaleEntryError (turned into {:error, :stale} in Bank). The caller re-reads the row and tries again.

$ mix run -e '
$ first = Bank.account("alice")
$ second = Bank.account("alice")
$ {:ok, _} = Bank.deposit_locked(first, 10)
$ IO.inspect(Bank.deposit_locked(second, 10), label: "second writer")
$ IO.inspect(Bank.balances())
$ '
second writer: {:error, :stale}
%{"alice" => 90, "bob" => 80}

Pitfall: validations in a changeset (validate_number/3) check the value Elixir computed, from data that may already be stale. Only a constraint in the database sees every write, including concurrent ones and ones from other programs. Keep the rule in the database and use check_constraint/3 to turn the refusal into an error.

Pitfall: the simplest fix for a lost update is to let the database do the arithmetic: Repo.update_all(from(a in Account, where: a.id == ^id), inc: [balance: 10]). There's no read, so there's nothing to go stale. Optimistic locking is for changes Elixir has to compute.

Pitfall: don't do slow work (HTTP calls, sending email) inside a transaction. It holds locks the whole time, and if the transaction rolls back, the email was still sent. Record the intent in the transaction and do the work afterwards; the outbox pattern is exactly that.