KV as a release · Elixir by Example

Up to now KV lived inside scripts that started their own nodes. Deployed, it's a Mix project built into a release, and every server runs the same build. What differs per server (its node name, the cookie, which nodes to cluster with) comes from the environment at boot, through config/runtime.exs.

On this page:

KV as a Mix project: the replicated store from step 8, :pg and libcluster, all started by the application's supervisor.
One release, built once.
Started twice, as nodes kv_a and kv_b, with different environment variables. The nodes find each other through libcluster and replicate writes.

The project

libcluster is the only dependency.

mix.exs

defmodule KV.MixProject do
  use Mix.Project

  def project do
    [app: :kv, version: "0.1.0", elixir: "~> 1.20", deps: deps()]
  end

  def application do
    [mod: {KV.Application, []}]
  end

  defp deps do
    [{:libcluster, "~> 3.4"}]
  end
end

The node list is configuration like any other, so it comes from the environment, in runtime.exs.

config/runtime.exs

import Config

# The nodes to cluster with, read when each node boots:
# CLUSTER_NODES=kv_a@127.0.0.1,kv_b@127.0.0.1
nodes =
  System.fetch_env!("CLUSTER_NODES")
  |> String.split(",", trim: true)
  |> Enum.map(&String.to_atom/1)

config :kv, cluster_nodes: nodes

The supervisor starts :pg (the replicas find each other through it), then libcluster, then this node's replica.

lib/kv/application.ex

defmodule KV.Application do
  use Application

  @impl true
  def start(_type, _args) do
    hosts = Application.fetch_env!(:kv, :cluster_nodes)
    topologies = [kv: [strategy: Cluster.Strategy.Epmd, config: [hosts: hosts, timeout: 500]]]

    children = [
      %{id: :pg, start: {:pg, :start_link, []}},
      {Cluster.Supervisor, [topologies, [name: KV.ClusterSupervisor]]},
      KV.Replica
    ]

    Supervisor.start_link(children, strategy: :one_for_one, name: KV.Supervisor)
  end
end

The replica is step 8's, plus count/0, which we use to tell when both nodes can see each other.

lib/kv/replica.ex

defmodule KV.Replica do
  @moduledoc "One replica of the store; every write goes to all of them (step 8)."
  use GenServer

  def start_link(_opts), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def put(key, value) do
    for replica <- :pg.get_members(:kv_replicas) do
      :ok = GenServer.call(replica, {:put, key, value})
    end

    :ok
  end

  def get(key), do: GenServer.call(__MODULE__, {:get, key})

  @doc "How many replicas this node can see."
  def count, do: length(:pg.get_members(:kv_replicas))

  @impl true
  def init(map) do
    :ok = :pg.join(:kv_replicas, self())
    {:ok, map}
  end

  @impl true
  def handle_call({:put, k, v}, _from, map), do: {:reply, :ok, Map.put(map, k, v)}
  def handle_call({:get, k}, _from, map), do: {:reply, Map.get(map, k), map}
end

Build once

mix deps.get prints how long dependency resolution took, which changes on every run, so its output is silenced here.

$ mix deps.get > /dev/null
$ MIX_ENV=prod mix release --quiet

==> jason
Compiling 10 files (.ex)
Generated jason app
==> kv
===> Analyzing applications...
===> Compiling telemetry
==> libcluster
Compiling 15 files (.ex)
Generated libcluster app
==> kv
Compiling 2 files (.ex)
Generated kv app

Start two nodes from the same build

Settings shared by every node go in the environment once. RELEASE_NODE with RELEASE_DISTRIBUTION=name gives a node its full name, and RELEASE_COOKIE replaces the random cookie the build generated, so both nodes share one. The kv shell function runs the release's script as node kv_a or kv_b.

$ export RELEASE_DISTRIBUTION=name
$ export RELEASE_COOKIE=kv-demo-cookie
$ export CLUSTER_NODES=kv_a@127.0.0.1,kv_b@127.0.0.1

$ kv() { node=$1; shift; RELEASE_NODE="kv_$node@127.0.0.1" _build/prod/rel/kv/bin/kv "$@"; }

$ kv a daemon
$ kv b daemon

Both nodes boot, and libcluster connects them in the background. We wait until each node sees both replicas.

$ until [ "$(kv a rpc 'IO.write(KV.Replica.count())' 2>/dev/null)" = 2 ] &&
  [ "$(kv b rpc 'IO.write(KV.Replica.count())' 2>/dev/null)" = 2 ]; do
  sleep 0.2
done
$ kv a rpc 'IO.inspect(Node.list(), label: "a is connected to")'

a is connected to: [:"kv_b@127.0.0.1"]

One store

A write on kv_a, read on kv_b: two copies of the same build, configured only by their environment, acting as one store.

$ kv a rpc 'KV.Replica.put("lang", "elixir")'
$ kv b rpc 'IO.inspect(KV.Replica.get("lang"), label: "b reads lang")'

b reads lang: "elixir"

$ kv a stop
$ kv b stop

Pitfall: String.to_atom/1 on configuration you control is fine; on anything a user can send, it isn't (atoms are never garbage collected, and the atom table has a limit).

The store still forgets everything when both nodes stop, and new nodes start empty (steps 9 and 10 showed the fixes). The next steps put this release in a container and run it on Kubernetes.