IO.inspect(System.schedulers_online(), label: "schedulers")schedulers: 1Schedulers and Reductions
The BEAM runs one scheduler thread per CPU core, and each scheduler runs processes from its own run queue. It doesn't wait for a process to give up the CPU: it counts the work the process does in reductions (roughly, function calls) and switches to the next process once a budget of a few thousand is spent. So one process stuck in a loop can't freeze the others, and reductions are the number to look at when a node is busy and you want to know who's doing the work.
This page runs with a single scheduler (elixir --erl "+S 1"), so every process has to share one thread.
A process that never stops: an endless tail-recursive loop. It never waits for a message, so it never gives up the CPU by itself.
defmodule Spin do
def forever(n), do: forever(n + 1)
end
busy = spawn(fn -> Spin.forever(0) end)
Process.register(busy, :busy)And one that only waits.
idle = spawn(fn -> receive do: (:never -> :ok) end)
Process.register(idle, :idle)With the loop running on the only scheduler, another process still gets to answer a message, and this script still gets to read the answer. The scheduler preempts the loop each time its budget runs out.
me = self()
pinger = spawn(fn -> receive do: ({:ping, from} -> send(from, :pong)) end)
send(pinger, {:ping, me})
receive do: (:pong -> IO.puts("got :pong while the loop runs"))got :pong while the loop runsWhich process is burning the CPU? Every process's
reduction count only goes up, so take two snapshots
and compare. The process whose count grew the most is
the busy one. This is what recon's proc_window/3
and observer_cli do on a live node.
snapshot = fn ->
for pid <- Process.list(),
{:reductions, count} <- [Process.info(pid, :reductions)],
into: %{},
do: {pid, count}
end
before = snapshot.()
Process.sleep(100)
later = snapshot.()
[{top, grew} | _] =
later
|> Enum.map(fn {pid, count} -> {pid, count - Map.get(before, pid, 0)} end)
|> Enum.sort_by(fn {_pid, grew} -> grew end, :desc)
IO.inspect(Process.info(top, :registered_name), label: "busiest process")
IO.inspect(grew > 1_000_000, label: "over a million reductions in 100 ms")
{:reductions, idle_count} = Process.info(idle, :reductions)
IO.inspect(idle_count < 100, label: "the idle one barely ran")
Process.exit(busy, :kill)busiest process: {:registered_name, :busy}
over a million reductions in 100 ms: true
the idle one barely ran: truePitfall: preemption only happens in Erlang and Elixir code. A NIF (a function written in C or Rust and loaded into the VM) runs until it returns, and the scheduler running it can't serve anyone else in the meantime. A NIF that may take more than about a millisecond must run on a dirty scheduler, a separate pool of threads for long native work (
ERL_NIF_DIRTY_JOB_CPU_BOUNDin C,#[rustler::nif(schedule = "DirtyCpu")]with Rustler). See Ports and NIFs.
Pitfall:
Process.flag(:priority, :high)looks like a fix for a slow process, but a high-priority process that is always busy starves every normal one, including the ones the system needs to stay healthy. Find out why the process is slow instead.
Pitfall: high CPU with low throughput often means run queues are long, not that one process is slow.
:erlang.statistics(:total_run_queue_lengths)shows how many processes are waiting for a scheduler. A number that stays above the number of schedulers means the node has more work than cores.
Go deeper: chapter 10 of The BEAM Book by Erik Stenman, "Scheduling", covers run queues, process states, load balancing between schedulers, and how ports are scheduled.