Protect the store
After the Production chapter, KV runs as a release on a cluster, and you can debug it live. This chapter is about what happens to it on a bad day: one client that won't stop writing, a burst of traffic bigger than the disk can take, a replica that hangs instead of failing.
The concept pages of this track are about keeping data right when steps fail: transactions and constraints, the outbox pattern, and sagas. The patterns these steps use live next to the tools they're built from: rate limiting, backpressure and the circuit breaker are on the Processes and OTP track.
- Limit each client. A per-client rate limit in ETS keeps one broken script from taking the store for itself.
- Bound the writes. Writes go to disk through a bounded buffer. A burst makes writers wait instead of filling memory.
- Replicate behind a breaker. A hung replica trips a circuit breaker, so writes stop waiting for it, and the replica is caught up when it recovers.
Like the OTP chapter's steps, each one is a single script that runs on its own, in the browser too.
Before you start
The steps build on Rate LimitingETSBackpressureGenServerCircuit BreakerIdempotency and Retries
Steps
- Limit each client KV now serves many clients, and one of them is a script stuck in a loop, writing as fast as it can. It isn't malicious, just broken, and every write it makes...
- Bound the writes KV keeps a log of every write on disk, so it can rebuild its table after a restart. The disk is slow: each write takes 10 ms. Making every put wait for the d...
- Replicate behind a breaker KV copies every write to a replica on another node. When that node is down, Erlang notices quickly and the call fails at once. When it hangs (a long garbage ...