Availability Patterns: Fail-over and Replication
How systems stay up when machines fail: active-passive and active-active fail-over, leader-follower and multi-leader replication, and the traps that come with them, such as split brain and lost writes.
In a nutshell
- Remove single points of failure: run at least two of everything that matters.
- Active-passive: a standby takes over when the active node stops sending heartbeats. Active-active: all nodes serve traffic.
- Leader-follower replication is simple and common; multi-leader accepts writes in several places but must resolve conflicts.
- Fail-over risks split brain (two leaders) and lost writes (async replication); design for both.
Single points of failure
Any component that, if it fails, takes the whole system down is a single point of failure: one database server, one load balancer, one network link. High availability starts with finding them and adding redundancy, then making sure traffic actually moves to the healthy copy when one fails. That second part is fail-over.
Active-passive fail-over
One node (active) handles all traffic; a standby (passive) waits. They exchange heartbeats. If the standby stops hearing from the active node, it takes over the shared address and starts serving.
Active-passive fail-over
All steps Only the active node serves. When its heartbeats stop, the standby promotes itself and takes over the address.
- Clients send requests to the active node through a shared address (for example a virtual IP).
- The active node sends a heartbeat to the standby every second or so.
- The active node crashes.
- The standby notices that heartbeats have stopped for longer than a timeout.
- It promotes itself to active and claims the shared address.
- Clients now reach the new active node. They may have seen errors during the switch.
Diagram as text
- Clients send requests to the active node through a shared address (for example a virtual IP).
- The active node sends a heartbeat to the standby every second or so.
- The active node crashes.
- The standby notices that heartbeats have stopped for longer than a timeout.
- It promotes itself to active and claims the shared address.
- Clients now reach the new active node. They may have seen errors during the switch.
- Simple to reason about: only one node writes at a time.
- Wasted capacity: the standby sits idle until something breaks.
- Downtime during the switch: detection timeout plus promotion, often seconds to a minute. A hot standby (already running, in sync) switches faster than a cold one (started on demand).
Active-active fail-over
All nodes serve traffic at the same time behind a load balancer (or DNS). If one fails, health checks remove it and the others absorb its share.
Active-active
Diagram as text
- Clients → Load balancer
- Load balancer → Node A: 100% now
- Load balancer → Node B: removed
Every node is useful, and losing one only reduces capacity. The catch: each node must be able to take the extra load (plan for N + 1 capacity), and if the nodes hold data, writes on several nodes at once need coordination.
Replication
For stateless servers, fail-over is enough. For databases, the standby also needs a copy of the data, which is what replication provides. (The Databases module covers replication in depth.)
Leader-follower replication
Diagram as text
- App → Leader: writes
- Leader → Follower 1: replicate
- Leader → Follower 2: replicate
| Leader-follower | Multi-leader | |
|---|---|---|
| Writes accepted by | One leader | Several leaders (often one per region) |
| Conflicts | None: one place decides the order | Two leaders can change the same record; conflicts must be resolved |
| If the leader fails | Promote a follower (fail-over) | Other leaders keep accepting writes |
| Best for | Most applications | Writes in several regions, offline-capable apps |
Synchronous replication waits for followers before confirming a write: no data loss on fail-over, but slower and stuck if a follower is down. Asynchronous replication confirms immediately: fast, but writes not yet copied are lost if the leader dies.
What goes wrong
Common mistakes
- Adding a standby but never testing fail-over, then discovering on the day that it does not work.
- Setting heartbeat timeouts too short, so a brief pause triggers fail-over and possible split brain.
- Running active-active at more than about 50% each across two nodes: when one fails, the survivor is overloaded.
- Using asynchronous replication for payments and losing confirmed writes on fail-over.
Interview questions
Try answering out loud before you open each one.
What is the difference between active-passive and active-active?
In active-passive, one node serves and a standby takes over when it fails; simple, but the standby is idle and fail-over takes time. In active-active, all nodes serve and a failed one is simply removed; better use of capacity and faster recovery, but each node must handle extra load and shared data needs coordination.
What is split brain and how do you prevent it?
Two nodes both believe they are the leader, usually because a slow or partitioned leader was declared dead. Both accept writes and the data diverges. Prevent it by electing leaders with a majority vote (an odd number of voters), and fencing the old leader so its writes are rejected, for example with increasing epoch or token numbers.
Check yourself
Related topics
Further reading
Your notes
Saved in this browser as you type.
Finished this topic?
Mark it read to track your progress, then carry on with the next one.