Reliability Patterns: Availability and Resiliency
Patterns for staying up when whole servers, zones or regions fail: deployment stamps (cells), geodes, multi-zone and multi-region designs, plus a map of the resiliency patterns covered elsewhere in the course.
In a nutshell
- Spread across availability zones first: it handles the most common large failures at modest cost.
- Deployment stamps (cells) run many independent copies of the whole stack, each serving a subset of users, limiting the blast radius.
- Geodes run full copies in many regions, each able to serve any user, for global low latency and region-level resilience.
- Within each copy, use the resiliency patterns: timeouts, retries, circuit breakers, bulkheads, health checks, throttling.
Zones and regions
Cloud providers group data centres into availability zones (separate power, cooling and networking, close enough for low-latency links) inside regions (far apart geographically). Running instances in at least two or three zones survives a data-centre failure with little extra complexity. Running in several regions survives a regional outage, but data replication over long distances brings latency and consistency trade-offs.
| Setup | Survives | Cost and complexity |
|---|---|---|
| Single zone | Server failures | Lowest |
| Multi-zone | A data-centre outage | Moderate: synchronous replication is still practical |
| Multi-region, active-passive | A region outage, after failover | Higher: asynchronous replication, failover drills |
| Multi-region, active-active | A region outage with no failover pause | Highest: conflict handling, global routing |
Deployment stamps (cells)
Instead of one giant deployment serving everyone, run many identical stamps, each a complete copy of the stack (app servers, databases, queues) serving a slice of customers. A routing layer maps each customer to their stamp. A bad deploy or a noisy tenant affects one stamp, not everyone; growth means adding stamps; and stamps can be placed in the regions their customers need.
Customers routed to independent stamps
Diagram as text
- Customers → Cell router
- Cell router → Stamp 1: A–H
- Cell router → Stamp 2: I–P
- Cell router → Stamp 3: Q–Z
Geodes
A geode deployment runs full copies of the service in many regions, each able to serve any user, with data replicated between them (usually through a globally distributed database). Users are routed to the nearest healthy region. The result is low latency everywhere and survival of a whole region going down, at the cost of a data layer that handles multi-region writes.
The resiliency toolkit
| Pattern | Protects against | Topic |
|---|---|---|
| Timeouts, retries with backoff | Transient failures, hung calls | Timeouts, retries, backoff and jitter |
| Circuit breaker | Repeatedly calling a broken dependency | Circuit breaker |
| Bulkhead | One failure exhausting shared resources | Bulkhead |
| Health endpoint monitoring | Sending traffic to broken instances | Health checks |
| Throttling, load leveling | Overload from spikes | Throttling and queue-based load leveling |
| Compensating transaction | Half-finished multi-step operations | Compensating transactions |
| Leader election | Two coordinators at once | Leader election and consensus |
Common mistakes
- Running everything in one zone and calling it highly available.
- Multi-region designs whose failover has never been tested.
- One global deployment where a single bad config change reaches every customer at once.
Interview questions
Try answering out loud before you open each one.
How would you make this service survive a region outage?
Deploy to at least two regions. For active-passive, replicate data asynchronously to the standby region and fail over with DNS or global load balancing, accepting a small data-loss window (RPO) and failover time (RTO). For active-active, route users to the nearest region and use a data store that supports multi-region writes, handling conflicts. Within each region, run across zones. Practise failover regularly.
Check yourself
Related topics
Further reading
- Deployment Stamps pattern Microsoft Azure Architecture Center
- Geode pattern Microsoft Azure Architecture Center
- Reducing the Scope of Impact with Cell-Based Architecture AWS whitepaper
Your notes
Saved in this browser as you type.
Finished this topic?
Mark it read to track your progress, then carry on with the next one.