Alerting Without Alert Fatigue

Page, ticket or nothing

Burn-rate alerts

With an SLO, you can alert on how fast the error budget is being used. Burn rate is the current error rate divided by the error rate the SLO allows. A burn rate of 1 uses exactly the whole budget over the SLO window; 14.4 uses a 30-day budget in about 2 days.

The Google SRE workbook recommends combining a long and a short window: page only if the burn rate is high over the last hour and still high over the last 5 minutes. The long window avoids paging for a brief spike; the short window makes the alert stop quickly once the problem is fixed.

class BurnRate {
    static double burnRate(double errorRate, double slo) {
        return errorRate / (1 - slo);
    }

    // Page when both the long window (e.g. 1 h) and the short window (e.g. 5 min) burn too fast.
    static boolean shouldPage(double longWindowErrorRate, double shortWindowErrorRate, double slo, double threshold) {
        return burnRate(longWindowErrorRate, slo) >= threshold
            && burnRate(shortWindowErrorRate, slo) >= threshold;
    }
}

def burn_rate(error_rate, slo):
    return error_rate / (1 - slo)

def should_page(long_window_error_rate, short_window_error_rate, slo, threshold):
    """Page when both the long window (e.g. 1 h) and the short window (e.g. 5 min) burn too fast."""
    return (burn_rate(long_window_error_rate, slo) >= threshold
            and burn_rate(short_window_error_rate, slo) >= threshold)

The tests show the rule at work: a sustained 2% error rate pages; the same hour that has since recovered (0.05% now) does not; and a 5-minute spike in an otherwise healthy hour does not either.

Dashboards and runbooks

• Dashboards tell a story top-down: the service's golden signals first, then its dependencies, then resources. Someone woken at 3 a.m. should see in seconds what is broken.
• Every alert links to a runbook: what it means, how to confirm it, first steps, how to escalate.
• Alerts carry context: which service, which SLO, current value, a link to the dashboard.
• Review alerts regularly: delete or downgrade ones that never led to action; fix the causes of ones that fire often.
• Blameless post-incident reviews: focus on how the system allowed the failure, and track the follow-up fixes.

Common mistakes

• Paging on causes (high CPU) instead of symptoms (slow responses).
• Alerts with no runbook or owner.
• Static thresholds that fire on every brief spike.
• Letting noisy alerts stay noisy until people mute the channel.

Interview questions

Try answering out loud before you open each one.

Check yourself

Related topics

• Logs, Metrics, Traces and SLOsMonitoring and Operations
• Monitoring: What to WatchMonitoring and Operations
• Circuit BreakerReliability and Performance

Further reading

• Alerting on SLOs Google SRE workbook
• My Philosophy on Alerting Rob Ewaschuk