SRE Alerting Guidelines

Using This Guide with LLMs

/plugin install sre-alerting-guidelines --marketplace conallob/SRE-Alerting-Guidelines

Why This Document

There are many foundational monitoring texts for engineers—most notably the Monitoring Distributed Systems chapter of the SRE Book, and the design advice from the Monitoring chapter of the SRE Workbook.

This guide isn’t here to rehash broad architectural theory. Instead, it’s a complementary, tactical, style guide designed to give you direct “do-this, not-that” guardrails for citation in code reviews, design debates, explicitly accounting for modern systems and the realities of human operational psychology. Earlier versions assumed the reader already had hands-on monitoring experience; that assumption is being deliberately distilled out through multiple iterations, as AI coding agents absorb more of the mechanics.

When implementing/refactoring an alert, sometimes you just need a specific best practice (or cautionary tale) to cite during review, or AI prompt construction.

Supported System Paradigms

These best practices have been collected from various systems of the following paradigms:

• User Facing Systems
  • Stateless User Facing Systems
  • Stateful User Facing Systems
• Infrastructure
  • Stateless Infrastructure (e.g a ValKey cache)
  • Stateful Backend Infrastructure (e.g Storage systems)
• Control Systems (e.g Service Meshes such as Istio)

Unsupported System Paradigms

• Pipeline Systems (since they behave fundamentally differently to stateless/stateful and/or interactive systems)

Guiding First Principles

Each principle represents an important goal; at times a compromise between two principles may be necessary, but tradeoffs should be made with an understanding of the intention behind the principles in tension

Coverage

• We need to know about significant user harm as much as possible.
  • Corollary: user reports of significant issues will always highlight a monitoring blindspot. User reports of systemic issues should rarely catch a well-tuned monitoring system by surprise.
• End to end verification for every service is desirable, end to end verification for critical user workflows is paramount.

Priorities

• Choose alerts that have a high signal to noise ratio, and focus on excellent quality and thoroughness for those alerts, rather than a scattershot of alerts which are not well-tuned.
• Alerts should always be actionable, either directly or indirectly (e.g escalating to another team/ SaaS vendor, etc.). See Alerts as a State Machine for a routing pattern that keeps every alert in a defined, actionable state.
• For a large number of SLO definitions, grouping by defined SLO groups (aka bucket), will be more effective than individual alerts per SLO definition

Maintainability

• Maintain effective monitoring with minimal time commitment
• Every alert requires ongoing maintenance and tuning or risks an increase in false positives or negatives.
• Aim to have the smallest number of alerts that grant adequately thorough coverage; defence in depth may be appropriate in some cases, but excessive redundant or overlapping alerts increases maintenance cost and ultimately quality.

Guidelines

Synthetic vs Exposed Metrics (neé Blackbox vs Whitebox)

There are an infinite number of ways things can break, which means the amount of alerting logic required for good coverage would be staggering.

Although synthetic (neé blackbox) monitoring (AKA dye testing, when referring to pipeline systems) has less coverage, its ability to cover multiple scenarios means its cost-benefit is much more appealing compared to exposed metrics (neé whitebox) monitoring. Prefer synthetic monitoring where possible.

There is a misconception that the choice is only ever an XOR between monitoring synthetic vs exposed metrics. A monitoring system needs both; synthetic checks to tell you when (and ideally where) something is wrong; and exposed metrics to facilitate in depth troubleshooting to discover exactly what has gone wrong.

Mature systems, with defined SLOs and error budget based alerting will employ both synthetic probes and exposed metrics in their observability stack.

Alert on Symptoms, not Causes

Alert based on symptoms, not root causes/edge cases. The number of root causes will always vastly outnumber symptoms, so symptom based alerting will always result in higher coverage. Moreover, root causes are very specific to the current implementation; as the system evolves, behaviours and thus symptoms are more likely to remain consistent than are details of the implementation.

Root cause alerts can accelerate debugging, but only for known failure modes. Due to the number of potential root causes, edge cases or unknown failure modes, the maintenance cost of cause based alerting will continue to grow in perpetuity.

For complicated systems with multiple failure modes, multiple symptom based signals may be employed to perform differential diagnosis. For validation, root cause signals can be surfaced on dashboards to help aid debugging almost as quickly.

Examples

• HTTP 500 Errors can occur for both network issues behind HTTP reverse proxies (such as load balancers) and for backend HTTP servers experiencing localised issues. TCP retransmits between load balancers and backend components is a more specific symptom signal than HTTP 500s, when diagnosing network issues compared to HTTP 500 errors

Additional Resources

• SREcon Conversations Europe/Middle East/Africa with Štěpán Davidovič, Google Inc.

Use Error Budgets for SLO Alerting Thresholds

If your service has an error budget, implement a pageable alert for approximately one day of the error budget (as determined by the SLO) consumed in a condensed timeframe (e.g 1 - 6 hours). Smaller events which do not cause significant risk to the SLO do not warrant immediate attention, but waiting more than one day’s worth of damage risks consuming too much error-budget too fast.

E.g.

Service X handles 100M queries per quarter and the SLO is 99.9% available.
So the paging alert is configured to be:

  100M x (100 - 99.9)% x 1% = 1000 queries fail

Service Y has a latency SLO that < 1% of queries can take 2 seconds and serves 100M queries

  100M x (100 - 99)% x 1% = 10,000 queries are slow

For more examples and for more advanced, multi window SLO burn down alerting, see Alerting on SLOs from the SRE Workbook and Implementing SLOs. For feature rich implementations, see Sloth (Open Source Framework) or Nobl9 (SaaS).

Counter Examples

• Traditional networking alerts assume every packet is unique and precious, and consider the loss of individual packets to be a terrible thing. This approach does not consider that lower level network events fail to consider the layers of abstraction in the OSI networking model and which layers provide redundancy. In reality, higher level applications anticipate network issues and rarely notice, thanks to features such as the retry logic in TCP, multi region deployments, caching, etc.

Avoid Redundant Alerts

Avoid redundant alerting. If alert A always fires whenever alert B fires, then remove alert B (since alert A is a superset which includes edge case B). A notification controller such as the Prometheus alertmanager provides rate limiting, grouping and even alert suppression controls to tame redundant alerts that cannot simply be deleted.

<alert expression>
  unless
ALERTS{alertname='InhibitingAlertName',alertstate='firing'}

Example

• Once implemented, the degraded score signal makes a Jobs Missing alert redundant.
  • Combining the degraded score signal with a dashboard of health signals provides the same insights, without all the alerts.

Counter Example

• If a system X employs both symptom and root cause based alerting, these alerts will often have overlapping thresholds and/or scope.
  • Common failure modes will include regular storms of pages and/or tickets, only a handful of which will actually be useful.

Use Predictive (Extrapolation) Alerting Sparingly

Predictive alerting, which extrapolates, based on current thresholds, a future event is imminent, should only be used when there is a high likelihood (50%+) of a major failure (e.g significant error budget burn) or significant MTTM or MTTR. Predictive alerting makes sense for systems with long feedback loops, where metrics cannot update quicker than the system’s latency.

Justify predictive alerts with direct observation, simulations, or empirical testing — not anecdotes or speculation.

Counter Examples

• Quota usage alerts for a hard ceiling are directly actionable — enforcement is imminent. Alerts for soft ceilings or elevated-but-non-critical usage are still actionable (e.g. reduce the offending resource usage) but lower urgency, and should be routed accordingly (ticket rather than page) rather than dismissed as unactionable noise.

Avoid Anticipating Data Correctness in the Monitoring Data

Monitoring data should always be treated as an approximation of what a system is actually doing. It is always a snapshot in time, often with a known value for staleness.

Embrace these imperfections. Attempting to work around data correctness issues within the monitoring system can likely leverage complicated operations. The potential for bugs would be greater than the original risk of data correctness. In time, it will be hard to distinguish justified caution from paranoia.

Take Responsibility where Others Cannot or Do Not

Catastrophic coverage gaps can occur when multiple teams make assumptions about each other’s monitoring. The obvious exception is when both teams have either made an explicit agreement or agreed lines of demarcation in their respective scopes.

Example

• An infrastructure team providing a shared service to multiple teams may only monitor their system per class of users. A specific team should not assume their service is the canonical example of their user class, and should therefore create an end-to-end test to cover their own usage of the managed infrastructure

Counter Example

• An infrastructure team who explicitly chooses to not implement synthetic checks of their service, instead abdicating this responsibility to other teams with overlapping coverage and expecting manual escalations.

Strategies for Improving Alerting Signal

Combatting Alert Fatigue

Humans receiving unactionable alerts are susceptible to alert fatigue. Once it sets in, cognitive biases often form and perceived root causes or a perception that the alert is low-signal may discourage responders to do adequate investigation or skew root-cause analysis. Since alert fatigue is a signal to noise issue compounded by a perception issue, it can lead to a normalisation of deviance.

Once an alert is known to have a long history of questionable signal, it is not enough to refactor and tune the alert to have more value; it is useful to also delete or rename the offending alert, to address the perception issue and reset biases going forward.

Let Your Thresholds Do The Waiting

A common anti-pattern is a ticket generating alert which triggers after a few minutes (detection time), but routinely waits days or weeks for attention (response time). Once triaged, the underlying issue has resolved itself and ticket incidents of this alert are regularly marked as Obsolete or Not Repeatable.

Since detection time is a subset of response time, redistribute some of the implicit waiting from overall response time by explicitly adding it to the alert thresholds itself.

Application and regular tuning of hysteresis settings (aka hold up/hold down timers in Networking terminology) can also remove hair triggers.

Acknowledgements

The curator would like to recognise and thank the following for their invaluable feedback on multiple generations of this document:

• John Truscott Reese
• Laura Nolan
• Štěpán Davidovič
• Trisha Weir
• The Late Cody Smith

Additional Reviewers of a Previous Version:

• Rob Ewaschuk
• Joel Becker
• Trevor Mattson-Hamilton
• Karl-Michael Schneider
• Kate Ward

About The Author

Conall O’Brien (LinkedIn) has 20+ years experience as a Site Reliability Engineer. He has not read the SRE Books, he lived them.