SRE Alerting Guidelines

Using This Guide with LLMs Tip This guide is LLM friendly. The same text is served as plain Markdown at writings.conall.dev/documents/sre-alerting-guidelines/index.md, and a Claude Code skill reviews alerts against the current guide. Install it with: /plugin install sre-alerting-guidelines --marketplace conallob/SRE-Alerting-Guidelines See Use the guide as a Claude Code skill for more details. Why This Document There are many foundational monitoring texts for engineers—most notably the Monitoring Distributed Systems chapter of the SRE Book, and the design advice from the Monitoring chapter of the SRE Workbook. ...

October 11, 2026 · Last Edited October 28, 2026 · 12 min · Conall O'Brien

Alerts as a State Machine: Promote, Demote, Repeat

Most alerting treats an alert as static: you write a rule, it pages someone, and it keeps doing so until somebody deletes it. But alerts have a lifecycle. They start unproven, become trustworthy, get noisy as the system underneath them changes, and sometimes need different handling at 11am on a Tuesday than at 3am on a Sunday. Model that lifecycle explicitly and a lot of painful, ad-hoc process becomes a one-line change. Treat one routing label as the alert’s state, and Alertmanager as the machine that maps each state to a destination. ...

October 1, 2026 · 8 min · Conall O'Brien