SRE Alerting Guidelines

Using This Guide with LLMs Tip This guide is LLM friendly. The same text is served as plain Markdown at writings.conall.dev/documents/sre-alerting-guidelines/index.md, and a Claude Code skill reviews alerts against the current guide. Install it with: /plugin install sre-alerting-guidelines --marketplace conallob/SRE-Alerting-Guidelines See Use the guide as a Claude Code skill for more details. Why This Document There are many foundational monitoring texts for engineers—most notably the Monitoring Distributed Systems chapter of the SRE Book, and the design advice from the Monitoring chapter of the SRE Workbook. ...

October 11, 2026 · Last Edited October 28, 2026 · 12 min · Conall O'Brien

Alerts as a State Machine: Promote, Demote, Repeat

Most alerting treats an alert as static: you write a rule, it pages someone, and it keeps doing so until somebody deletes it. But alerts have a lifecycle. They start unproven, become trustworthy, get noisy as the system underneath them changes, and sometimes need different handling at 11am on a Tuesday than at 3am on a Sunday. Model that lifecycle explicitly and a lot of painful, ad-hoc process becomes a one-line change. Treat one routing label as the alert’s state, and Alertmanager as the machine that maps each state to a destination. ...

October 1, 2026 · 8 min · Conall O'Brien

Automating Myself Out of a Job (Again): Introducing o11y-analysis-tools

There’s a lot of anxiety right now about AI taking jobs. But SREs have been automating themselves out of their own jobs for decades — the reward for good work, inevitably, is yet more work. My latest example: Over my career, I’ve personally performed a few thousand code reviews of PromQL (neé Borgmon). Any issues I catch can be distilled down into a handful of categories, none of which I consider exotic. ...

July 29, 2026 · 1 min · Conall O'Brien