Alerts as a State Machine: Promote, Demote, Repeat
Most alerting treats an alert as static: you write a rule, it pages someone, and it keeps doing so until somebody deletes it. But alerts have a lifecycle. They start unproven, become trustworthy, get noisy as the system underneath them changes, and sometimes need different handling at 11am on a Tuesday than at 3am on a Sunday. Model that lifecycle explicitly and a lot of painful, ad-hoc process becomes a one-line change. Treat one routing label as the alert’s state, and Alertmanager as the machine that maps each state to a destination. ...