Label: event driven + events/notifications, with an incident workflow (state machine).
Clarify
- Rule types: thresholds, SLO burn, log patterns? Evaluation interval?
- Channels and on-call schedules? Should the AI agent start automatically?
- What's worse, a missed page or a duplicate page? (Missed. Say so.)
Sketch
A rule evaluator, sharded by rule ID, runs each rule on its interval against the TSDB or the log stream and keeps alert state (pending, firing, resolved). Transitions become alert events. A dedup and grouping step computes a fingerprint from labels, drops repeats, and groups related alerts (same service, 5 minute window) into one incident. The incident service is a state machine (triggered, acknowledged, resolved) that pages the primary on call, starts an escalation timer, and kicks off the AI investigation. Notifiers send push, SMS, phone and Slack with retries and fallback.
Deep dives
Flapping and noise
Example CPU hovers at 89 to 91% against a 90% threshold and pages 30 times an hour.
Default answer Require the condition to hold for N minutes, resolve at a lower threshold, and alert on SLO burn rate instead of raw resource metrics.
Evaluator high availability without double paging
Example Two evaluator replicas both fire the same alert during a failover.
Default answer Run replicas, let both evaluate, and rely on the fingerprint dedup downstream; or use leader election per shard. Duplicates are cheap; a missed page is not.
Page delivery
Example The push provider is down at 3 AM.
Default answer At-least-once delivery with retries, escalate channel (push, then SMS, then phone call) if no ack in N minutes, and track delivery receipts.
Grouping too much vs too little
Example A database outage triggers 300 alerts across 40 services.
Default answer Group by time window and topology (service dependency graph), so the 300 alerts become one incident with the database as the likely root.