← All interview notes
Resolve AI · system design prep · tech interview 2026-11-17

Resolve AI: system design prep

About 25 minutes of reading, then five practice questions you can work through in Excalidraw. Everything here maps to the ten patterns, so you can label first and then draw.

The prep plan, step by step →

1. What they build, in one breath

Resolve sells an AI SRE: an agent that goes on call, triages alerts, investigates incidents and suggests fixes, working in Slack, Teams and the terminal. It mostly reads the customer's existing tools (Datadog, Splunk, Prometheus, Kubernetes, GitHub deploy history) through integrations, rather than being a new place to send all telemetry. The founders built Splunk Observability and co-created OpenTelemetry, so expect them to know ingestion pipelines cold.

Customers are big enterprises (Coinbase, DoorDash, Salesforce, Zscaler). That means multi-tenancy, security and per-customer credentials matter in every design. Their blog keeps returning to agent accuracy, token efficiency, harness design and human-in-the-loop fatigue, and Resolve AI Labs works on evals and simulated environments.

2. Which patterns they will probe

AreaPattern labelsWords to use
Telemetry inWrite heavy + batch/analytics + searchBack pressure, partition key, cardinality, sampling, hot/cold tiers
Alerts and pagingEvent driven + notifications + workflowDedup fingerprint, grouping window, flapping, escalation timer, at-least-once paging
The agentWorkflow orchestration + searchDurable run, tool calls, context budget, read-only tools, human approval, evals
IntegrationsRead heavy + transactional (credentials)Per-tenant rate limits, credential vault, cache, normalised schema, audit log
QualityBatch/analyticsReplay, frozen snapshots, LLM judge, regression gate

3. Ten terms they'll assume you know

MELT
Metrics, events, logs, traces: the four kinds of telemetry.
OpenTelemetry (OTel)
The open standard and SDKs for emitting telemetry; OTLP is its wire protocol and the Collector is its agent/pipeline.
Cardinality
How many distinct label combinations a metric has. http_requests{user_id=...} with 10M users makes 10M time series and melts a TSDB. The number one ingestion risk.
TSDB
Time-series database (Prometheus, M3, Mimir). Stores samples per series, compresses by time.
Head vs tail sampling
Head: decide to keep a trace when it starts (cheap, blind). Tail: decide after it finishes, so you can keep all errors and slow traces (needs buffering).
SLO / error budget / burn rate
SLO: target such as 99.9% of requests succeed. Error budget: the 0.1% you may fail. Burn rate: how fast you're spending it; alert on burn rate, not on raw CPU.
Alert fingerprint
A hash of an alert's identifying labels, used to dedupe repeats of the same alert.
Flapping
An alert firing and resolving over and over. Fix with "must be true for 5 minutes" and a different threshold to resolve (hysteresis).
MTTD / MTTR
Mean time to detect and to resolve. Resolve's whole pitch is cutting MTTR (Coinbase: 72% less time to investigate).
Tool calling / MCP
The LLM asks for a function to be run (query logs, get deploys) and gets the result back. MCP is a standard way to expose such tools; Resolve integrates via MCP.

4. How to run the 45 minutes

  1. Requirements, 5 min. Functional, then non-functional (latency, availability, consistency), then what's out of scope. Ask about tenants and scale.
  2. Numbers, 3 min. Round hard. 1 day is about 100k seconds. 1M events/s at 500 B is 0.5 GB/s, about 43 TB/day.
  3. Label the pattern out loud, then API and data model, 5 min.
  4. High-level boxes, 10 min. Drag the template, then adapt it to what they said.
  5. Deep dives, 15 min. Pick the two riskiest parts yourself, and name the trade-off and your default answer for each.
  6. Wrap, 2 min. Failure modes, what you'd monitor, what you'd build next.

The "Interview canvas" template in the general library lays out these five columns.

5. Practice questions

Try each one cold for 20 minutes in Excalidraw, then open the sketch answer. The matching diagram is in your library under the same Q number.

Q1Design telemetry ingestion for logs, metrics and tracesMulti-tenant, 1M events/s peak, queryable within 30 s, retained a year.

Label: write heavy / event driven, feeding search (logs) and batch/analytics (long-term).

Clarify

  • Which signals, and is it push (agents) or pull (scrape)?
  • Freshness for alerting vs for queries? Retention per tier?
  • How many tenants, and is any tenant 100× the others?

Numbers

1M events/s × 500 B = 0.5 GB/s, about 43 TB/day raw; compressed maybe 5 to 10×, so 5 to 8 TB/day stored. A year of that hot is too expensive, so tier it.

Sketch

OTel collectors batch and send OTLP to an ingest gateway that authenticates, checks the tenant's quota and acks quickly. It appends to Kafka with topics per signal, keyed by tenant + stream so each stream stays ordered. Processors parse, enrich (service, k8s labels), sample and derive metrics from logs. Logs go to an inverted index for 7 days and Parquet on S3 for a year; metrics go to a TSDB; traces to a trace store with tail sampling. A query service fans out to hot and cold stores with per-tenant limits.

Deep dives

  1. Noisy neighbour

    Example One tenant ships a debug-logging bug and sends 20× normal volume. Without quotas, everyone's ingestion lags.

    Default answer Per-tenant rate limits at the gateway, separate partitions for the biggest tenants, and drop or sample over-quota debug logs, never errors.

  2. Cardinality explosion

    Example Someone adds request_id as a metric label and creates millions of series per minute.

    Default answer Enforce a series limit per metric and tenant at ingest, reject or aggregate away high-cardinality labels, and tell the customer.

  3. Back pressure

    Example The log index falls behind during an incident, which is exactly when people query most.

    Default answer Kafka absorbs hours of backlog; prioritise error and warn logs; autoscale indexers on consumer lag; alerting reads the stream so it isn't blocked by indexing.

  4. Parse at ingest vs at query

    Example Parsing every log line into fields at ingest makes queries fast but costs CPU on data nobody reads.

    Default answer Extract a few common fields (service, level, trace_id) at ingest; parse the rest at query time.

Q2Design alerting and on-call pagingEvaluate 100k rules every minute, group into incidents, page a human within 60 s, escalate if no ack.

Label: event driven + events/notifications, with an incident workflow (state machine).

Clarify

  • Rule types: thresholds, SLO burn, log patterns? Evaluation interval?
  • Channels and on-call schedules? Should the AI agent start automatically?
  • What's worse, a missed page or a duplicate page? (Missed. Say so.)

Sketch

A rule evaluator, sharded by rule ID, runs each rule on its interval against the TSDB or the log stream and keeps alert state (pending, firing, resolved). Transitions become alert events. A dedup and grouping step computes a fingerprint from labels, drops repeats, and groups related alerts (same service, 5 minute window) into one incident. The incident service is a state machine (triggered, acknowledged, resolved) that pages the primary on call, starts an escalation timer, and kicks off the AI investigation. Notifiers send push, SMS, phone and Slack with retries and fallback.

Deep dives

  1. Flapping and noise

    Example CPU hovers at 89 to 91% against a 90% threshold and pages 30 times an hour.

    Default answer Require the condition to hold for N minutes, resolve at a lower threshold, and alert on SLO burn rate instead of raw resource metrics.

  2. Evaluator high availability without double paging

    Example Two evaluator replicas both fire the same alert during a failover.

    Default answer Run replicas, let both evaluate, and rely on the fingerprint dedup downstream; or use leader election per shard. Duplicates are cheap; a missed page is not.

  3. Page delivery

    Example The push provider is down at 3 AM.

    Default answer At-least-once delivery with retries, escalate channel (push, then SMS, then phone call) if no ack in N minutes, and track delivery receipts.

  4. Grouping too much vs too little

    Example A database outage triggers 300 alerts across 40 services.

    Default answer Group by time window and topology (service dependency graph), so the 300 alerts become one incident with the database as the likely root.

Q3Design an AI agent that investigates an alertGiven a firing alert, find the likely root cause in under 5 minutes using the customer's tools, and post evidence to Slack. (The most Resolve-shaped question.)

Label: workflow orchestration (a durable, budgeted run) over search and read-heavy tool calls.

Clarify

  • Which tools can it reach (metrics, logs, traces, k8s, deploys, feature flags)? Read-only, or can it act?
  • Latency and cost budget per investigation? How many concurrent incidents?
  • How do we know it was right? (Leads to evals.)

Sketch

An alert or Slack mention starts a durable investigation run with an ID, a time budget and a token budget. The orchestrator loops: ask the LLM for the next step given what's known, call a typed tool (query error logs for service X in the last 15 min, list deploys, get pod restarts), summarise the result to fit the context window, record it, and repeat until confident or out of budget. A context store supplies the service dependency graph, runbooks and similar past incidents (retrieval). The output is a short finding in Slack: likely cause, evidence links, confidence, and suggested next action. Every step is logged for replay and evals.

Deep dives

  1. Context window vs evidence

    Example One log query returns 50,000 lines; pasting them in blows the context and the budget.

    Default answer Tools return summaries (top error messages with counts, anomalies vs baseline), not raw dumps; keep raw results in a store the final answer links to.

  2. Autonomy vs safety

    Example The agent concludes a bad deploy caused the incident. Should it roll back?

    Default answer Investigation tools are read-only. Actions like rollback are proposed with evidence and need a human click, with an audit log. Widen autonomy per customer as trust grows.

  3. Speed vs thoroughness

    Example Checking every service in a 200-service graph takes too long.

    Default answer Start from the alerting service and walk the dependency graph outward, run independent queries in parallel, and stop when a hypothesis is confirmed by two independent signals.

  4. Hallucinated root causes

    Example The model says "memory leak" with no memory data looked at.

    Default answer Require each claim to cite a tool result, show confidence, and say "not sure" when evidence is thin. Measure this in evals.

  5. Durability

    Example A deploy restarts the orchestrator mid-investigation.

    Default answer Persist each step (Temporal-style durable execution), so the run resumes instead of restarting and re-querying everything.

Q4Design the integration layer to customers' toolsHundreds of customers, each with Datadog, Splunk, Prometheus, k8s and GitHub, queried safely by agents.

Label: read heavy with strict tenancy; credentials handled with transactional care.

Sketch

Agents call one unified query API with (tenant, tool, query). It fetches that tenant's credentials from a vault, checks a per-tenant per-provider rate limiter, checks a short-TTL result cache, then dispatches to connector workers that translate to each provider's API and normalise results into a common shape. Private clusters are reached through a small relay the customer runs in their VPC, which dials out to Resolve so no inbound firewall holes are needed. Every query is audit-logged.

Deep dives

  1. Rate limits

    Example During a big incident, 10 parallel investigations hit the customer's Datadog API limit and their own dashboards start failing.

    Default answer A token bucket per tenant and provider, set below the provider's limit; honour 429 with backoff; cache identical queries for 30 to 60 s.

  2. Tenant isolation

    Example A bug sends tenant A's query with tenant B's API key.

    Default answer Tenant ID flows through every call and is checked at the vault; connectors run without shared state between tenants; consider per-tenant worker pools for the largest customers.

  3. Partial failure

    Example Splunk times out but Prometheus answers.

    Default answer Tight timeouts, return partial results labelled as partial, and let the agent reason about missing data.

Q5How would you know the agent is getting better?Design the evaluation system for the investigation agent.

Label: batch/analytics over replayed runs. Resolve AI Labs works on exactly this, so it may come up as a follow-up to Q3.

Sketch

Build a set of past incidents labelled with the true root cause. Freeze the telemetry around each one, and serve it through simulated tools so a run is reproducible. An eval runner plays N incidents through M agent versions in parallel. A scorer checks the root cause (exact match where possible, an LLM judge plus human spot checks otherwise), whether evidence was cited, steps taken, tokens and time. Results feed a dashboard, and a release only ships if it doesn't regress.

Deep dives

  1. Live data vs frozen snapshots

    Example Re-running last month's incident against live Datadog returns today's data, so scores change for no reason.

    Default answer Snapshot the data at incident time and replay from it; it costs storage but makes comparisons fair.

  2. Accuracy vs cost

    Example A bigger model is 5 points more accurate but 4× the cost and twice as slow.

    Default answer Track accuracy, cost and time together; route easy cases to a cheaper model and keep the big one for hard ones.

6. Lines worth having ready

  • Before I draw, let me label it. This is write heavy on the way in and search on the way out, so I'll spend my deep dive on back pressure and cardinality.
  • For the agent I'd treat each investigation as a durable workflow with a budget. Tools are read-only, and anything that changes production needs a human.
  • Duplicate pages are annoying; missed pages are incidents. So delivery is at-least-once, and dedup happens on a fingerprint.
  • I'd want an eval set of replayed incidents before changing the prompt or model, otherwise we're tuning by anecdote.

7. Excalidraw

Load two libraries. They don't overlap.

General (11 items)
The blank interview canvas and the ten pattern starters. system-design-patterns.excalidrawlib · add to excalidraw.com
Resolve AI (5 items)
The diagrams for Q1 to Q5 above. resolve-interview.excalidrawlib · add to excalidraw.com

In Excalidraw Plus, open the Library panel (book icon) and drag each file onto it. During the interview, drag the closest template onto the canvas and edit it to match what they ask. It's a starting point, not a finished answer. Single files open with Menu → Open:

More: the pattern guide and the Resolve AI company profile.

8. Questions to ask them

  • Do you mainly query customers' existing tools, or do you also ingest and store telemetry yourselves? What drove that choice?
  • How do you evaluate an investigation as correct? How big is the replay set?
  • Where do agents get to take actions in production today, and what approval model do customers want?
  • How is engineering organised day to day, and how do product engineering and Labs work together?