R085 · Infrastructure for agents at scale › Inference providers and serving reliability

NeMo Switchyard: route coding-agent turns on tool-result signals

Read the original

model-routingcostcoding-agentsprompt-cachingbenchmarks

Source: NVIDIA NeMo Switchyard (github.com/NVIDIA-NeMo/Switchyard, Apache-2.0). Surfaced by André Lindenberg's LinkedIn post (2026-11-15).

What it is

Switchyard is an open-source library that decides, per request, which model in a pool an agent should use. Its Rust core, libsy, is meant to be embedded in a gateway or agent harness: you keep the model clients and credentials, Switchyard returns the routing decision. Integrations exist for Agent Router (Envoy), LiteLLM (experimental), NeMo Relay and OpenRouter (model = nvidia/switchyard). A standalone server and experimental macOS/Windows desktop and terminal apps exist for local evaluation only; the desktop app's default profile targets Codex CLI.

The core idea

The post's claim: a repeated failure is a better difficulty signal than the task description. Switchyard's Execution (Stage) router acts on that. For each LLM call it reads the conversation's tool-result history and scores two directions:

The signed score is tanh-squashed into a confidence; the default threshold is 0.5. Repeated failures, critical errors and context compaction are hard overrides to the capable tier. After a switch to capable, capable_hold_turns (default 2) holds that tier for the next requests, so a switch plus hold is three requests on the capable model. Holding avoids flip-flopping between models, which would also throw away the prompt cache on each switch. Read and search tool contents do not count as errors unless the tool itself failed.

Other algorithms:

Auto, the default, is a fixed preset (Stage, efficient-first, threshold 0.5, no classifier call); it does not compare strategies at runtime.

Published numbers (vendor-run)

BenchmarkStrategySolvedCost per run
Terminal-Bench 2.1 (Opus 4.8 + GLM 5.2)Opus 4.8 only77.15%$148.83
GLM 5.2 only74.16%$71.31
Stage79.03%$109.81
LLM classifier79.40%$109.04
Escalation79.80%$122.63
DeepSWE v1.1, 113 tasks (GPT-5.6 Luna + Sol)Sol only62.8%$720.95
Stage61.36%$466.28
Luna only52.2%about $49

On Terminal-Bench the routed pool beats the strong model alone at about 26% lower cost. On DeepSWE, Stage keeps most of Sol's score for about 35% less. These are vendor runs, mostly three-run means.

Why it matters for agent builders

Caveats

R027 (provider tool-call error rates), R060 (planner and executor tiers in one family), R042 (routing a prover's compute on checker signals), R059 (memory and context cost).

Connections

See R085 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.