R027 · Infrastructure for agents at scale › Inference providers and serving reliability

The Tool-call index: the same model breaks tool calls at very different rates by provider

Read the original

agent-reliabilitybenchmarksinference-costagent-harnessobservabilitydev-infra

TL;DR

.txt (the company behind the Outlines structured-generation library) launched the Tool-call index, a weekly table of how often each inference provider returns invalid tool calls for the same open-weight model (launch post, with a comment from cofounder Rémi Louf). The claim: the deployment, not just the model, decides whether your agent's tool calls parse.

What is measured

The numbers are OpenRouter's published per-endpoint tool-call error rates, not traffic .txt runs itself. Of the requests that ended in tool calls, the error rate is the share with at least one invalid call: bad JSON, an unknown function name, or arguments that fail the tool's schema.

Why it matters

Agent harnesses usually catch an invalid call and feed the error back for a retry. That hides the failure from dashboards: it shows up as extra tokens, extra latency, and occasionally a derailed run, not as an error. A 5% rate on a 30-call task means about one retry per run on average; at 30% the agent spends a large share of its budget repairing malformed calls.

The likely causes are deployment choices the model card doesn't describe: aggressive quantization (fp4 and fp8 variants are often among the worst for a given model), the provider's chat template and tool-call parser, and whether the server applies grammar-constrained decoding.

Tradeoffs and caveats

Implications

For teams running agents on open-weight models through a router or a third-party host, provider choice is a reliability parameter that belongs in config review and evals, not only in cost and latency comparisons.

Sources

Connections

See R027 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.