TL;DR
.txt (the company behind the Outlines structured-generation library) launched the Tool-call index, a weekly table of how often each inference provider returns invalid tool calls for the same open-weight model (launch post, with a comment from cofounder Rémi Louf). The claim: the deployment, not just the model, decides whether your agent's tool calls parse.
About 49 open-weight models and 53 to 57 providers (the site's pages disagree), last 7 days, data as of 2026-11-10.
Headline: across models, the median ratio of the worst provider's error rate to the best's is about 13×.
Extremes in the current data: gpt-oss-120b at 58.42% on one provider, Llama 3.1 8B at 51.30% on another, while the best provider for most models sits under 0.1%.
Each model gets a ready-made OpenRouter routing object (
orderthe three most reliable,ignoreanything at 3% or worse), plus anllms.txtand an installable agent skill so an agent can choose the provider itself.
What is measured
The numbers are OpenRouter's published per-endpoint tool-call error rates, not traffic .txt runs itself. Of the requests that ended in tool calls, the error rate is the share with at least one invalid call: bad JSON, an unknown function name, or arguments that fail the tool's schema.
Provider means deployment. Each quantization or tier (
fp8,fp4,fast,zdr) is listed as its own provider; regions are merged.Threshold. A provider is ranked for a model only at 1,000 or more tool calls a day. Providers whose own docs don't document tool calling are excluded by hand.
Undercount by design. Responses the provider couldn't turn into a tool call at all are not counted, so real failure rates are higher.
Structured outputs are tracked separately with the same rules, because providers that are fine at tool calls can be poor at
response_format, and the reverse.
Why it matters
Agent harnesses usually catch an invalid call and feed the error back for a retry. That hides the failure from dashboards: it shows up as extra tokens, extra latency, and occasionally a derailed run, not as an error. A 5% rate on a 30-call task means about one retry per run on average; at 30% the agent spends a large share of its budget repairing malformed calls.
The likely causes are deployment choices the model card doesn't describe: aggressive quantization (fp4 and fp8 variants are often among the worst for a given model), the provider's chat template and tool-call parser, and whether the server applies grammar-constrained decoding.
Tradeoffs and caveats
Vendor data, vendor interest. .txt sells constrained decoding that makes invalid tool calls impossible, so the index is also marketing for the problem it solves. The data source is third-party, which helps, but the framing and thresholds are .txt's.
OpenRouter only. It sees only the traffic routed through OpenRouter. A provider's direct API, private deployments and self-hosting are not measured.
Mixed workloads. Rates are not normalized for task difficulty or tool-schema complexity. A provider serving harder agent traffic will look worse for reasons that aren't its fault.
The headline is a median of ratios, with the best rate floored at 0.1%. Ratios between small numbers swing a lot week to week, and the post's example figure (over 30% for gpt-oss-120b) is already lower than the current table.
Invalid is not wrong. A schema-valid call to the wrong tool with the wrong intent counts as a success. This measures syntax, not agent quality.
Moving target. Providers patch parsers and templates often, which is why the routing objects carry an expiry (
valid_until) and a confidence flag.
Implications
For teams running agents on open-weight models through a router or a third-party host, provider choice is a reliability parameter that belongs in config review and evals, not only in cost and latency comparisons.
Log invalid tool calls and retries per provider and model before the harness repairs them, and alert on rate changes
Pin providers (and quantization) for production agents instead of letting a router pick on price
Re-run your own tool-use eval when a provider or quantization changes, not just when the model changes
Use constrained decoding or strict mode wherever the server supports it
Track structured-output reliability separately from tool calls
Treat any public index as a starting shortlist; confirm on your own schemas and traffic
Related entries
R024 InfraRSI: the same lesson from training, that configuration and numerical choices outside the model can silently change outcomes, and only measurement catches it.
R020 The model proposes, the harness enforces: schema validation is harness-side, and this index measures how often it has to intervene.
R006 6 must-know terms for AI agent harnesses: tool-call retries are part of the loop the harness owns.
R026 HybridAgent: another reminder that agents built on open weights inherit the serving stack's behavior.
Sources
Tool-call index, .txt, data as of 2026-11-10; also its method and findings page and llms.txt
.txt launch post on LinkedIn, 2026-11-10