R115 · Infrastructure for agents at scale › Inference providers and serving reliability

Cutting agent batch makespan: schedule the longest chains first

Read the original

inferenceschedulingevaluationrlkv-cache

Source: Feng et al. (Ling team), Reducing Batch Makespan for LLM Agent Workloads through Inference-Environment Coordination, arXiv 2611.16661 (submitted 2026-11-12). Highlighted by DAIR.AI.

The problem

Benchmark evaluation, synthetic-data generation and RL rollouts all run batches of agent tasks, and a batch is done only when its slowest task finishes. Agent tasks alternate between LLM inference (GPU) and environment steps (sandboxes, tools), so each side idles while waiting for the other, and a few long trajectories leave most workers idle at the tail.

The techniques

  1. Workload-based resource model. For hybrid models with both attention and recurrent layers, it splits GPU memory between KV cache and recurrent state and sets a reference concurrency that exploits parallelism without heavy recomputation.

  2. Trajectory-aware routing. Each trajectory stays on the worker that holds its cache, balanced against worker load.

  3. Chain-Advancing Action Scheduling (CAAS). When environment capacity is limited, it decouples admitting a sample from executing its environment step, and prioritizes the chains with the most observed progress so the long tail starts and finishes sooner.

  4. Controlled replay. Real inference combined with recorded environment responses and durations, so each optimization can be ablated without re-running live environments.

Results (as reported)

On BFCL, SWE-bench Multilingual, WideSearch and MCP-Atlas with Ling-3.0-Flash and Ling-3.1-Flash: batch makespan down 10.4% to 65.0% with evaluation scores unchanged. The methods are in production use for training and evaluating the Ling-3.x series.

Tradeoffs and failure modes

Review checklist

Other entries on inference serving and reliability (index subtopic 2.4) and on evaluation infrastructure (6.3), including R107.

Connections

See R115 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.