Source: Feng et al. (Ling team), Reducing Batch Makespan for LLM Agent Workloads through Inference-Environment Coordination, arXiv 2611.16661 (submitted 2026-11-12). Highlighted by DAIR.AI.
The problem
Benchmark evaluation, synthetic-data generation and RL rollouts all run batches of agent tasks, and a batch is done only when its slowest task finishes. Agent tasks alternate between LLM inference (GPU) and environment steps (sandboxes, tools), so each side idles while waiting for the other, and a few long trajectories leave most workers idle at the tail.
The techniques
Workload-based resource model. For hybrid models with both attention and recurrent layers, it splits GPU memory between KV cache and recurrent state and sets a reference concurrency that exploits parallelism without heavy recomputation.
Trajectory-aware routing. Each trajectory stays on the worker that holds its cache, balanced against worker load.
Chain-Advancing Action Scheduling (CAAS). When environment capacity is limited, it decouples admitting a sample from executing its environment step, and prioritizes the chains with the most observed progress so the long tail starts and finishes sooner.
Controlled replay. Real inference combined with recorded environment responses and durations, so each optimization can be ablated without re-running live environments.
Results (as reported)
On BFCL, SWE-bench Multilingual, WideSearch and MCP-Atlas with Ling-3.0-Flash and Ling-3.1-Flash: batch makespan down 10.4% to 65.0% with evaluation scores unchanged. The methods are in production use for training and evaluating the Ling-3.x series.
Tradeoffs and failure modes
Progress is a proxy for length. Scheduling on observed progress assumes long chains reveal themselves early; tasks that turn long late still land in the tail.
Cache affinity vs balance. Pinning trajectories to their cache can hot-spot a worker; the router has to trade reuse against load.
Replay fidelity. Recorded environment timings may not match live behaviour under different load, so ablation numbers are indicative.
Model-specific memory model. The KV vs recurrent-state split matters for hybrid architectures; pure-transformer stacks get less from step 1.
Review checklist
Plot per-worker utilization across a batch: is the end of the run a long idle tail?
Are GPU inference and sandbox capacity scheduled together or by separate systems?
Do trajectories keep their KV cache across turns, or get re-prefilled on another worker?
Can you replay environment responses to test scheduler changes cheaply?
Before adding GPUs, try longest-first or progress-first ordering.
Related entries
Other entries on inference serving and reliability (index subtopic 2.4) and on evaluation infrastructure (6.3), including R107.