ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale

SummaryWe introduce ThunderAgent, a system for high throughput agentic inference. By introducing a novel program abstraction for agentic LLM request scheduling, ThunderAgent achieves up to 2.5× higher single-node throughput in our synthetic data generation pipeline, and delivers 2.4× speedup on an 8-node cluster with near-linear throughput scaling with respect to GPU nodes.Key Results→ More than 2× single-node throughput, with roughly 10× lower P50 latency at high concurrency→ 2.4× speedup on 8 nodes, near-linear scaling from 16 to 64 GPUs→ Drop-in: one program_id field, OpenAI-compatible, works with your existing engine-level optimizations (like speculative decoding)ThunderAgent was accepted to ICML 2026 as a Spotlight paper.This paper was a collaboration between researchers at Georgia Institute of Technology, the University of Illinois Urbana-Champaign, Carnegie Mellon University, and Together AI.LLMs are increasingly deployed as agents. Systems like Claude Code, Codex, and OpenClaw reason, call tools, read results, and reason again, often for dozens of turns before completing a task. Training these agents requires large scale synthetic data generation, since natural web corpora do not contain multi-turn, tool-using agent trajectories. To generate agentic datasets like our recently released CoderForge, we need to run agentic inference at high concurrency.However, existing inference engines schedule at the request level, where each LLM call is an independent unit with no awareness that it belongs to a longer, multi-turn workflow. When an agent pauses for a tool call, its KV cache can be evicted to make room for other requests, only to be recomputed from scratch when the agent resumes. This creates fundamental inefficiencies in engines such as SGLang, vLLM, and TensorRT-LLM when serving agentic workloads at high concurrency, including KV cache thrashing, imbalanced workloads across mutli-nodes, and resource waste in tool management. In this blog, we focus on the most critical: KV cache thrashing. We refer readers to our paper for a more comprehensive discussion.What hinders agentic inference at scaleAn agentic workflow alternates between two phases: the GPU-heavy reasoning phase where the model is generating tokens and the GPU-idle acting phase where the agent is waiting for a tool like a compiler to return. When hundreds of agents run concurrently, their KV caches grow at each phase, competing for limited GPU memory. Under this memory pressure, a traditional inference engine like vLLM would evict KV caches using a simple policy: least recently used, regardless of whether that cache will be needed again in seconds.The result is a vicious cycle. Agent A pauses for a tool call. Its KV cache gets evicted to make room for Agent B's prefill. When Agent A's tool returns, the engine must recompute Agent A's entire conversation history from scratch, which in turn evicts cache for Agent C's context. At high concurrency, this cascade of evictions and recomputations can lead to severe throughput and latency degradation. We refer to this problem as KV cache thrashing.Simply throwing more GPU nodes at the problem doesn't fully resolve it. Existing multi-node routers such as SGLang Model Gateway pin each agent to a fixed node to preserve cache locality. But because agentic context lengths grow unpredictably, some nodes are assigned agents with long contexts that exhaust memory, while others sit idle with capacity to spare. The overloaded nodes still face thrashing.Adopting KV cache offloading doesn't resolve the problem either. Strategies like LMCache and HiCache offload KV caches to CPU memory or disk, which expands total cache capacity but only delays thrashing. When the working set of concurrent agents exceeds all available storage tiers, evictions resume and the same vicious cycle returns.Under memory pressure, a request-level engine evicts an agent's KV cache the moment it pauses for a tool call, then must recompute the whole conversat