How agent architectures affect model serving characteristics and performance

When building agents, the default assumption is to treat the model API as a blackbox that serves a model of a certain capability and the job of an agent engineer is to build a agent system (harness) that uses the capability of said model to the maximum to solve the task(s) at hand. For example, GPT 5.x can handle certain complexity of problems until a certain context length, therefore the harness should employ context management to match those constraints. However, the agent architecture or the agent topology that's chosen can very heavily affect the overall performance (mainly throughput) of the system as a whole. This is more relevant for internal model deployments where the model serving setup has to cater for a relatively lesser number of tasks/agents.

With many optimizations like prefix-caching, continuous batching, speculative decoding etc to choose from, it becomes even more important that the agent architecture and the serving config would need to go hand in hand when it comes to local/internal deployments.

To see how agent architecture affects serving we will setup 2 agent architectures:

• Single ReAct Agent
• Orchestrator: A parent agent that can break the task into sub-problems and spawn sub agents to solve them

Experiment Setup

The serving stack

We simulate a vLLM-style inference server running Kimi K2 — a 1T-parameter Mixture-of-Experts model with 32B active parameters per token — on a single 8×NVIDIA B200 node (1,536 GB HBM3e total). This is the reference single-node configuration for K2: FP8 weights occupy ~1,000 GB, leaving roughly 476 GB for the KV cache. The engine models continuous batching (up to 128 sequences decoding together), chunked prefill at ~60k tokens/s aggregate, and decode at ~32k tokens/s per node — in line with published B200 numbers for this model class.

The workload

Instead of synthetic prompts, we replay 295 real agent trajectories from the Toolathon benchmark (a tool-use benchmark where agents operate MCP tools: filesystem, browsers, cloud APIs, terminals, spreadsheets etc.,). We use the runs produced by kimi-k2-0905 itself, and reconstruct every LLM request with realistic token counts. A typical task makes ~28 LLM calls; the median request carries a ~15.5k-token prompt and produces only ~53 output tokens.

Both architectures solve the same 295 tasks. The ReAct agent replays each trajectory sequentially, its prompt growing by one assistant turn plus one tool result per step. The orchestrator makes one planning call, then splits the task by tool domain into parallel subagents (median 3, up to 8 concurrent), each carrying only the shared system prompt, the task, its tool subset, and its own history. We submit all 295 tasks at once and measure the makespan — the time to finish everything.

Results

Baseline: continuous batching, no caching

With plain serving, every request is prefilled from scratch. The queue you see below is the prefill queue: prompts are huge, outputs are tiny, so the GPU spends its life reading prompts, not writing answers. Hit play — watch how differently the two architectures load the server.

Enter prefix-caching

Prefix caching (vLLM's APC, SGLang's RadixAttention) stores the KV tensors of computed prompt prefixes in GPU memory — in 16-token blocks, content-hashed and chained — so that a new request sharing a prefix can skip recompute entirely. Blocks no longer referenced are evicted LRU when the cache fills. For agent loops this is a natural goldmine: step k's prompt is a strict prefix of step k+1's.

But the cache is bounded — 476 GB here — and 295 agents generate more than 6.8M tokens of distinct KV. Watch the cache fill, and watch what gets evicted:

The overall picture

The caveat that surprised us: parallelism by itself buys nothing on a prefill-bound server. The orchestrator's win is an information-locality win — shorter prompts, smaller working set, gentler cache pressure. Architecture is a memory-system decision as much as a reasoning-quality one.

Caveats

• Agentic workloads are prefill-dominated: the median request reads a ~15.5k-token prompt and writes only ~53 tokens, so everything above speaks to a prefill-bound server. Decode-heavy workloads (long reasoning traces, long generations) would shift the bottleneck — and possibly the conclusions.
• This is a simulation: prefill/decode rates are calibrated to published 8×B200 figures for Kimi K2's model class, not measured on live hardware.
• Token counts use the cl100k_base tokenizer as a proxy for Kimi K2's tokenizer — close enough for relative comparisons, not exact.
• The orchestrator's decomposition is synthesized by grouping each trajectory's tool calls by tool domain. Real planners won't partition this cleanly.
• Tool execution latency is synthetic (exponential, mean 0.6s); agent "think time" between steps comes only from the trace structure.
• Decode memory-bandwidth contention is modeled and turns out negligible here — MLA's KV is small enough that decode stays weight-bound at batch 128. With a GQA model (or much longer contexts) this changes.
• All 295 tasks arrive at once (fixed-batch makespan). A Poisson arrival process would change queueing dynamics, though not the token-volume conclusions.

Code, traces, and the full simulation report are available on request — the animations above replay real event streams from the simulator.