How agent architectures affect model serving characteristics and performance

In a typical settings we always treat the model API as a blackbox that serves a model of a certain capacity and the job of an agent engineer is to build a agent system (harness) that uses the capability of said model to the maximum to solve the task(s) at hand. For example, GPT 5.x can handle certain complexity of problems until a certain context length, therefore the harness should employ context management in that matches those constraints. However, the agent architecture or the agent topology that's chosen can very heavily affect the overall performance (mainly throughput) of the system as a whole. This is more relevant for internal model deployments where the model serving setup has to cater for a relatively lesser number of tasks/agents compared to that of public LLM APIs like that of OpenAI or Anthropic.

With many optimizations like prefix-caching, continuous batching, speculative decoding etc to choose from, it becomes even more important that the agent architecture and the serving config would need to go hand in hand when it comes to local/internal deployments.

To see how agent architecture affects serving we will setup 2 agent architectures:

• Single Agent (ReAct based)
• Orchestrator: A parent agent that can break the task into sub-problems and spawn sub agents to solve them

Experiment Setup

The serving stack. We simulate a vLLM-style inference server running Kimi K2 — a 1T-parameter Mixture-of-Experts model with 32B active parameters per token — on a single 8×NVIDIA B200 node (1,536 GB HBM3e total). This is the reference single-node configuration for K2: FP8 weights occupy ~1,000 GB, leaving roughly 476 GB for the KV cache. The engine models continuous batching (up to 128 sequences decoding together), chunked prefill at ~60k tokens/s aggregate, and decode at ~32k tokens/s per node — in line with published B200 numbers for this model class.

One architectural detail matters a lot here: Kimi K2 uses MLA (Multi-head Latent Attention), which compresses the KV cache to ~69 KB per token — about 14× smaller than a conventional GQA model of similar size. 476 GB therefore holds ~6.8M tokens of KV. How much of that an agent architecture can actually reuse turns out to be the whole story.

The workload. Instead of synthetic prompts, we replay 295 real agent trajectories from the Toolathon benchmark (a tool-use benchmark where agents operate MCP tools: filesystem, browsers, cloud APIs, terminals, spreadsheets…). We use the runs produced by kimi-k2-0905 itself, and reconstruct every LLM request with realistic token counts. A typical task makes ~28 LLM calls; the median request carries a ~15.5k-token prompt and produces only ~53 output tokens. Agentic workloads are prefill-dominated — keep that in mind.

Both architectures solve the same 295 tasks. The ReAct agent replays each trajectory sequentially, its prompt growing by one assistant turn plus one tool result per step. The orchestrator makes one planning call, then splits the task by tool domain into parallel subagents (median 3, up to 8 concurrent), each carrying only the shared system prompt, the task, its tool subset, and its own history. We submit all 295 tasks at once and measure the makespan — the time to finish everything.

Results

Baseline: continuous batching, no caching

With plain serving, every request is prefilled from scratch. The queue you see below is the prefill queue: prompts are huge, outputs are tiny, so the GPU spends its life reading prompts, not writing answers. Hit play — watch how differently the two architectures load the server.

The orchestrator wins even here — not because parallelism speeds up a prefill-bound GPU (it doesn't; the queue just gets longer), but because subagents send 2.2× fewer prompt tokens. When the bottleneck is reading prompts, the best architecture is the one that writes fewer of them.

Now add prefix caching

Prefix caching (vLLM's APC, SGLang's RadixAttention) stores the KV tensors of computed prompt prefixes in GPU memory — in 16-token blocks, content-hashed and chained — so that a new request sharing a prefix can skip recompute entirely. Blocks no longer referenced are evicted LRU when the cache fills. For agent loops this is a natural goldmine: step k's prompt is a strict prefix of step k+1's.

But the cache is bounded — 476 GB here — and 295 agents generate more than 6.8M tokens of distinct KV. Watch the cache fill, and watch what gets evicted:

The overall picture

Two effects stack. The orchestrator sends 2.2× fewer prompt tokens because subagents don't drag the whole conversation behind them — decisive in a prefill-bound regime. And its cache working set is smaller: shared shallow prefixes instead of deep per-agent histories, so it saturates the prefix cache at roughly half the memory budget (~100 GB vs ~200 GB in our sweeps) and suffers fewer than half the evictions. Under a full 476 GB cache both topologies approach their hit-rate ceilings (91.5% vs 87.2%), and the orchestrator still finishes first.

The caveat that surprised us: parallelism by itself buys nothing on a prefill-bound server. The orchestrator's win is an information-locality win — shorter prompts, smaller working set, gentler cache pressure. Architecture is a memory-system decision as much as a reasoning-quality one.

Caveats

• This is a simulation: prefill/decode rates are calibrated to published 8×B200 figures for Kimi K2's model class, not measured on live hardware.
• Token counts use the cl100k_base tokenizer as a proxy for Kimi K2's tokenizer — close enough for relative comparisons, not exact.
• The orchestrator's decomposition is synthesized by grouping each trajectory's tool calls by tool domain. Real planners won't partition this cleanly.
• Tool execution latency is synthetic (exponential, mean 0.6s); agent "think time" between steps comes only from the trace structure.
• Decode memory-bandwidth contention is modeled and turns out negligible here — MLA's KV is small enough that decode stays weight-bound at batch 128. With a GQA model (or much longer contexts) this changes.
• All 295 tasks arrive at once (fixed-batch makespan). A Poisson arrival process would change queueing dynamics, though not the token-volume conclusions.

Code, traces, and the full simulation report are available on request — the animations above replay real event streams from the simulator.