Inferact 谈 vLLM 跑智能体负载的缓存要点

Agent traffic is a strange workload for an inference server.

Each turn resends a long, nearly identical prefix (system prompt, tool schemas, history) and asks for a short tool call back.

At DigitalOcean’s Open Intelligence Summit, Yifan Qiao of Inferact is giving a Serve track session on running vLLM on agentic production workloads.

That shape of traffic is why prefix caching and KV-cache reuse matter more for agents than raw tokens per second. A cache miss on a long prefix means paying for the full prefill again, on every turn of a long task.