Latency budgets for agentic systems

Forty seconds is a different product

Multi-step systems accumulate latency the way monoliths accumulate dependencies: invisibly, then all at once. Each step is defensible. A retrieval call here, a verification pass there, a second opinion on the risky ones. Nobody adds thirty seconds; everybody adds three.

Users notice long before the evaluation does, because the evaluation is measuring answer quality and the user is measuring whether it was worth waiting. Those are not the same axis, and the second one decides whether the feature gets used.

Where the time actually goes

When we have instrumented these systems the distribution is rarely what the team expects. Model inference is usually not the largest share. The largest shares tend to be:

• Serialised steps that had no data dependency on each other and could have run at once.
• Retrieval against a store that was sized for correctness and never for latency.
• Verification passes that fire on every request because it was easier than deciding when they are needed.
• A cold path — a rarely used tool, an unpooled connection — that is fast on average and terrible at the ninety-fifth percentile.

The last one is worth dwelling on. Averages are close to meaningless here. A system with a two-second mean and a twenty-second tail is experienced as a twenty-second system, because the tail is what people remember and repeat.

Budgets, not optimisations

The approach we are testing is the one that works for page weight: agree a number first, allocate it across steps, and treat an over-run as a design problem rather than something to profile later.

A budget forces the useful conversations early. If the verification pass costs four seconds and the budget is six, then it cannot run on every request, and the team has to decide what makes a request risky enough to deserve it. That is a better conversation than the one where an engineer is asked to make everything faster the week before launch.

Total time is a design constraint. It should be decided at the same meeting as the scope.

Interface consequences

Some latency cannot be removed, and the honest response is to change the interface rather than keep pretending the wait is not happening.

Answering early and refining works when the first answer is genuinely useful and the refinement is visibly an improvement, not a correction — watching a system contradict itself is worse than waiting. Showing the steps works when the steps mean something to the person reading them, and is theatre when they do not. And for genuinely long work, the correct interface is not a spinner at all; it is telling the user it will take a while and giving them somewhere to go.