Your AI agent feels slow. Here is how to fix it

Users forgive an AI agent a lot, but they do not forgive slow. A reply that takes eight seconds to begin feels broken no matter how good it turns out to be, and people abandon the conversation, retry, or quietly decide the thing does not work. Here is the uncomfortable truth most teams discover after launch: the model generating tokens is rarely the main reason an agent feels sluggish. The seconds hide in the plumbing around it - tool calls made one at a time, a bloated prompt, a retrieval hop, a heavyweight model doing a job a small one could have done instantly. Fixing agent latency is partly about making it genuinely faster and partly about making it feel faster, and the two need different moves. Get both right and the same agent goes from "why is this so slow" to "that was instant".

The number that matters most: time to first token

Perceived speed is dominated by one metric: how long until the user sees something happen. A response that starts appearing in half a second and streams out over three seconds feels fast; the same total time spent staring at a blank spinner before everything arrives at once feels broken. This is why streaming the response - showing tokens as the model produces them rather than waiting for the full answer - is the single highest-leverage latency change you can make. It does not reduce the total generation time at all, but it collapses the wait the user actually experiences, because they are reading while the model is still writing. Pair streaming with an honest interim state for the slow parts: when the agent is calling a tool or retrieving data, say so, so the pause reads as work rather than a hang. Half of "fast" is real speed; the other half is never leaving the user looking at nothing.

The hidden cost: serial work you could parallelise

Once the response streams, go after the real time, and it is usually not where teams expect. The model itself is often the smaller slice; the bigger slice is everything the agent does in sequence before and between generations.

• Serial tool calls. An agent that needs three independent pieces of information - a customer record, an order status, a stock level - often fetches them one after another, so their latencies add up. If they do not depend on each other, fire them in parallel and the three-call wait collapses to the slowest single call.
• Oversized prompts. Every token you send is a token the model must read before it responds, and stuffing the whole knowledge base or a giant history into every request adds latency and cost for context the model does not need. Send what this turn actually requires, not everything you have.
• Unnecessary steps. A retrieval hop on a question that did not need one, a reasoning chain on a trivial request, a tool call you could have cached - each adds a round trip. Do the cheap check first and skip the expensive step when it is not warranted.

Caching is a big lever here too: an answer you already computed is the fastest answer there is, and the economics of that are the subject of cutting LLM cost with caching - the same cache that saves money also saves time.

Route by difficulty instead of paying premium latency everywhere

Not every turn deserves your biggest model. A simple classification, a short factual reply, a routine extraction - a small fast model handles these in a fraction of the time, and the user cannot tell the difference except that it was quicker. Model routing means sending each request to the cheapest, fastest model that can do it well, and reserving the large slow model for the genuinely hard turns that need its reasoning. This cuts both latency and cost on the bulk of traffic, because most real conversations are mostly easy turns with a few hard ones. The design work is deciding how to classify difficulty and where to draw the line, and it pairs naturally with the build-versus-buy and hosting choices we cover in when self-hosting an LLM beats an API. Where you run the model matters for latency too - a model close to your users on infrastructure you control, like the setups we build on AI on AWS, avoids the round trips a distant endpoint adds.

Measure it, then keep measuring

You cannot fix latency you do not measure, and averages lie. Track time to first token and total time separately, and look at the worst cases - the 95th percentile - not just the mean, because it is the slow tail that users remember and complain about. Break the timeline down: how much time in the model, how much in tools, how much in retrieval, how much in your own code. That breakdown almost always reveals a surprise - a tool that is slower than the model, a retrieval step nobody profiled, a prompt that doubled in size over three months of additions. Then keep watching in production, because latency drifts as data grows, traffic patterns shift, and features accrete. Treating speed as a live signal rather than a one-time launch check is part of proper agent observability, and it is how the agents we build stay fast after launch instead of quietly getting slower.

Frequently asked questions

What is time to first token and why does it matter more than total time?

Time to first token is how long the user waits before anything appears on screen. It matters more than total time because perceived speed is dominated by the initial wait - a response that starts in half a second and streams out feels fast, while the same total time spent staring at a blank spinner feels broken. Optimising when the user first sees output, primarily through streaming, transforms how fast your agent feels even if the total generation time is unchanged.

Does streaming actually make the agent faster?

Not in total time - the model takes just as long to generate the full answer. What streaming changes is the wait the user experiences: instead of staring at nothing until everything arrives, they start reading immediately as tokens appear. Since people read at roughly the speed a model generates, the response feels close to instant. It is the highest-return latency change precisely because it costs little to implement and dramatically improves perceived speed.

How does model routing reduce latency without hurting quality?

By matching each request to the smallest model that can handle it well. Most turns in a real conversation are easy - a classification, a short reply, a routine extraction - and a small fast model answers them in a fraction of the time with no visible quality loss. You reserve the large, slower model for the genuinely hard turns that need its reasoning. Since the bulk of traffic is easy turns, routing cuts average latency and cost substantially while the quality on hard turns stays exactly where it was.

Where should I look first if my agent is slow?

Measure the breakdown before changing anything: how much time is in the model, in tool calls, in retrieval, and in your own code. Teams routinely assume the model is the bottleneck and find it is serial tool calls or an oversized prompt instead. Add streaming first for an immediate perceived-speed win, then attack whatever the breakdown shows is actually eating the seconds. Guessing wastes effort; the profile points you straight at the fix.

The short version: a slow agent loses users before they ever see how good it is. Stream the response so the wait feels short, parallelise the serial tool calls and trim the bloated prompts where the real seconds hide, route easy turns to a fast small model, and measure time to first token and the slow tail so you fix what actually matters. Do that and your agent stops feeling like it is thinking and starts feeling like it just knows.