You picked your agent model on vibes, not evidence

Here is how most teams choose the model behind an AI agent: someone read that a particular model topped a benchmark, or used it in a chat and liked it, so that is what ships. Then the agent goes to production and the bill is higher than expected, the responses are slower than the demo, and every so often it calls the wrong tool or returns malformed output that breaks the next step. The model choice was made on vibes, and vibes do not survive contact with production. The model you pick shapes your agent cost, speed, and reliability more than any prompt tweak will - so it deserves an actual decision, not a leaderboard glance.

Why the leaderboards mislead you

Public benchmarks measure a model against a fixed set of questions - reasoning puzzles, coding challenges, exam-style tasks. They are useful for a rough sense of a model's ceiling. They are close to useless for predicting how it will run your agent, for one blunt reason: your agent is not taking an exam. It is calling tools, filling structured outputs, following a multi-step workflow, and staying reliable across thousands of runs. A model can be brilliant at a hard maths problem and mediocre at reliably emitting valid JSON for your schema, or at choosing the right tool from your set of twelve.

Two other traps hide in the rankings. First, contamination and overfitting: popular benchmarks leak into training data, so scores drift upward without the real-world skill following. Second, the average hides the tail - a model with a great mean score can fail badly on the specific shape of task you actually run. The only benchmark that predicts your production performance is one built from your own tasks. Everything else is a starting shortlist, not a decision.

The four things that actually decide it

For a production agent, work through these in order. The first one that fails is usually your answer, and it is rarely "which model is smartest".

• 1. Tool-use and structured-output reliability. An agent lives or dies on calling the right tool with valid arguments and returning output your code can parse. A model that is 98% reliable here versus one that is 90% is the difference between an agent that works and one that needs a babysitter. Test this first, on your tools - it is the constraint that breaks most agents, and it is the backbone of reliable tool calling.
• 2. Cost per completed task. Not cost per token - cost per finished job. A cheaper-per-token model that fumbles, retries, and makes extra tool calls can cost more per successful task than a pricier model that gets it right in one pass. Measure the whole run, not the sticker price.
• 3. Latency. If a human waits on the agent, time-to-first-token and total response time decide whether it feels usable. A slightly less capable model that responds in a second can beat a stronger one that takes eight - the latency levers matter as much as the model.
• 4. Context-window fit. How much do you need to put in front of the model - retrieved documents, conversation history, tool results? Most agents need far less than the largest windows advertise, and stuffing the window hurts both cost and accuracy. Size to what the task genuinely needs.

Cost per token is the wrong number

Model pricing is quoted per token, so teams compare per-token rates and pick the cheaper one. That comparison can lead you straight to the more expensive agent. Here is why: an agent's real cost is the whole run - the system prompt, the retrieved context, every tool-call round trip, and the retries when something comes back malformed. A weaker model that needs two attempts, calls an extra tool to recover, and occasionally loops burns far more tokens per completed task than its headline rate suggests.

The number to optimize is cost per successful task. Run both candidates over a representative batch, count total tokens and tool calls to a correct result, and compare that. Very often the "expensive" model is cheaper in practice because it finishes cleanly. And where raw spend genuinely matters, caching repeated context is a bigger lever than shaving the per-token rate - the approach in our guide to prompt and semantic caching.

You do not have to pick just one

The framing of "which single model" is often the mistake. Real agents run a mix of steps: a lot of simple classification, extraction, and routing, and a little genuinely hard reasoning. Sending every step to your most capable, most expensive model means paying premium rates for work a small model does perfectly well. Model routing - a small fast model for the easy, high-volume steps and a strong one for the hard ones - usually cuts cost sharply with no drop in quality, because most traffic never needed the big model. That routing pattern is part of cutting agent latency, and it is where a lot of real savings live.

One more decision sits underneath all of this: closed API model versus open model you host. A hosted API is the fastest way to ship and usually the right default. Self-hosting an open model can win on cost or data control at high, steady volume - but only past a break-even point, and with real operational work attached. We put actual numbers on that in when self-hosting an LLM beats an API. For most teams starting out, a hosted model behind a clean interface is the pragmatic choice.

Make the choice reversible

Whatever you pick today will not be the best choice in six months - prices drop, new models land, and your workload shifts. The teams that stay fast are the ones who never hard-coded a single provider into their agent. Put the model behind a thin interface, keep your eval harness ready so you can re-score a new model on your real tasks in an afternoon, and treat the model as a swappable component. Then a better or cheaper option is a config change and a test run, not a rewrite. That is the difference between a decision that ages well and one you regret. This is how our AI agent team and AI development practice build every production agent.

Frequently asked questions

Should I just default to the biggest, most capable model?

Only if you have proven you need it. The biggest model is the safe demo choice, but in production it is often slower and more expensive than the task requires. Start by testing whether a mid-tier model clears your reliability bar on your tasks - frequently it does, at a fraction of the cost and latency. Reserve the top model for the steps that genuinely need it.

How big an eval do I need to choose well?

Smaller than you think. A few dozen representative tasks with known-good outcomes - covering your common cases and the tricky edge ones - is enough to separate models on reliability and cost. The point is not statistical perfection; it is replacing a hunch with evidence from your actual workflow. You expand the set over time as you learn where the agent fails.

Does the newest model release mean I should switch?

Not automatically. A new release is a reason to re-run your eval, not to swap on trust. If it wins on cost per task, reliability, or latency for your workload, switch - and because you kept the model swappable, that is cheap to do. If it does not move your numbers, the hype is not your problem to chase.

Is the model choice more important than the prompt?

They work together, but the model sets the ceiling. A great prompt on a model that is unreliable at tool use will still break; a solid prompt on a reliable model will not. Get the model right first, then tune the prompt and the surrounding scaffolding - retries, validation, routing - to get the most out of it.

The short version: your agent model is an engineering decision, not a leaderboard pick. Judge it on tool-use reliability, cost per completed task, latency, and context fit - measured on your own work - route across models instead of forcing one, and keep the choice reversible. Do that and the model stops being a bet you hope pays off and becomes a component you can reason about. If you want it chosen on evidence, our AI audit is where we start.