AI Agents
Cost and Latency Budgets for Production AI Agents
Treat cost and latency like features with specs, not variables to optimise later.
In a demo, cost and latency are invisible. In production, they are the difference between a system that scales and one that gets capped at 1,000 runs a day. The teams that ship treat cost and latency as features with specifications, not variables to optimise after launch.
Set the budget before you build
Define a cost-per-task and a latency target upfront, derived from the value of the task. If a task saves $3 of human effort, a $0.10 agent cost is fine; a $0.40 cost is not. If a customer is waiting, 4 seconds is acceptable and 30 is not. Write the numbers down and design to them.
Model the cost before you incur it
Estimate tokens per task, multiply by model pricing, and add tool-call overhead. A surprising number of agents are economical at 100 runs and ruinous at 100,000. The model is cheap per call but the calls compound; do the multiplication early.
Cache aggressively
Many agent tasks repeat — the same retrieval, the same classification, the same summary of an unchanged document. Cache the expensive steps and invalidate on change. Caching is the single most effective cost lever, and the most often ignored.
A budget is a design constraint that produces better systems. Agents built to a cost and latency spec are leaner, faster, and more likely to survive contact with a finance review.
