The first version of an AI pipeline almost always gets built against accuracy alone: does it extract the right fields, does it answer the question correctly. Cost shows up later, usually as a surprised look at the first month's invoice once volume is real instead of a test batch of twenty documents. It's a solvable problem, but only if you design for it instead of discovering it.
Where the cost actually comes from
It's rarely the model call itself that's the surprise, it's what's being sent with every call. Full document text on every request when only a section changed. Long few-shot examples repeated on every single call instead of cached. Retry logic that resends the whole prompt on a transient failure instead of just the failed piece. Multiply any of these by daily volume and a call that looked cheap in testing adds up fast.
- Route by difficulty: a cheaper, faster model for simple classification, a stronger model only for the cases that actually need it.
- Cache aggressively: repeated prompts, static instructions, and reference examples shouldn't be re-sent and re-billed every call.
- Batch where the workflow allows it: a nightly batch job is usually cheaper and simpler than a real-time call for anything that doesn't need to be instant.
- Set a hard ceiling: a per-run token budget that fails loudly is better than a runaway loop that fails expensively.
Model choice is a cost decision, not just a quality one
Teams default to the strongest available model for everything, because it's the safest choice for accuracy and nobody wants to be the one who picked the cheap model that got something wrong. But most pipelines have a mix of easy and hard cases, and routing the easy majority to a smaller, cheaper model while reserving the expensive one for genuinely hard cases can cut cost substantially without moving accuracy on the metric that matters.
Put a number on it before launch
Before anything goes live at volume, we run the cost math explicitly: expected calls per day, tokens per call, at the target model, times thirty. If that number surprises the client, it's much better to have that conversation before launch than after the first invoice. Cost is a design constraint like latency or accuracy. Treat it as one from the start and it never becomes a crisis.
Put this to work
3 linksRelated reading
Automating sensitive data without creating a compliance problem
When automation touches health, financial, or personal data, security can't be an afterthought. How to get the efficiency without the risk.
Agree terms → Least privilege → Secrets in a vault → Audit trail
Google Apps Script: the most underrated automation tool
Before you buy another SaaS subscription, look at what's already sitting inside your Google Workspace. It's more capable than people think.
Schedule → Pull data → Summarize → Email report
The security review every AI integration needs before launch
AI integrations open new paths for data to leak that a standard security checklist doesn't cover. Here's what to check before anything touches production.
Job description → Keycard → Manager → Probation → Weekly review
Continues · Got a workflow like this?