What we learned putting frontier models in production this year

A model playground is a different animal from a model pipeline. In the playground you write a careful prompt, read the answer, maybe tweak it, and move on. In production the same prompt runs a thousand times a day against inputs you didn't anticipate, and the failure modes that show up have almost nothing to do with the model being 'smart enough.' They're about plumbing.

The failure modes nobody demos

Every demo shows the happy path: clean input, clear question, good answer. Production input is messy. A support ticket with three languages mixed together. A form field somebody pasted an entire email into. An invoice PDF that's actually a scanned photo rotated ninety degrees. The model handles more of this than you'd expect, but not all of it, and the gap is where your pipeline needs to be honest about what it doesn't know.

• Rate limits, access to the top-tier models is still capacity-constrained; a burst of requests queues or errors, and your retry logic needs backoff, not a tight loop.
• Latency, a frontier model's responses can take several seconds longer than a smaller, faster one, which matters a lot if a human is staring at a spinner.
• Cost, the frontier tier runs roughly an order of magnitude more expensive per token than the fast tier; using it for a task the cheaper model handles fine is money left on the table.
• Non-determinism, the same prompt can return a differently shaped answer twice; anything downstream that parses the output needs to tolerate that, not assume a fixed format.

Function calling changed the shape of the problem

Before function calling shipped, getting structured output out of a chat model meant asking nicely for JSON and writing a parser resilient enough to survive the model occasionally wrapping it in a sentence anyway. Function calling turns that into a defined schema the model fills in, which cuts out an entire category of brittle string-parsing code. It's not magic: the model can still pick the wrong function or leave a field empty, so you still validate before you trust it. But the failure mode moved from 'unparseable text' to 'a malformed but structured object,' which is a much easier thing to catch.

The model got good enough that the bottleneck moved from intelligence to plumbing. Nobody puts plumbing in a demo.

The two-model pattern

The pattern that's actually holding up in our pipelines: use a fast, cheap model for triage and simple extraction, where speed and cost matter more than nuance, and reserve the strongest model for the step that genuinely needs the extra reasoning, like resolving ambiguity or handling the fifteen percent of cases that don't fit the common shape. Routing between them isn't complicated, it's usually one confidence check or one classification call. The savings are real, and the pipeline degrades gracefully instead of falling over the first time the top-tier model is briefly overloaded.

None of this is about the model getting smarter next quarter. It's about accepting that a language model is one component in a system, not the whole system, and building the parts around it with the same care you'd give any other unreliable network dependency.