What 2023 actually taught us about shipping AI, not demoing it

A year ago, 'AI automation' mostly meant a chatbot widget in the corner of a website. This year, frontier reasoning models got real API access, function calling gave models a reliable way to take structured action, and RAG went from a niche technique to the default architecture for anything that needs a model to answer questions about a business's own information. The technology moved fast. What we learned about actually shipping it moved slower, and it's the part worth writing down before the year ends.

The gap between a demo and a system didn't close

If there's one lesson that repeated across nearly every project this year, it's this: a working demo and a production system are different projects, not two stages of the same one. The demo proves the model can do the thing once, with clean input, while someone watches. The system has to do it correctly a thousand times, unattended, against input nobody anticipated, and keep doing it when a supplier changes their invoice template or an email thread gets forwarded twice. Every failure mode we hit this year, layout drift, rate limits, malformed output, prompt injection, lived in that gap. None of it was about the model not being smart enough.

What actually held up all year

• Boring infrastructure around a smart core: retries, logging, dedup, and validation mattered more to reliability than which model version we used.
• Routing between models by task: cheaper models for simple steps, the strongest reasoning model reserved for the step that actually needs it, kept costs sane without sacrificing quality where it counted.
• Retrieval over retraining: for 'make the AI know our business,' pulling relevant documents into context beat fine-tuning almost every time, and it stayed current without a retraining cycle.
• Human approval on anything destructive: the projects that gave an agent real permissions always kept a checkpoint before the irreversible step.

The no-code question, answered honestly

This was also the year no-code AI builders multiplied, promising a chatbot or workflow without writing code. They're genuinely fine for a narrow, low-stakes task: a simple FAQ bot, a basic form router. They get uncomfortable fast the moment you need custom logic, error handling that does more than 'try again,' or an integration the platform didn't anticipate. We watched a few clients start on a no-code tool and hit that ceiling within weeks, not because the platform was bad, but because business logic that matters is rarely as simple as the builder assumed.

The tools got dramatically more capable this year. The discipline required to ship them safely didn't get any smaller.

Going into next year

Nothing about this year suggested the pace slows down. What it did confirm is that the businesses getting real value aren't the ones chasing the newest model release, they're the ones who picked one boring, frequent, rules-based task, automated it properly with error handling and ownership built in, and then moved on to the next one. That approach worked in January and it worked in December. We don't expect that part to change even as the models keep getting better.