Getting an AI feature into production

The first plausible output takes an afternoon. That is why AI features are so easy to start and so hard to finish: the demo arrives before anyone knows whether it is good, and every change after it is judged by eye. One person’s prompt tweak fixes the example in the meeting and quietly breaks three others. The team argues from screenshots.

The feature does not stall because the model is not capable. It stalls because nobody can tell whether a change made it better, and nobody decided what should happen when it gets something wrong. Those are engineering problems, and they are where a forward deployed engineer spends most of the time.

Before any clever prompting, build the thing that scores it: a set of real inputs from your own users and data, with what a good answer looks like for each, and a way to run the whole set and get a number. Start small — a few dozen well-chosen cases beat a few thousand scraped ones — and add every failure you meet in production.

The harness is what turns opinions into measurements. Without it, you cannot compare two approaches, you cannot catch regressions, and you cannot tell a stakeholder honestly whether the feature is ready.

Score the plainest version that could work — often search or retrieval over your own content with no model generating anything, or a single straightforward prompt. That number is the floor every cleverer design has to beat.

Plenty of features never need to go much further. Knowing that in week one is worth more than any architecture diagram: it tells you whether the expensive version is worth building at all.

Treat retrieval, prompting, model choice and routing as separate variables. Change one, run the harness, keep it only if the number moves. A change that does not improve the score does not ship, however good it looked in the meeting.

• —Retrieval — what the model is given to read, and how much of it.
• —Prompting — instructions, examples and output format.
• —Model choice — capability against latency and cost, measured rather than assumed.
• —Routing — sending easy cases down a cheap path and hard ones down a careful one.

Models are slow sometimes, wrong sometimes and unavailable sometimes. A production feature decides in advance what happens in each case:

• —A latency budget, and what the user sees when it is exceeded.
• —A fallback when the model is down — a simpler path, a cached answer, or an honest message.
• —Guardrails on what the feature may say or do, checked in code rather than hoped for in a prompt.
• —A way for people to flag bad answers, feeding straight back into the harness.
• —Cost limits, so a busy day or a loop cannot produce a surprise invoice.

The goal is a feature that degrades into something useful rather than a spinner or a confident wrong answer.

Roll out in stages. Run the feature in the background first and compare its answers with what people actually did. Then release it to a small share of users, watch the scores and the flags, and widen only when the numbers hold. Keep the harness in your CI, so a change that lowers the score fails the build like any other broken test.

Data access and privacy decide more than the model does. Settle early what the feature may read, where it runs, what leaves your infrastructure, and what is logged — and write it down, because your security review will ask.

This work sits across your data, your product and your infrastructure, which is why it rarely fits a single team’s backlog — and why it suits a forward deployed engineer. They work in your systems, build the harness on your real data, and own the feature through to the bad day.

It is the work behind Azenvoc’s AI features that ship practice. If you have a demo that needs to become a product, talk to an engineer.