Notes from building AI that has to work
We write down what we learn on engagements. Mostly the unglamorous parts: measuring whether a system actually works, deciding when not to use a model, and getting things past the pilot stage.
Evaluation sets before prompts
Most teams tune prompts before they can tell whether the output is right. Build the test set first and the prompt work becomes measurable instead of vibes.
You cannot improve what you have not agreed how to score.
Retrieval is a measurement problem, not a model problem
Swapping embedding models rarely fixes bad answers. Knowing which questions fail, and why, almost always does.
The fix was usually in the documents, not the vectors.
When not to use a model
A rule engine that is always right beats a model that is usually right, in more places than the current enthusiasm admits.
If an auditor has to sign it, a model should not decide it.
Evaluation sets before prompts
Most teams reach for prompt engineering first. It feels productive: change the wording, watch the output change, repeat. The problem is that you are steering without instruments. If you cannot say whether today's output is better than yesterday's, you are not improving the system, you are only moving it.
An evaluation set fixes that. It is a collection of real cases with known good answers, and fifty to two hundred is usually enough to start. Pull them from work your team has already done, because those carry the messiness that invented examples lack. Score against them every time you change anything, including the model version.
Two things happen once you have one. Disagreements become empirical, so nobody has to argue about whether the new prompt is better. And you find out which cases fail together, which almost always points at something other than the prompt: a document that was never indexed, a permission boundary, a field that is blank half the time.
The uncomfortable part is that building the set is slow, and it usually reveals the problem was never the model. That is exactly why it is worth doing first.
Retrieval is a measurement problem, not a model problem
When a knowledge assistant gives bad answers, the reflex is to change the embedding model, or the chunk size, or the vector database. Those changes are cheap to make and almost impossible to evaluate, so teams cycle through them for weeks and end up roughly where they started.
The more useful question is which questions fail, and why. Take the questions people actually ask, answer them properly by hand, then look at what the system retrieved. The failures usually cluster into a few dull causes. The document was never ingested. The answer lives in a table that got flattened into noise. Two versions of the policy exist and the older one ranks higher. The question needs three documents combined and retrieval returns the single best match.
None of those are fixed by a better embedding model. They are fixed by ingesting the missing source, parsing tables properly, deleting the stale copy, or retrieving more passages.
Measure retrieval separately from generation. If the right passage was never fetched, no amount of prompting will recover it, and you will spend your time on the wrong half of the system.
When not to use a model
There is a class of work where a language model is simply the wrong instrument. If the rules can be written down, and the same input must always produce the same output, and somebody may have to justify the result to an auditor, then write the rules down and execute them.
Billing calculations. Eligibility checks. Regulatory thresholds. Reconciliation. These are not hard problems in the machine learning sense. They are tedious problems in the specification sense. The real difficulty is getting the people who own the rules to agree on what the rules are, and that difficulty does not disappear when you add a model. It just gets hidden.
A rule engine has properties a model cannot offer. It does not drift when a vendor updates a checkpoint. It costs almost nothing to run. It explains itself by construction: this rule fired, therefore this result. And it fails loudly on cases it was not designed for, which is far safer than answering confidently anyway.
Use models where judgement is genuinely required and being occasionally wrong is acceptable. Everywhere else, be boring on purpose.