Evals Are the Contract, Not the Gate

Here is a meeting I have sat in more than once. A team has built an AI feature. It demos well. Everyone has been using it internally for a couple of weeks and likes it. Then, close to launch, someone senior asks how often it's wrong — and the room discovers it has no answer. Not a bad answer. No answer. What follows is a two-week scramble to build, retroactively, the thing that should have defined "done" in the first place.

The instinct in that moment is to add a gate: nothing ships until it passes an AI quality review. I understand the instinct and I think it's the wrong fix. A late gate doesn't create quality, it just relocates the argument to a point where changing anything is expensive. What you actually want is for the definition of acceptable to exist before a line of code is written.

Why the old model doesn't transfer

Deterministic software has a comfortable property: a test either passes or it doesn't, and if it passes today it passes tomorrow. You can write acceptance criteria in prose because the prose maps cleanly onto assertions.

Model-driven features don't work that way. The output is a distribution, not a value. The same input can produce different results, "correct" is frequently a judgment call rather than a string comparison, and the behavior shifts when anything upstream moves — a model version, a prompt, a retrieval index, a chunking strategy, a tool description. "It worked when I tried it" is a sample size of one against a system whose whole character is variance.

So the acceptance criteria have to change shape. Instead of "the summary should be accurate," you need a set of cases, a way of scoring them, and a stated bar. That set is not a QA artifact produced after the build. It is the specification.

What "contract" actually means here

When I say evals are the contract, I mean something fairly literal. Before the team builds, we agree on three things and write them down.

The cases. What inputs does this feature have to handle, including the ugly ones? This is where product judgment lives. Assembling that set forces the conversation nobody wants to have up front — what are we actually promising, and to whom — and it surfaces disagreements about scope while they're still cheap.

The grading. How do we decide an output was good? Sometimes exact match, more often a rubric applied by a model or a human, occasionally a structural check. The grader is a piece of product design, and writing it makes you say out loud what you actually value.

The bar. What pass rate ships? This number is a product decision, not an engineering one, and it should differ by feature. A drafting assistant a human reviews can ship at a bar that an autonomous remediation action absolutely cannot.

Alongside that, traces — the record of what the system actually did on each case: what it retrieved, which tool it called, what it got back. A pass rate tells you whether you have a problem. A trace tells you where it is. Without traces, an eval failure is a shrug.

What changes when you do this

The most valuable effect isn't quality, it's that acceptance moves left. A team designing against a known bar behaves differently from a team hoping to clear an unknown one later. They pick simpler approaches, they notice retrieval problems in week one instead of week six, and they stop treating the review as an adversarial event.

The second effect is that arguments become empirical. "This prompt is better" and "we should switch models" and "the retrieval is the problem" stop being opinions held by whoever is most senior in the room. Someone runs the suite. This is a bigger cultural shift than it sounds like, and in my experience it's the thing that makes AI teams calm.

The third is that change becomes survivable. Every AI system will need its model swapped, its prompts rewritten, its retrieval reworked. Without an eval suite, each of those is a leap of faith and teams avoid them until forced. With one, they're a Tuesday.

The honest caveats

Eval suites rot. They get written against the failures you knew about at the time and quietly stop representing what users actually do. If production traces aren't feeding new cases back into the set, the suite slowly becomes a monument to last quarter's problems.

Graders have their own error rate, and a model-based grader can be confidently wrong in exactly the correlated way the system under test is. Spot-checking the grader is unglamorous, ongoing work.

And a suite that passes at 95% can still be measuring the wrong thing. High pass rates on a set that omits the cases where being wrong is expensive is worse than no suite at all, because it manufactures confidence. The cases you're most tempted to leave out — ambiguous, adversarial, low-frequency, high-consequence — are the ones carrying most of the risk.

None of that is an argument against the practice. It's an argument that the eval set is a living product surface with an owner, not a checkbox. The bar isn't perfection. It's legibility: at any moment, someone should be able to answer "how often is this wrong, in what way, and is that acceptable?" without scheduling a meeting to find out.