Know whether a change made it better

Someone on your team rewrites a system prompt and says the new one is clearly better. They have three examples where it is. You have no way to disagree that does not sound like an opinion, so it ships. Two weeks later something is worse and nobody can say what changed.

This happens because evaluating these systems is genuinely hard, and the obvious approach fails in a specific way: you try the new version on a few cases, it looks good, and you have measured nothing except which cases you happened to try.

Three examples cannot tell you anything

Two reasons, and they compound.

The examples you reach for while editing a prompt are the ones that motivated the edit. If you changed the wording because it was too verbose on a particular input, of course it is less verbose on that input now. You have confirmed the change did the thing you just did.

And output varies run to run. If a change improves quality on 55% of cases and degrades it on 45%, three examples will show you almost anything. Three good results is entirely consistent with a change that made the system slightly worse.

You need a fixed set of cases and more than one run each. There is no way around this that also works.

What an evaluation needs

A fixed set of cases, chosen before the change. Fixed matters more than large. Fifty cases you keep across every experiment beat five hundred assembled fresh each time, because only the fixed set lets you compare two versions honestly. Sample randomly from real inputs and then add the hard ones on purpose.

A definition of better that two people would apply the same way. This is the hard part and the part that gets skipped. "Higher quality" is not a definition. Write criteria: does it answer the question asked, does it stay under the length limit, does every factual claim appear in the source, does it refuse when it should. Each one gets scored on its own.

If you cannot write criteria two people would score the same, you do not yet know what you are optimising, and no evaluation will rescue that.

Repeats, because of variance. Run each case at least three times and take the average. If a change moves your score by two points and your run to run spread is five points, you have measured nothing. Report the spread alongside the number, always. A result without its variance is not a result.

A rejection condition, written before you look. "We ship if the average improves and no individual criterion regresses by more than five percent." Decide this first. Deciding afterwards means deciding while looking at the answer, and you will find a reading that supports whatever you already wanted.

Who does the scoring

Three options, and you will probably use all of them for different things.

People. Most accurate, slowest, and the only real option when the criterion is genuinely subjective. Expensive enough that you will do it rarely, which means saving it for the decisions that matter.

Code. Best where it applies, which is more often than people assume. Length limits, valid JSON, required fields present, no forbidden terms, every quoted passage actually appearing in the source. These are exact, free, and instant. Push as much as you can into this bucket.

A model as judge. Scales well and is genuinely useful, with real caveats. Give it the same explicit criteria you would give a person, score criteria separately rather than asking for one overall number, and do not let it see which version produced which output. It tends to prefer longer answers and its own style, so check it against human scores on a sample before you trust it. When the judge and your people disagree systematically, believe the people and fix the judge.

The practical split: code for anything checkable, judge for the rest, humans on a sample to keep the judge honest.

The comparison people get wrong

Do not score version A, then change things, then score version B, and compare the numbers. Too much moves in between: the cases, the scorer's mood, the model version underneath you.

Run both versions on the same cases in the same session, ideally interleaved, and compare per case. Then you are asking "did this specific input get better", which is a much sharper question than "did the average go up", and it surfaces the thing averages hide: a change that improves the mean while badly breaking a small category.

Always look at the cases that got worse. Every change has some. If they are all in one category, you have found a real problem that the average concealed.

Where this is weak

Evaluation measures what you thought to measure. A change can improve every criterion on your list and still make the product worse in a way you never wrote down: tone, or how it handles a kind of request that is not in your set, or how it fails when it fails. The set is a proxy and it will drift out of date as the product changes.

There is also a real cost. Building and maintaining a good evaluation is a serious piece of work, and for a small internal tool it is genuinely not worth it. The honest threshold is roughly: when a bad output reaches someone outside your team, you need one. Before that, careful manual checking is a reasonable answer.

The move

Before your next prompt change: freeze fifty cases, write down what better means as separate criteria, decide the rejection condition, then run old and new on the same cases three times each and compare per case. Look hardest at what got worse.