The hundredth run should match the first

A prompt that works on your ten test cases is not a working prompt. It is a prompt that has not been tested. The gap between those two things is where most production disappointment lives.

Here is the pattern. You build something, try it on a handful of examples, get good results, and ship it. A week later someone forwards you an output that is obviously wrong, and when you go looking you find it has been wrong maybe four percent of the time since launch. Nobody noticed, because nobody was looking.

Why ten examples lie to you

You picked them. Not deliberately, but you reached for cases you understood, and cases you understand tend to be the clear ones. The messy input, the one with a missing field, the one written in two languages, the one that is three tickets mashed together: none of those were in your ten, because you would not have chosen them as examples.

The other problem is that the same input does not give the same output twice. Run your prompt five times on one case and you may get four good answers and one that drops a required field. On a sample of ten you will probably never see it. On four thousand you will see it a hundred and sixty times.

So before you scale, do two things: get a harder sample, and run the same case more than once.

Build a set you did not choose

Pull fifty cases at random from real data. Not curated, not cleaned. Random is doing the work here, because random includes the ugly ones and your judgement does not.

Then deliberately add the cases you know are hard. The empty input. The one where the answer is genuinely ambiguous. The one where the correct response is to refuse. Ten of those on top of the fifty random ones.

Label them yourself, once, and keep the labels. This is the boring part and it is the part that pays. Without labels you can look at outputs and feel good about them. With labels you can count.

Make failure visible in the output

The single most effective change is to constrain the shape of what comes back.

If a ticket must be classified into exactly one of five categories, do not accept free text. Constrain it to those five values. Then a malformed answer is not a subtly wrong classification you will never notice. It is a validation error you can count, alert on, and route to a human.

The same logic applies to anything with structure. Ask for fields, require them, and check they arrived. You are not doing this because you need the structure downstream. You are doing it so that failure becomes loud instead of quiet.

Anything you cannot constrain mechanically, spot check on a schedule.

Reduce the variance you can

A few levers, roughly in order of how much they help:

Narrow the task. One prompt doing classification and summarisation and priority scoring will be inconsistent at all three. Three prompts, each doing one thing, are steadier and easier to debug when one drifts.

Constrain the output shape. As above. A closed set of values removes an entire class of variation.

Show the edge cases. Two or three examples covering the awkward inputs teach the boundary better than a paragraph describing it. Include the case where the answer is "none of these".

Say what to do when unsure. Most inconsistency on ambiguous inputs is the model guessing because nothing told it not to. Give it an explicit escape: return "unclear" rather than picking. Then route those to a person. Your accuracy on the ones it does answer goes up, and the ones it skips were the ones you wanted to see anyway.

Watch the right thing afterwards

Accuracy on your test set tells you how it launched. It tells you nothing about Tuesday.

The cheap and surprisingly effective monitor is the distribution of outputs. If category three is 12% of tickets for a month and then jumps to 31%, something changed. Maybe your inputs changed. Maybe a product launch created a new kind of complaint that does not fit your five categories and everything is being forced into the nearest one. Either way you want to know, and you can see it without labelling anything.

Add a small sample of human review on top. Twenty cases a week, labelled properly, is enough to catch a real regression and cheap enough that it actually happens.

Bug or variance

When something comes back wrong, you have two very different problems, and the fix differs.

Run the same input five times. If it fails all five, it is a bug: something about that input or your prompt is broken, and it is reproducible, so go fix it. If it fails one time in five, that is variance, and no prompt edit will remove it entirely. Variance is handled with validation and retries, not with better wording.

People waste a lot of time rewriting prompts to fix what was never a prompt problem. Five runs costs almost nothing and tells you which job you are doing.

Where this is weak

All of this assumes you can label a correct answer. Plenty of tasks have no single right output. Summaries, drafts, anything creative. For those, the distribution monitoring still works, the labelled test set does not, and you fall back on sampled human review with a written rubric. Slower, less satisfying, and still better than looking at nothing.

It also assumes your inputs stay roughly similar over time. When the underlying population shifts hard, a test set built last quarter is measuring a world that no longer exists. Refresh it.

The move

Before you scale anything: pull fifty random real cases, label them, run each three times, and count. Then constrain the output shape so failures are loud, and put a distribution monitor on it before you walk away.