What is prompt optimization?

Prompt optimization means changing a prompt and checking, against a set of cases with known expected answers, whether the change made things better. The loop has four parts: a metric, a variant, a measurement and a choice. Without the first, the other three decide nothing. It’s the part nearly everyone skips, which is why most prompt improvements you hear about are guesses.

That set of cases has a name: an evaluation set. It’s the prerequisite for everything downstream, from automatic optimizers like DSPy to using a model as judge. None of them let you skip the set. All of them consume it.

Two things worth pulling apart, because they usually arrive together. Writing a good prompt is prompt engineering. Optimizing is a different job: deciding, with evidence, which of two versions ships.

The loop

The metric is defined once and then sits still. It answers the question “what counts as a correct answer here?”, and answering that is harder than it sounds, because it forces you to settle things that were comfortably vague. If the model extracts the right amount in the wrong shape, is that a pass? If it gets the category right but adds three paragraphs, does that count?

The other three parts spin. You produce a prompt variant, run the whole set through it, get a score, compare it against the previous score and keep whichever did better. A 50-case round takes a few minutes, and what comes out is a number that ends an argument.

Note that the variant can come from anywhere. It can be you rewriting the instruction, a script generating ten versions, or another model proposing candidates. As far as the loop cares, it makes no difference: the variant is the cheap part. The measurement is the expensive part, and it’s the one that decides.

Without a metric the loop doesn’t close

Here’s the usual flow on a team without a set. Someone spots bad behaviour, edits the prompt, tries two or three examples in the chat, decides it looks better and ships. The shape is identical. There’s a variant, there’s a test, there’s a decision.

What’s missing is the only part that matters: knowing whether it improved. Three examples picked by the person who just wrote the change are the most biased sample available, because they’re the cases the change was written to fix. Nobody answers the question of what happened to the other ninety-seven.

There’s a quick way to check. Ask the team what the pass rate was before the last prompt edit. If there’s no number, no optimization happened — a guess was swapped for another guess, which sometimes helps and sometimes doesn’t, and nobody finds out which.

What counts as a metric

Pick the cheapest criterion that still separates right from wrong. In order of cost:

Programmatic checks. The JSON parses, the total field matches the expected value, the category is in the allowed list, the generated code passes its tests, the SQL runs. It’s deterministic, it costs almost nothing, and it covers more than most people assume before trying. Whenever the output can be reduced to something checkable, reduce it.

Human labels. Someone reads the output and says whether it’s good. That’s the gold standard and it’s expensive: 100 cases reviewed properly is an afternoon. In practice you spend that once to build the set, and again periodically to audit it, not on every round.

A model as judge. You ask an LLM to compare two answers, or to score one against a written rubric. A 2023 paper measured over 80% agreement between GPT-4 as judge and human preference on MT-Bench, and found that two humans agree with each other at roughly the same rate¹. That’s good enough for plenty of work, and the same paper is blunt about the biases: the judge favours whichever answer it sees first, favours long answers, and favours text from its own model.

One practical detail that saves a lot of debate: the metric doesn’t have to be a score out of ten. One binary per case, passed or didn’t, is steadier, easier to audit, and enough to compare two versions.

How many cases

Twenty cases catch gross errors: the prompt that breaks on empty input, the format that comes out wrong half the time. To choose between two similar versions, twenty won’t do.

The arithmetic is simple. At 50 cases and an 80% pass rate, the 95% confidence interval runs from roughly 69% to 91%. At 200 cases, the same rate gives roughly 74.5% to 85.5%. Which means: on a set of 50, a three-point improvement on its own tells you nothing.

There’s good news. Because you run both prompts over exactly the same cases, the comparison is paired, and what matters isn’t each one’s absolute rate but which cases they disagree on. If the new prompt fixes six cases the old one failed and breaks one the old one passed, that’s far more informative than the gap between 74% and 84% suggests. Store per-case results, not just the aggregate.

What you actually change

The loop doesn’t tell you what to vary. In practice the levers are few, and this is roughly their order of return per unit of effort.

The output shape. Asking for one lowercase word, or JSON with named fields, instead of letting the model choose. It tends to be the highest-impact change per character written, and the easiest to measure, because the criterion turns programmatic in the same move.

The examples. How many, which ones, and where. Example-based prompting settles boundary ambiguity that no instruction settles cleanly, and it’s the lever automatic optimizers reach for first: the compiler described in the 2023 DSPy paper works by creating and collecting demonstrations².

The instruction. Reordering, cutting, making explicit what was implicit. This is where people spend the most time and where returns are least predictable, because the space of variants is far too large for manual search.

The decomposition. Splitting one call into two, classify first and then extract, often returns more than any rewrite, and it changes the set with it, because now there are two places to measure.

The model and its parameters. Not the prompt, but it goes through the same comparison and uses the same set. Measure it early: sometimes the answer is that the prompt is fine and the model is wrong for the job.

One rule that saves rework: one lever per round. Shape and examples changed together give you a number that can’t say which of the two worked, and you’ll carry both forever.

Automatic optimization

An automatic optimizer is a program that occupies the variant and selection boxes of the loop. The idea is always the same: propose candidates, score each one, keep the winner. What differs between approaches is how candidates get proposed.

OPRO, from 2023, uses the model itself as the optimizer: it’s handed the instructions already tried along with the score each one got, and proposes new ones. The authors report prompts beating human-written ones by up to 8% on GSM8K and by up to 50% on Big-Bench Hard tasks³. Those are relative gains over a specific baseline, not a number that travels to your problem.

DSPy treats the whole pipeline as a program and compiles the prompts against a metric you supply. In the 2023 paper, compiled programs beat standard few-shot prompting by generally over 25% with GPT-3.5 and over 65% with llama2-13b-chat, and beat pipelines using expert-written demonstrations by 5% to 46% and 16% to 40% respectively². Follow-up work from 2024 optimizes each module’s instructions and demonstrations separately and reports up to 13 accuracy points over the baselines on five of seven programs tested with Llama-3-8B⁴.

The pattern across all four is the same, and it’s the point of this article: the optimizer consumes a metric and a set. Adopting DSPy without an evaluation set automates nothing — it just relocates the guess into a library.

What sits around it

An evaluation set doesn’t come out of a brainstorm. The good cases come from what already happened in production: the inputs that showed up, the outputs that caused trouble, the tickets people opened. Without observability you invent cases, and invented cases test the system you imagine you have.

At the other end sits cost, which is not a footnote. It caps how big the set can be and how many candidates you can afford to score, and those two multiply. A sensible LLMOps checklist treats all four of these as one system.

Where it fails

You nail the set and miss production. If you tried forty variants against the same fifty cases, the winner won partly by luck: with a few points of noise and forty attempts, the best-by-chance sits above the mean simply for being the luckiest. The fix is mundane and almost never done: hold out a validation set at the start, never look at it while choosing, and confirm the winner on it at the end.

The metric becomes the target. If the criterion is “the judge liked it”, the process will discover that the judge likes long, polite answers, and you’ll optimize verbosity. That verbosity bias is documented in the same 2023 LLM-as-a-judge work¹. Every proxy metric has a version of this; the defence is to read the winning outputs now and then, not only the score.

The set goes stale. The input distribution shifts when the product changes, when a new customer arrives, when support rewrites the intake form. A set built eight months ago measures a problem that may no longer be yours.

Swapping the model invalidates the result. An optimized prompt is an artefact fitted to the model and version it was measured on. The published evidence on how much of that transfers is thin, and the comfortable assumption, that better here means better there, shouldn’t be made without measuring. A model swap is a reason to run everything again.

Tasks with no right answer. Creative text, tone of voice, editorial calls. There the metric is preference, preference is noisier than correctness, and you need more cases and paired comparison to conclude anything at all. Worth doing, with calibrated expectations.

How to start

1. Pull 20 real cases from your logs. Don’t invent them. Take what actually showed up, ugly ones included.
2. Write the acceptance criterion before looking at the outputs. Look first and you’ll write a criterion the current output happens to meet.
3. Score the prompt that’s in production today. That number is your baseline, and simply having it puts you ahead of most teams.
4. Change one thing at a time. Two changes together worth +4 points might be +9 and −5.
5. Only accept a difference bigger than the noise. When in doubt, grow the set before growing your confidence.
6. Hold out a slice you never use to choose. Twenty per cent is enough.
7. Automate afterwards. An automatic optimizer on top of a bad set just optimizes noise faster.

What a round costs

Order of magnitude, to size the thing. A 100-case set, with a 2,000-token prompt and a 500-token input, burns roughly 250,000 input tokens per round plus about 30,000 output. If the judge is another LLM, roughly double it.

An automatic optimization session that scores 30 candidates multiplies that by 30: around 7.5 million input tokens and 900,000 output. General purpose models in July 2026 run between US$1 and US$5 per million input tokens and between US$5 and US$25 per million output, which lands the session somewhere between 10 and 60 dollars. Cheap for a decision that stays in production for months, and the arithmetic shifts a lot if your set has a thousand cases instead of a hundred.

What’s rarely cheap is building the set the first time: labelling 100 cases properly costs an afternoon from someone who knows the domain. That cost, not the token bill, is why the step gets skipped.

Footnotes

1. Zheng et al. (2023) measured agreement between LLM judges and human preference on MT-Bench and Chatbot Arena, and catalogued the position, verbosity and self-preference biases. ↩ ↩²

2. Khattab et al. (2023) showed that compiling a declarative pipeline against a metric beats hand-written few-shot prompting and, in several cases, demonstrations assembled by experts. ↩ ↩²

3. Yang et al. (2023) used the model as the optimizer, feeding it previously scored instructions so it could propose the next batch. ↩

4. Opsahl-Ong et al. (2024) separated instruction optimization from demonstration optimization in multi-module pipelines, without per-module labels. ↩