How does DSPy optimize a prompt?

Compiling, in DSPy, means searching for the best combination of instruction text and examples for each module of your program. The compiler proposes candidates, runs the whole program over your set of cases, scores each output with your metric and keeps whatever scored best. No weights change: what changes is the text that reaches the model.

The word “compile” is an analogy and it asks for care. A language compiler does a deterministic translation; this one runs a search guided by measurement, and the result depends on the sample. Running it twice can give you two different prompts, and both can be right.

This article is the mechanism inside what DSPy calls compilation. Everything here assumes an evaluation set with an automatic metric, and the newest family of optimizers, of which GEPA is the 2025 example, changes how candidates are proposed without changing that dependency.

The loop

The loop has four steps and none of them is clever.

Propose. The optimizer assembles candidates. A candidate is a pair: the instruction text for a module and the set of demonstrations that accompanies that module. In a three-module program a candidate is one choice per module, and the space grows multiplicatively.

Run. The whole program executes over the cases in the set with that candidate applied.

Score. Your metric function receives the example and the output and returns a number, or a boolean. It is the only value judgement in the process.

Keep. Candidates with better scores survive and feed the next proposal.

The return arrow is what separates this from random sampling. In the optimizers that propose instructions, the model generating the next proposal receives the earlier attempts along with their scores. That idea was formulated in 2023 as using the language model itself as an optimizer: you feed it the solutions already evaluated and their scores, and ask for the next ones¹.

Where the examples come from

The name in the documentation and the tutorials is bootstrapping, and the mechanism explains the name.

The optimizer runs the program over the training cases, with the initial instruction and no demonstrations at all. For each case it stores the execution trace: the input and output of every module on that pass. Then it applies your metric to the final result and discards the traces from cases where it failed.

What remains is a stock of runs that worked. Each one is a ready-made demonstration for any module of the program, with input and output in that module’s exact shape. This is the piece the 2023 paper describes when it says modules are parameterized in the sense of learning by creating and collecting demonstrations².

Notice what that solves. Hand-writing a demonstration for the middle module of a pipeline is tedious, because you have to invent the intermediate input, which does not exist until the program runs. Bootstrapping invents nothing: it harvests what showed up.

And notice what it does not solve. If the initial instruction is too weak, almost no case passes and the trace stock comes out empty. That is the most common failure of a first compilation, and the symptom is a suspiciously fast run whose result matches the baseline.

Where the instruction text comes from

Harvesting examples is the cheap half. Proposing instructions is the hard half, because the space is open-ended.

The 2024 work that set the current standard splits the problem into two parts that can be optimized separately, instructions and demonstrations, and proposes three strategies worth understanding³.

The first is proposing instructions with awareness of the program and the data. Instead of asking a model to “improve this prompt”, the proposer receives the program code, a sample of the dataset and the history of attempts, and writes an instruction from that. An instruction grounded in the task beats a generic rewrite.

The second is minibatch evaluation. Running the whole set on every candidate is expensive, so the algorithm evaluates samples and uses those evaluations to learn a surrogate model of the objective, which estimates which candidates deserve a full evaluation. It is the logic of Bayesian optimization applied to a discrete space of texts.

The third is meta-optimization: refining over time how the model constructs proposals, not just the proposals themselves.

Alongside that comes credit assignment. You supply a metric at the end of the program and no per-module labels, so the algorithm has to decide which stage deserves the change. The authors report up to 13 accuracy points above the baselines on five of seven programs tested with Llama-3-8B³.

The size of the search space

Worth doing the arithmetic once, because it explains why the algorithm looks the way it does.

Take a three-module program. For each module the optimizer holds ten candidate instructions and eight candidate demonstration sets. That is eighty combinations per module and, since the choice is independent across them, 512,000 combinations for the whole program. Scoring all of them on 100 cases would be 51.2 million program runs.

None of them get scored. The algorithm samples a handful of combinations, measures on minibatches, updates its estimate of where spending is worthwhile and concentrates full evaluation on the few promising ones. A typical session scores dozens of combinations, not hundreds of thousands.

That carries a consequence which resets expectations: the result of a compilation is not the optimum, it is a good sample found on a limited budget. Running again with a different seed can return a different winner with a similar score, and that is expected rather than a bug.

The metric has to be cheap

One design decision has an outsized effect on cost, and it is the choice of metric.

The metric runs once per case, per candidate scored. In a session with 30 candidates and 100 cases, it is called thousands of times. If it is a programmatic check, the JSON parses, a field matches, the test passes, its cost is zero and drops out of the arithmetic. If it is a call to a judge model, it doubles the session’s calls and starts dominating the bill.

There is a second effect, less obvious and more expensive. A model judge has variance: the same input and output pair can get different scores on different runs. The optimizer cannot separate variation in the metric from variation in the candidate, so part of the search chases the judge’s noise. With a deterministic metric that problem disappears.

Hence the practical rule: reduce the output to whatever code can verify, and use a judge only for the remainder. A binary metric per case, passed or failed, is usually more stable and cheaper than a zero-to-ten score, and it is enough to rank candidates.

Before and after

Print the final prompt the first time you compile, because comparing it with what you wrote is instructive and occasionally uncomfortable.

A hand-written prompt is usually a prose instruction, with the format described in text and no examples. A compiled one is usually a drier instruction followed by three to eight complete demonstrations, with fields labelled in a rigid, repetitive way.

It is not prettier. Often it reads worse. The only thing it has going for it is having been measured against dozens of alternatives on your dataset, and that is the property you are buying.

On the size of the gain, the public numbers come with scope attached. In the 2023 paper, compiled programs beat standard few-shot generally by over 25% with GPT-3.5 and over 65% with llama2-13b-chat, and beat pipelines with expert-written demonstrations by 5% to 46% and 16% to 40% respectively². Those are two case studies, with those models, on those tasks. What transfers to you is the method, not the percentage.

The optimizer families

They differ in how they propose candidates, and the choice moves the cost a lot.

Demonstration bootstrapping. It only harvests passing traces and assembles example sets. Cheap, no instruction proposal, and it settles more cases than you would expect. Start here.

Instruction proposal with search. The 2024 family: propose an instruction, select demonstrations, score on minibatches and use a surrogate model to decide where to spend full evaluation³. This is the default as of July 2026.

Reflective evolution. Instead of treating the score as a scalar, the optimizer reads execution traces in natural language, diagnoses what went wrong and proposes the fix, keeping a Pareto frontier of its own attempts. GEPA, from 2025, reports coming in over 10% above MIPROv2 and beating GRPO by 6% on average, with up to 35 times fewer rollouts⁴. The rollout saving is the number that moves the arithmetic of when compiling is worth it.

Joint weight and prompt optimization. The compiler also generates fine-tuning data from the runs. A 2024 paper shows that alternating between optimizing weights and optimizing prompts beats doing only one, by up to 60% and 6% on average respectively across the models and tasks tested⁵.

Where it fails

Overfitting to the set. Thirty candidates scored on the same fifty cases produce a partly lucky winner. It is the likeliest failure and the easiest to avoid: hold out a slice before compiling and confirm on it afterwards.

The metric becomes the target. The optimizer will exploit any shortcut your metric rewards. If it compares text by word overlap, the winner will be verbose. Read the winning outputs, not just the score.

Bootstrapping never takes off. A weak initial instruction produces few passing traces, and without traces there are no demonstrations. The fix is improving the initial instruction by hand before compiling, which sounds contradictory and is not: the compiler needs a starting point that works sometimes.

The prompt grows. Demonstrations take context, and the per-request cost in production rises with them. A two-point accuracy gain that doubles the prompt may not survive the arithmetic, and that call belongs to product.

The artefact cannot be reviewed in a pull request. A compiled prompt is a generated blob, and diffing two compilations tells a reviewer nothing useful. What gets reviewed is the dataset, the metric and the final number. Treating the artefact as reviewable code burns the reader’s time and catches no errors.

The result does not transfer. The artefact is fitted to the model and version measured. Store the dataset, the metric and the seed, because recompiling is the operation you will repeat on every model change.

How to run a compilation

1. Measure the baseline uncompiled. The program with a simple instruction and no examples. Without that number, nothing else has anything to compare against.
2. Hold out a slice before anything else. Twenty percent, untouched until the end.
3. Start with bootstrapping. Cheap and often sufficient. Move to instruction proposal only when the gain stalls.
4. Cap the candidate count on the first round. Few candidates on an honest dataset beat many on a small one.
5. Print the winning prompt and read it. It is the only way to notice that the metric rewarded the wrong thing.
6. Measure prompt size alongside accuracy. The two quantities decide together.
7. Version the compiled artefact with the dataset and the model version. Without all three you cannot reproduce or explain the result later.

What it costs

Order of magnitude. The cost of a compilation is roughly the candidate count times the size of the dataset times the cost of one program run. With 30 candidates, 100 cases and a two-module program, that is around 6,000 module executions per session, which at July 2026 prices lands in the tens of dollars. Minibatch evaluation cuts that, because not every candidate reaches full evaluation.

Measure one small session before firing the big one: ten candidates on twenty cases gives you the real cost per run and spares you the surprise of a bill thirty times larger than the estimate.

If the metric uses a model as judge, roughly double the number of calls. And add the cost that never appears in the session: a larger compiled prompt pays the difference on every production request, every month, for as long as it is live.

Footnotes

1. Yang et al. (2023) formulated using the model itself as an optimizer, fed with the solutions already evaluated and their scores. ↩

2. Khattab et al. (2023) describe the compiler that creates and collects demonstrations, with compiled programs beating standard few-shot generally by over 25% with GPT-3.5 and over 65% with llama2-13b-chat. ↩ ↩²

3. Opsahl-Ong et al. (2024) separated instruction proposal from demonstration selection, with stochastic minibatch evaluation and a surrogate model of the objective, reaching 13 points above the baselines on five of seven programs with Llama-3-8B. ↩ ↩² ↩³

4. Agrawal et al. (2025) read execution traces in natural language to diagnose and correct, coming in over 10% above MIPROv2 and 6% above GRPO on average, with up to 35 times fewer rollouts. ↩

5. Soylu et al. (2024) alternated weight and prompt optimization in the same pipeline, beating weight-only optimization by up to 60% and prompt-only by up to 6%, on average across models and tasks. ↩