What is DSPy?

DSPy is a Python framework that swaps hand-written prompts for declarative code. You declare which fields go into each call and which come out, pick a module, and supply a metric with a handful of example cases. A compiler then generates the prompt text to maximize that metric. Writing the instruction leaves your hands and becomes the output of a measured process.

The line that sums up the project, and the 2023 paper makes it, is that model pipelines are usually implemented with fixed prompt templates discovered by trial and error, and that they can be treated as programs instead¹. The exact mechanics of that generation are the subject of how DSPy compiles prompts; here the goal is the programming model.

Start with what decides whether the tool is for you. Automating optimization only makes sense with a metric and a set of cases in hand, and that holds equally for DSPy and for alternatives such as TextGrad. Anyone arriving at the framework expecting it to produce a good prompt out of nothing is asking the wrong thing of the right tool. The original paper is explicit about that.

The signature

The basic unit in DSPy is the signature: a declaration of which fields go into a call and which come out. In its shortest form it fits in a string, such as question -> answer. In full form you write a class with input and output fields, each carrying a one-line description, plus a docstring saying what the call does.

What separates this from a prompt is what is missing. The signature carries no natural-language instruction about how to answer, defines no output format in prose, and holds no examples. It declares the data contract and stops there.

The compiler fills in the rest. It assembles the instruction text, decides how fields are presented and picks which examples go in, all against the metric you supplied. The product of that assembly is an ordinary string, which travels to the model API exactly like any other prompt. Nothing magical happens in between: what changed is who wrote the string.

The practical gain shows up in maintenance. When you add an output field, you do not rewrite a paragraph and hope the model honours it. You add the field to the declaration and recompile.

The modules

A signature alone does nothing. It is executed by a module, which defines the strategy used to get from input to output. Three modules cover most cases:

Predict makes the direct call. Input goes in, the declared output comes out, with no intermediate step.

ChainOfThought adds a reasoning field before the answer, applying the technique Wei and colleagues described in 2022 when they showed that generating intermediate steps improves performance on arithmetic, commonsense and symbolic reasoning tasks in sufficiently large models². In DSPy that is a one-word change in the code.

ReAct interleaves reasoning with tool calls, following the pattern proposed by Yao and colleagues in 2022, where the model alternates between generating a thought and taking an action that brings new information from the environment³.

The property that matters is composition. Modules are ordinary Python classes, and a DSPy program is a module that calls other modules, with loops, conditionals and everything Python offers. A retrieval pipeline with two search rounds and a final synthesis is normal code, and the compiler sees the whole tree when it optimizes.

Swapping Predict for ChainOfThought is one line, and that is the most direct demonstration of the project’s argument: reasoning strategy becomes a parameter instead of a text rewrite.

Writing your own module follows the same shape as any Python neural network library: you subclass the base class, instantiate submodules in the constructor and write the step in a method. The analogy is deliberate and it helps predict behaviour. As in a network, the compiler walks the submodule tree to discover what is optimizable, and an already-compiled submodule can be frozen so it is not touched again when you optimize the program containing it.

A minimal program

Support ticket triage, classifying the ticket and extracting the product it mentions, fits in a few lines:

import dspy

class Triage(dspy.Signature):
    """Classify a support ticket and extract the product it mentions."""
    ticket: str = dspy.InputField(desc="text written by the customer")
    category: str = dspy.OutputField(desc="billing, bug, question or other")
    product: str = dspy.OutputField(desc="product name, or empty")

triage = dspy.ChainOfThought(Triage)
print(triage(ticket="June invoice came in twice on the Pro plan").category)

What is absent is the point. There is no instruction saying “you are a support assistant”, no prose description of the output format and no example. The docstring becomes the initial instruction, and the compiler may replace it with one that measures better.

Swapping ChainOfThought for Predict drops the reasoning field and makes the call cheaper. Adding a fourth output field is one line. In both cases the prompt reaching the model is reassembled on its own, and that is where the maintenance saving lives: the change stays local instead of being a paragraph rewrite with side effects elsewhere in the text.

Who writes what

Three things stay on your side. The signature, which is the data contract. The modules and how they compose, which is the program structure. And the metric with the set of cases, which is the criterion.

Three things move to the framework. The instruction text, the examples selected from your set, and the format in which fields are presented to the model.

Notice which column concentrates the risk. No optimization repairs a metric that measures the wrong thing, and no example selection repairs a bad decomposition of the task. The framework is very good inside the space you delimited and completely blind to the space you did not.

Where it came from

DSPy did not appear from nowhere. Its direct ancestor is DSP, from 2022, by the same group: a framework for composing a language model and a retrieval model into pipelines that pass text between them, instead of the simple retrieve-then-read arrangement. That work reported relative gains of 37% to 120% over the plain model, 8% to 39% over the standard retrieve-then-read pipeline, and 80% to 290% over a contemporaneous self-ask pipeline⁴.

DSPy, in 2023, generalized the idea and added the missing piece: the compiler. Modules became parameterized, in the sense that they can learn by creating and collecting demonstrations, and the compiler optimizes any pipeline to maximize a given metric¹.

The paper’s numbers hold with their scope attached. Compiled programs beat standard few-shot generally by over 25% with GPT-3.5 and over 65% with llama2-13b-chat, and beat pipelines with expert-written demonstrations by 5% to 46% and 16% to 40% respectively. The authors also report that programs compiled for open and relatively small models were competitive with approaches relying on expert-written prompt chains for GPT-3.5¹. Those are two case studies, with those models, on those tasks.

The follow-up work in 2024 refined the compiler for multi-module programs, separating instruction proposal from demonstration selection and navigating credit assignment across modules without intermediate labels⁵. That is where the optimizer the library has recommended by default since then comes from. One vocabulary trap survives from that era: the 2023 paper calls these components teleprompters, the documentation moved to calling them optimizers, and the dspy.teleprompt import path stayed. It is the same object.

One detail tends to surprise people arriving from the prompt side: the same compiler also optimizes weights. The library ships optimizers that generate fine-tuning data from the program’s own runs, and a 2024 paper by the same group shows that alternating between optimizing weights and optimizing prompts beats doing only one of them, by up to 60% and 6% on average respectively across the models and tasks tested⁶. That is why the documentation talks about prompts and weights rather than prompts alone.

DSPy is not LangChain

The comparison always comes up, and the two solve different problems.

LangChain and similar frameworks are about orchestration: chaining calls, connecting tools, managing memory and talking to different providers through one interface. The prompt text is still written by you, usually as a template with variables.

DSPy composes calls too, but its reason for existing is different: generating the prompt text from a metric. Remove the compiler and what remains is an orchestration framework with one more abstraction and fewer integrations.

In practice they coexist, and the useful question is not which to pick but where your problem is. If it is connecting systems, authenticating and handling network failure, the compiler does not help. If it is that nobody knows which of four prompt versions is better, orchestration does not help.

Where it fails

Without a dataset it does nothing useful. This is the limit that voids all the others. The compiler consumes a metric and cases; without them you get an abstraction layer and no gain in quality.

The generated prompt is opaque. It exists, you can print it and read it, and it tends to be long and odd. Changing a business rule six months later means editing the signature and recompiling, not editing a sentence. Teams used to fixing prompts in production in five minutes feel that trade.

The abstraction leaks at the format. When output must match a very specific shape imposed by an external contract, a declared field does not always produce exactly what you need, and you end up writing prose instructions inside the field description. It works, and it undoes part of the argument.

Compilation ages with the model. The artefact is fitted to the model and version it was measured on. Switching models calls for recompiling, and anyone without the dataset stored and the pipeline reproducible finds that out at the worst moment.

The API surface moves. DSPy is an active project, and optimizer names and method signatures have changed between versions since 2023. Pin the version and read the documentation for the version you pinned, because a two-year-old tutorial frequently will not run.

When it is not worth it

Three situations where the honest answer is to skip it.

One call, short prompt, simple output. The search space is too small to justify the setup, and three measured variants by hand get you to the same place in an afternoon.

A prompt that changes weekly by product decision. If the instruction tracks a moving business rule, the recompile-and-revalidate cycle becomes the bottleneck, and the readability you gave up starts costing every week.

A team with nobody to maintain Python. The tool is code. In a team where the prompt is edited by whoever writes the product rather than whoever writes the backend, adopting it transfers ownership of the prompt to engineering, and that is an organizational decision before it is a technical one.

How to start

1. Pick a task with verifiable output. Classification, field extraction, an answer with an exact value. That is where the metric comes for free.
2. Gather 30 real cases from your logs. With the expected answer. Hold 20% back, unexamined.
3. Write the signature first. Fields and short descriptions, no prose instruction.
4. Run it with Predict, uncompiled. That number is your baseline.
5. Compile and compare on the same set. If the difference is smaller than the noise, the answer is that your task did not need this.
6. Confirm on the holdout. Only then consider changing the module or raising the candidate count.

What it costs

Order of magnitude. A typical compilation scores dozens of candidates against your set, so the cost is roughly the size of the set times the number of candidates times the cost of one call. A 100-case set with 30 candidates lands in the millions of input tokens, which at July 2026 prices means tens of dollars per session.

The forgotten cost is production. A compiled prompt is almost always larger than a hand-written one, because examples help and the compiler will include them. If it doubles in size and your application makes a million calls a month, that difference outweighs every compilation combined. Measure the size of the winning prompt before accepting the accuracy gain.

Footnotes

1. Khattab et al. (2023) describe the programming model and the compiler, with compiled programs beating standard few-shot generally by over 25% with GPT-3.5 and over 65% with llama2-13b-chat across two case studies. ↩ ↩² ↩³

2. Wei et al. (2022) showed that generating intermediate steps improves performance on arithmetic, commonsense and symbolic reasoning from a certain model scale onward. ↩

3. Yao et al. (2022) proposed interleaving thought generation with action in the environment, which is the pattern the ReAct module implements. ↩

4. Khattab et al. (2022) proposed DSP, the direct ancestor, with relative gains of 37% to 120% over the plain model on open-domain and multi-hop question answering. ↩

5. Opsahl-Ong et al. (2024) separated instruction proposal from demonstration selection in multi-module programs, without intermediate labels. ↩

6. Soylu et al. (2024) alternated weight and prompt optimization in the same pipeline, beating weight-only optimization by up to 60% and prompt-only by up to 6%, on average across models and tasks. ↩