How do you give the model reasoning examples?

Few-shot CoT means pasting worked examples into the prompt where the answer comes after the steps. The model copies the shape of the reasoning and applies it to the new question. It is the original form of chain-of-thought, with two to eight exemplars and no training at all, and it is still the version that gives you the most control over what comes back.

The difference from an ordinary example is one line. In a few-shot prompt you show input and output; here you show input, route and output. The route is what the model imitates.

It is also the expensive alternative to zero-shot CoT, which gets the same effect from a single phrase and no examples. Choosing between them is choosing between control and prompt size, and this article is about the control side. The mechanism both share is in what chain-of-thought is.

What the exemplar transmits

An exemplar without steps carries two things: the shape of the output and a mapping from input to answer. If the problem is “53 items, 8 per box” and the answer is “7”, the model has to work out on its own where the 7 came from. The simplest hypothesis consistent with that data is division, and division gives 6.

An exemplar with steps carries a third thing: the procedure. “53 divided by 8 is 6, remainder 5. The remainder needs a box. The answer is 7.” Now the rounding rule is written down, and written down is what the model copies.

This isn’t teaching in the training sense. No weight moves. What moves is what sits in the context at generation time, and an explicit rule in the context beats a rule that would have to be inferred from a single input-output pair.

Good news comes with it, and it cuts against what we know about plain few-shot. In prompts of examples without reasoning, exemplar order is a known nuisance variable: shuffle them and the result moves. In the original paper’s arithmetic experiments, few-shot CoT turned out to be robust both to the ordering of the exemplars and to how many there were¹. That is one fewer dial, which is worth mentioning when somebody proposes testing all 24 permutations of four examples.

It matters most on tasks where the correct answer and the plausible answer diverge because of a convention. Rounding up, tie-breaking, business days versus calendar days, proration that starts on the signup date. In those cases an exemplar with steps is the cheapest way to write a business rule into a prompt.

What matters inside the exemplar

Here the evidence cuts against almost everyone’s intuition, and it is worth reading carefully before you rewrite your prompt.

A December 2022 study rewrote the exemplars with deliberately invalid reasoning: steps that do not derive the answer, with the numbers and relations changed. This wasn’t a subtle perturbation; it was a drastic rewrite. The model held on to more than 80% of normal CoT performance on GSM8K and Bamboogle, and more than 90% when the measure was recall of the correct intermediate values². The exemplars were nonsense and the model kept reasoning fine.

What the authors did find decisive were two other properties. Relevance: the steps have to talk about the objects that appear in the question. And order: the steps have to arrive in the sequence in which they depend on each other. Breaking either one costs far more than inventing the arithmetic².

A September 2022 paper reached a nearby conclusion by another route. Using counterfactual prompts, where one component at a time was altered, the authors concluded that the presence of factually correct patterns in the prompt is practically immaterial to CoT’s success, and that the role of the intermediate steps isn’t to teach the task: it is to signal which symbols should be replicated in the output³.

One caveat before anyone takes the wrong lesson from that. Those experiments were run on GSM8K and a multi-hop question-answering set with 2022-era models, and the authors were isolating a mechanism, not recommending a practice. Nothing here says you should ship exemplars with wrong arithmetic. What it says is where your review effort belongs: an exemplar whose steps are correct but whose objects are unrelated to your real questions is worse than one with a typo in a subtotal.

The practical reading of both results: the exemplar is a mould, not a lesson. It says “an answer to this kind of question looks like this, and passes through these objects, in this order”. Spending an afternoon checking the arithmetic in your exemplars pays less than spending ten minutes making sure they talk about the same objects your real questions do.

How many exemplars, and from which task

The task has to match. This is the most expensive mistake and the easiest to make. The zero-shot CoT paper has a table showing the size of the damage: swapping arithmetic exemplars for commonsense ones dropped MultiArith from 88.2% to 27.0%. On AQuA, where the answer format matches because both tasks are multiple choice, the drop was 39.0% to 31.9%⁴. An exemplar from the wrong task is worse than no exemplar, and plain zero-shot beat both cases.

Longer exemplars pay more. An October 2022 paper proposed picking exemplars by step count: the more complex the demonstrated chain, the better the result. They reported an average gain of 5.3 points and up to 18 across three maths benchmarks and two BIG-Bench Hard tasks⁵. The criterion is easy to apply: between two candidate examples, take the one that needs more stages.

Quantity saturates. Eight was the original paper’s number, chosen for that experiment¹. In production, two to four usually deliver almost everything, and each extra exemplar travels on every call. The test that settles it is running your cases at two, four and eight and looking at the curve.

Writing them by hand isn’t mandatory. An October 2022 method generates them automatically: cluster the questions you already have, pick a representative from each cluster, and use zero-shot CoT to produce each chain. The chains come out with mistakes, and the authors’ defence against that is diversity in the questions picked. Across ten benchmarks with GPT-3, the result matched or beat hand-written exemplars⁶. Worth it when you have many questions and little time.

Who writes the exemplar changes the result

The variance between authors is large, and knowing that changes how you read a prompt A/B test.

In the original paper, three co-authors independently wrote chains for the same eight exemplars. On the coin-flip tracking task, one hit 99.6% and another 71.4%. Both cleared the 50% of standard prompting, but 28 points separate two people doing the same job with the same care¹. The authors also tested chains written by annotators with no machine learning background, taken from GSM8K’s training set, and those beat standard prompting comfortably as well.

The implication is tedious and useful. If you swap an exemplar and accuracy rises three points, you don’t know whether you improved the prompt or landed on a better spot in that variance. Only a test set with dozens of cases tells the two apart.

When it beats zero-shot

A trigger phrase produces steps and costs no prompt. If it works, building exemplars is wasted effort. Three situations where it doesn’t work and few-shot wins.

A program reads the output. The exemplar pins the last line into a format you chose, and the parser reads a field instead of guessing where the number is. Zero-shot only reaches the same place with a second extraction call, which costs another request and another round of latency.

The task has a convention of yours. Rounding rules, deadlines in business days, proration starting on the signup date. None of that is in the model’s training the way your company does it, and writing the rule inside a worked exemplar beats describing it in prose in the instruction.

The reasoning has a specific shape. Diagnosis that has to eliminate hypotheses in an order, tax calculations that apply bases in sequence, contract review that checks clauses by category. A trigger phrase asks for generic steps; an exemplar asks for these steps.

Outside those three, start with zero-shot and don’t look back.

Where it breaks

The exemplar becomes a template. If all your exemplars solve the easy case, the model tends to force the hard case into the same mould. It shows up as an answer with the right structure and a missing stage, which is exactly why picking complex exemplars pays.

Too many steps teach verbosity. An exemplar where every operation becomes two paragraphs produces outputs where every operation becomes two paragraphs. You pay for that on every call and the human reviewer reads more to find the same information.

The block competes with the rest of the context. In an application with retrieved documents or conversation history, 700 tokens of exemplars fight for room and attention with what actually matters. In a long prompt, exemplars sitting in the middle are the part the model is most likely to skate over.

Formatting gets inherited wrong. If your exemplar writes dates as 03/04 and your real input means the fourth of March, the model picks one reading per call. Formatting details inside the exemplar get copied along, including the ones you never noticed were there.

The set goes stale quietly. Exemplars get written once and stay. When the business rule changes, the prompt keeps teaching the old rule with convincing steps, and nobody reviews prompt exemplars at the frequency they review code.

What it costs

Order-of-magnitude numbers, with the assumptions in plain sight.

A plain exemplar with a question and a short answer takes about 30 tokens. The same exemplar with three or four written steps takes something like 160. Four of them go from 120 to 640 tokens, and that rides along on every call next to the instruction and the question: a 175-token prompt becomes a 695-token one.

Four times the input sounds like a lot and is usually the smallest of the costs. Input tokens are the cheap side of the bill at most providers, and this block in particular is byte-identical between requests, which is exactly the condition prompt caching needs. With caching on, the gap between the two prompts shrinks to almost nothing after the first call.

The cost that doesn’t go away is output. The exemplar sets the length of the chain the model will write: a three-sentence exemplar produces three sentences, an eleven-sentence one produces eleven. Writing lean exemplars is the only direct lever you have on output tokens, and it is what decides the bill at volume.

How to build a set of exemplars

1. Take real questions, not invented ones. Two or three from your own backlog of cases. An invented example tends to be too clean and misses the mess of real input.
2. Prefer the case with more stages. Between two candidates, the more complex one teaches more, which is what complexity-based selection measured.
3. One operation per sentence. That is what lets the next step read a number already written down, and it is where arithmetic slips appear when you don’t.
4. Write the business rule inside the reasoning. “The remainder needs a box” is an exemplar sentence; it costs less and works better than a paragraph of instruction saying the same thing.
5. End every exemplar with the same answer line. That is what your parser reads. If the exemplars disagree here, the output disagrees with them.
6. Check that the exemplars talk about the same objects as the real questions. That is the property the evidence points to as decisive, and it is quick to verify.
7. Start at two, measure, then go to four. Save eight for when four have shown a gain worth extending.

Step 1 changes the result most and gets skipped most. An exemplar lifted from a real case carries, for free, every quirk of your domain you would never have thought to write down.

Footnotes

1. Wei et al. (2022) established few-shot CoT with eight exemplars, measured the variance between three annotators writing the same chains (99.6% against 71.4% on the coin task, against a 50.0% baseline), showed that chains written by annotators with no technical background also beat standard prompting, and found the technique robust to exemplar order and count on arithmetic tasks. ↩ ↩² ↩³

2. Wang et al. (2022) replaced the exemplar steps with invalid reasoning and measured retention of more than 80% of performance on GSM8K and Bamboogle, over 90% on the intrinsic measure, and identified relevance to the query and correct step ordering as the decisive factors. ↩ ↩²

3. Madaan and Yazdanbakhsh (2022) used counterfactual prompts to isolate exemplar components and concluded that factually correct patterns are practically immaterial, and that the role of the steps is to signal which symbols to replicate in the output. ↩

4. Kojima et al. (2022) measured the effect of using exemplars from another task: MultiArith fell from 88.2% to 27.0% and AQuA from 39.0% to 31.9%, with a smaller loss when the answer format matched. ↩

5. Fu et al. (2022) proposed selecting exemplars by step count and reported an average gain of 5.3 points, up to 18, across three maths benchmarks and two BIG-Bench Hard tasks. ↩

6. Zhang et al. (2022) generated exemplars automatically with zero-shot CoT and showed that diversity in the selected questions offsets the errors in the generated chains, matching or beating manual exemplars on ten benchmarks. ↩