What is chain-of-thought?

Chain-of-thought means asking the model to write the intermediate steps before the final answer. Instead of returning just the number, it returns the working. The technique raises accuracy on multi-stage problems, because every step it writes becomes part of the input to the next one. It costs output tokens and it does not help on every task.

The name arrived in January 2022, in a Google paper that measured the effect with eight worked examples in the prompt and no training at all¹. It is the simplest member of the LLM reasoning family, and the only one you apply by changing a sentence.

There are two ways to ask. You paste worked examples with their steps into the prompt, which is few-shot CoT, or you add a trigger phrase and no examples at all, which is zero-shot CoT. Both produce steps. What changes is who writes the shape of them.

Neither one alters anything inside the model. Chain-of-thought is a decision about text in the prompt and a bigger output-token bill.

The steps become context

A model generates one token at a time, and every token it generates joins the input for the ones after it. When it writes “4,800 × 1.15 = 5,520”, that 5,520 exists in the text. The August step then reads a number rather than an intention.

That is the short explanation, and on its own it is not enough. The team behind the paper tested three competing hypotheses and knocked two of them down with ablations worth knowing, because each one maps onto a common misreading of the technique¹.

“The gain comes from producing the equation.” They prompted for the equation alone before the answer, with none of the surrounding text. On GSM8K that barely helped. Those word problems are hard to translate straight into an equation, and the intermediate language is what makes the translation happen. On datasets of one or two steps, where the equation falls out of the wording, the variant did work.

“The gain comes from spending more computation.” They had the model emit a run of dots as long as the equation it would have needed, carrying no content. The result matched answering directly. Empty tokens buy nothing. What buys something is what is written in them.

“The gain comes from activating relevant knowledge.” They moved the chain to after the answer. If the effect were a warm-up, the model would benefit either way. That also matched answering directly. Order matters because the dependency is sequential.

On the theory side, the result underneath all this is from 2023: generating intermediate tokens genuinely widens what a decoder-only transformer can compute, and how much wider depends on how many tokens you let it generate. With a linear number of decoding steps, the class of problems it can solve grows demonstrably against answering immediately². That is the formal version of a plain sentence: depth per token is fixed, number of tokens is not.

The idea is also older than the name. In 2021 a paper called it a scratchpad and showed that emitting intermediate results let a model add long integers and even simulate program execution, tasks it failed when it had to answer in one pass³.

The two ways to ask

With few-shot CoT you write two to eight complete examples in the prompt, each with a question, its steps and its answer. The model copies the shape. This is the original paper’s version and it stays the more controllable one: you decide the level of detail, the vocabulary, and where the final answer lands.

With zero-shot CoT you add a phrase like “let’s think step by step” and give no examples. A May 2022 paper measured this on text-davinci-002: accuracy on MultiArith went from 17.7% to 78.7%, and GSM8K from 10.4% to 40.7%⁴. It costs almost nothing in prompt length, and the price is losing control of the output format.

One detail about zero-shot that nearly every tutorial drops: in the paper it is a two-call procedure. The first call adds the trigger phrase and collects the chain. The second pastes that chain back into the prompt with a second trigger, along the lines of “therefore, the answer in arabic numerals is”, and collects only the result⁴. Without that second call, somebody has to fish the number out of running prose, which is exactly the problem a few-shot exemplar solves for free.

The practical choice is less philosophical than it looks. If you need to parse the answer, few-shot wins, because the exemplar pins the last line and you pay for one call instead of two. If it is a one-off question you will read with your own eyes, zero-shot does the job without inflating the prompt.

The anatomy of an exemplar

A chain-of-thought exemplar has three parts, and the third is the one people forget.

The question establishes the kind of problem. The reasoning shows the route, in short sentences, one operation per sentence. The isolated final answer is a line at the end in a fixed shape, such as “The answer is 5”. It exists so your code can extract the result without parsing the whole chain.

Without that last line you end up running a regex over free text, and free text generated by a model changes shape between calls. The original paper ended every exemplar with “The answer is”, and not for style: it is the output contract.

There is a consequence that shows up in your first week of use. If your exemplar writes the steps in three sentences, the model writes three sentences. If it writes eleven, the model writes eleven and you pay for eleven. Chain length is inherited from the exemplar, not chosen by the model.

What the 2022 paper actually measured

Separating what was measured from what people concluded afterwards clears up most of the confusion around the technique.

What was measured: with PaLM at 540 billion parameters and eight worked examples, the solve rate on GSM8K went from 18% under standard prompting to 57%. The previous state of the art, a 175B GPT-3 fine-tuned with a verifier, sat at 55%, and the fine-tuned GPT-3 without a verifier at 33%¹. Eight examples beating a whole training run is what made the technique spread.

What was also measured, and gets quoted less: the gain depends on scale. Below roughly 100 billion parameters, asking for steps produced fluent wrong chains and accuracy fell below answering directly¹. “CoT always helps” is not in the paper.

Then there is the variance between the people writing the examples. Three authors independently wrote chains for the same eight exemplars. On the coin-flip tracking task, one reached 99.6% and another 71.4%, both above the 50% of standard prompting¹. The technique is robust in direction and fairly sensitive in magnitude.

Where it doesn’t help

This is where the literature changed its mind, and where most of the writing about CoT went stale.

A September 2024 meta-analysis reviewed more than a hundred papers using chain-of-thought and ran its own evaluation across 20 datasets and 14 models. The gain landed on maths and logical or symbolic reasoning. Everywhere else it was small. On MMLU, generating the answer directly gave accuracy nearly identical to CoT, with one revealing exception: when the question or the response contained an equals sign, signalling a symbolic operation⁵.

The same authors split planning from execution and found that much of CoT’s gain comes from improving symbolic execution, and that on that part it loses to an external solver⁵. The practical implication is direct: if your problem has a calculator, an interpreter or a validator available, delegating beats asking for steps.

Where the technique changes nothing in practice: pulling fields out of a document, classifying sentiment, rewriting text, answering about a single fact that is already in the context. In all of those you pay for the chain and get the same result.

Classification has a case worse than a tie. Asking for the justification before the label gives the model several sentences to build an argument, and a built argument tends to get finished. If the first three sentences leaned towards “serious complaint”, the fourth rarely comes out as “praise”. Flipping the order, label first and reasoning after, removes that effect and keeps the explanation for whoever audits later.

Where it breaks

One bad step contaminates the rest. If the July stage comes out wrong, the two after it will be perfectly consistent with it. The final answer arrives looking rigorous with a wrong number inside, which is the worst possible combination for anyone reviewing by sampling.

The chain is not proof of the process. The text you see justifies the answer, but it isn’t necessarily what determined it. That has its own measurements and its own article, in is the chain of thought faithful?. For debugging, the chain helps; for auditing, it isn’t enough.

On a reasoning model, it gets in the way. Models trained to run the process internally already do this work before answering. Instructing “think step by step” on top stacks a second process onto an existing one, and usually costs you both accuracy and money.

The chain leaks to the user. The reasoning contains false starts, hesitation and sometimes your system prompt paraphrased back. Render the raw output in your interface and you are publishing a draft. The last-line contract exists partly for this: you extract the answer and drop the rest.

What it costs

Order-of-magnitude numbers, with the assumptions in plain sight.

A short chain for an arithmetic problem runs 100 to 300 output tokens. The equivalent direct answer is 1 to 10. That puts the output cost of each call somewhere between 20 and 100 times the version without steps, and output tokens are the expensive line on almost every provider’s price sheet.

Latency is the cost the user feels. Generation runs on the order of tens of tokens per second, so a 250-token chain puts several seconds between the question and the line with the answer. In a batch queue that is irrelevant. In a search box it is the difference between a usable product and one that looks frozen.

With few-shot there is an input cost on top: eight exemplars with reasoning add up to a few thousand tokens repeated on every call. That is the one of the three costs prompt caching handles well, because the exemplar block is byte-identical across requests and doesn’t move with the question.

To give the bill a shape: 50,000 questions a month at 8 output tokens each is 400,000 tokens. The same 50,000 with a 250-token chain is 12.5 million. It isn’t the unit price that decides, it’s that factor of thirty — and it doesn’t move when token prices fall, because they fall on both sides.

How to write a chain-of-thought that works

1. Confirm the problem has steps. If you couldn’t write the solution out in stages yourself, neither can the model. A single-fact question has nothing to chain.
2. Start with zero-shot. A trigger phrase costs nothing and tells you in five minutes whether the task responds to the technique. Only build exemplars once you see a gain.
3. Write two to four exemplars, not eight. The gain saturates quickly and every exemplar rides along on every call. Eight was the paper’s number, not a recipe.
4. One operation per sentence. A chain with two calculations on the same line is where arithmetic slips hide, because the intermediate result never gets written down.
5. End every exemplar with the same answer line. That line is what your parser reads. Without it you are betting the model picks the same format twice in a row.
6. Delegate the arithmetic when you can. If an interpreter is available, four lines of code beat the most carefully written chain.
7. Re-run with the numbers swapped. Same problems, different values. If accuracy collapses, what you measured was memorisation.

Step 3 is what saves the most money in production, and step 7 is what most changes how you read a result. Both take under an hour.

Footnotes

1. Wei et al. (2022) measured the effect on arithmetic, commonsense and symbolic reasoning; reported GSM8K rising from 18% to 57% with PaLM 540B and eight exemplars; showed the gain only appears above roughly 100B parameters; and ruled out by ablation that the effect came from the equation, from extra computation, or from knowledge activation. ↩ ↩² ↩³ ↩⁴ ↩⁵

2. Merrill and Sabharwal (2023) proved that generating intermediate tokens widens the class of problems a decoder-only transformer can solve, and that how much wider depends on how many decoding steps are allowed. ↩

3. Nye et al. (2021) called the technique a scratchpad and showed that emitting intermediate results let a model add long integers and simulate program execution, tasks it failed when answering in a single pass. ↩

4. Kojima et al. (2022) measured the effect of a single trigger phrase with no examples: MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% on text-davinci-002. ↩ ↩²

5. Sprague et al. (2024) ran a meta-analysis of over a hundred papers plus their own evaluation on 20 datasets and 14 models: the gain concentrates in maths and symbolic reasoning and vanishes on MMLU when no symbolic operation is involved. ↩ ↩²