Does 'let's think step by step' actually work?

Zero-shot CoT means getting a reasoning chain out of a model by adding a phrase to the prompt and giving no examples at all. The famous one is “let’s think step by step”. In 2022 it took a model of that generation from 17.7% to 78.7% accuracy on an arithmetic benchmark. The effect was real, it was measured, and it has shrunk considerably since.

The technique is the cheap version of chain-of-thought: instead of writing worked examples, you write a sentence. All it does is change how the model starts its answer.

The alternative is few-shot CoT, the original form of the technique, with eight worked examples inside the prompt¹. It costs input tokens on every call and hands you control of the output shape. There is also a family of triggers that changes the kind of reasoning rather than its presence, such as step-back prompting, which asks the model to state the general principle before touching the specific case.

What the phrase does, and what it doesn’t

It teaches nothing. No weight changes, no procedure is transferred. What it does is more mundane: it fixes how the answer begins, and the beginning constrains everything after it.

A model predicts the likely continuation of the text it was handed. If the text ends in “A:”, the likely continuation is a short answer, because that is how most question-and-answer pairs go. If it ends in “A: let’s think step by step”, the likely continuation is a walkthrough — the model has seen millions of solutions written after a sentence like that, in forums, worked exam papers and textbooks.

The gain doesn’t come from the phrase. It comes from what the phrase prevents: answering immediately. Once the coin’s first state is written down, the second one is computed from text rather than reconstructed from memory.

That also explains why the technique helps on multi-stage problems and does nothing on single-fact questions. There is no intermediate state to write, so delaying the answer buys you nothing.

The original procedure takes two calls

This detail vanishes from almost every tutorial, and it changes the cost arithmetic.

In the paper, zero-shot CoT runs in two steps. The first call builds “Q: [the question]. A: let’s think step by step” and collects the chain. The second pastes the chain back and appends a second trigger, something like “therefore, the answer in arabic numerals is”, and collects only the result². The authors call the second prompt self-augmented, because part of it was written by the model itself.

The split exists for a practical reason. A chain ends in free text, and free text has no fixed slot for the answer. The paper cites outputs like “probably 375 and 376”, where any naive parser picks the wrong number. The second call is what turns that into a field.

If you are going to read the result yourself, skip the second call. If a program reads it, you either pay for that second call or you build a few-shot exemplar that pins the last line. Both cost something; choosing neither costs more, in the form of an intermittent bug.

Not every phrase works

The authors tested 16 phrases on MultiArith, sorted into three categories: instructive, misleading and irrelevant². The result reads two ways, and both matter.

The first reading is that the category rules. Every instructive phrase landed above the 17.7% baseline, and none of the misleading or irrelevant ones did. “By the way, I found a good restaurant nearby” scored 17.5%, effectively tied with writing nothing. “Abrakadabra!” scored 15.5%. There is no magic effect from adding text to a prompt; there is an effect from asking for reasoning.

The second reading is that within the right category the spread is wide. “Let’s think step by step” gave 78.7% and “Let’s think”, the same idea truncated, gave 57.5%. Twenty points between two phrases anyone would call equivalent. No theory predicts that; it is an empirical result on one specific model, and it is the best available evidence that these phrases are not interchangeable.

One line in the table usually goes unnoticed and says a lot about the mechanism. The runner-up was “First,” at 77.3%, a hair behind the winner. It doesn’t mention thinking, doesn’t mention steps, and asks for no reasoning whatsoever. What it does is open an enumeration: after “first,” the natural continuation is “second”, and an enumerated answer is an answer with stages. That supports reading the effect as a constraint on the shape of the opening rather than as the model understanding a request.

A phrase that discourages reasoning drops below the baseline. “Let’s think step by step but reach an incorrect answer” landed at 18.7% and “By using the fact that the earth is round” at 9.3%. A bad instruction is worse than no instruction, which is obvious written down and rarely remembered while stacking sentences into a prompt.

The best phrase wasn’t written by a person

A few months later, a group treated the trigger phrase as a search problem. They generated candidates with a model, filtered for the ones that reproduced correct chains, and kept the best. Out came “Let’s work this out in a step by step way to be sure we have the right answer”, which lifted MultiArith from 78.7% to 82.0% and GSM8K from 40.7% to 43.0% on the same text-davinci-002³.

Three-odd points of gain isn’t the interesting part. The interesting part is that the best known phrase for a task was found by search rather than by intuition, and that it is longer and clumsier than anything a person would write. If the gap between 78.7 and 57.5 were explainable by linguistic good sense, an automated search would have had nothing to find.

The practical lesson survives the specific phrase. If you are going to tune a trigger at all, tune it against your own task with your own scored examples, the way that search did. Arguing about wording without a scoreboard is how teams spend a week moving between two prompts that measure the same.

What changed since 2022

This is where most writing on the subject aged badly, and it’s worth being explicit about what we know and what we don’t.

The numbers above come from text-davinci-002 and its contemporaries. Those models answered directly by default, and the phrase changed that default. The instruction-tuned models that followed already produce steps on maths problems unprompted, because they were trained on data that does. The phrase asks for something the model was going to do anyway.

A 2024 meta-analysis, covering more than a hundred papers plus its own evaluation on 20 datasets and 14 models, found the chain-of-thought gain concentrated in maths and symbolic reasoning, and small everywhere else⁴. That holds for both ways of asking, and it tells you where to look for a gain today: where there is a symbolic operation, not wherever the question feels hard.

On a reasoning model the phrase goes from harmless to harmful. The process already runs before the answer; instructing on top stacks a second one, inflates the token count and improves nothing the model already handled.

One thing we don’t know, and it is worth saying plainly: every published number here comes from English prompts. I found no verified measurement of the same trigger phrases in another language. The mechanism has no obvious reason to care — the model saw worked solutions in many languages — but the magnitude might, and carrying 78.7% over to a translated phrase is extrapolation rather than evidence.

You can check this on your own task in five minutes, and the test beats any generic recommendation. Send twenty of your problems with no trigger and count how many answers already arrive with steps written out. If most do, the trigger is redundant and you are paying tokens for an instruction that changes nothing. If almost none do, the model is answering directly and the phrase has room to work.

Where it breaks

The output has no shape. This is failure number one in production. You get three paragraphs with the number somewhere inside, and your code expects a field. Without the second call or an exemplar, the parser breaks on some rare case and you find out in production.

The model keeps going after it was right. The authors documented this pattern: zero-shot CoT generates steps beyond what was needed, reaches the correct answer, then revises it into a wrong one². Anyone who stops reading at the first correct number never sees it; anyone parsing the last number does.

Sometimes it just restates the question. Instead of reasoning, the model paraphrases the prompt and answers the same way it would have. This shows up more on tasks that have no real steps, and it’s a signal that the technique is the wrong tool there.

It fights your format instruction. If your system prompt asks for JSON and the trigger asks for reasoning, the model has two incompatible orders. You get JSON with the chain stuffed into a text field, or the chain with JSON at the end, and which one varies between calls. The arrangement that works is declaring a reasoning field inside the schema instead of letting the model choose where to put it.

On commonsense tasks it didn’t help. In the original paper, zero-shot CoT produced no gain on commonsense benchmarks even when the generated chains looked reasonable². The technique improves chaining, not knowledge.

What it costs

The trigger phrase costs about 6 input tokens. That isn’t where the cost lives.

The cost lives in the output. A reasoning chain for an arithmetic problem runs 100 to 300 tokens against 1 to 10 for the direct answer. Add the second call and you resend the question plus the entire chain as input again. In practice, zero-shot CoT with answer extraction lands at roughly two to three times what an equivalent few-shot call would cost, when you measure by the invoice rather than by prompt length.

Some numbers to give it shape. A 40-token question that produces a 200-token chain: the first call spends 46 input and 200 output. The second resends 46 plus the 200 of the chain plus the extraction trigger, and returns 5 tokens. Total: about 300 input and 205 output, for a question that without the technique would cost 40 input and 5 output. That’s a factor of forty on the expensive side of the bill.

Against few-shot the shape differs and isn’t always cheaper. Few-shot pays a fixed input block on every call, which prompt caching cuts hard, and finishes in one call. Zero-shot pays two calls and has no stable block to cache. At low volume, zero-shot is simpler. At high volume with parsing, few-shot usually wins.

How to use it today

1. Test without the phrase first. Current models already show their steps on much of what looks like maths. If the steps are already there, you don’t need the trigger.
2. If you use one, use the long version. “Let’s work this out in a step by step way to be sure we have the right answer” carries the same cost as a short phrase and is what the automated search converged on.
3. Don’t stack triggers. Two or three sentences asking for reasoning in the same instruction don’t add up. They compete, and the output gets longer without getting better.
4. If a program reads the output, close the format. Either the second extraction call or an exemplar that pins the last line. Don’t trust a regex over free text.
5. Parse the last number, not the first. The model can carry on past a correct answer, and what counts is what it asserts at the end.
6. Measure on your cases, not on the benchmark. Thirty real questions with the right answers written down will tell you in an afternoon whether the phrase moves anything on your task.

Step 1 saves the most work and gets done the least. A good share of the prompts carrying “think step by step” in 2026 carry it because it was copied from a 2022 post, not because anybody compared the two versions.

Footnotes

1. Wei et al. (2022) established few-shot CoT with eight exemplars and showed the gain only appears above roughly 100 billion parameters. ↩

2. Kojima et al. (2022) measured the effect of a single trigger phrase with no examples, described the two-call procedure with answer extraction, ranked 16 templates on MultiArith, and recorded that no gain appeared on commonsense tasks. ↩ ↩² ↩³ ↩⁴

3. Zhou et al. (2022) treated the trigger phrase as a search problem and found one that beat the best hand-written phrase: MultiArith from 78.7% to 82.0% and GSM8K from 40.7% to 43.0%. ↩

4. Sprague et al. (2024) found, across a meta-analysis and their own evaluation, that chain-of-thought gains concentrate in maths and symbolic reasoning and stay small across other task categories. ↩