Do LLMs actually reason?

Reasoning in LLMs is the set of techniques that get a model to produce intermediate steps before its final answer instead of answering in one shot. The best known is chain-of-thought, but there are five families, and they differ in what they do with those steps. It’s part of the prompt engineering toolkit, and in some cases it now comes built into the reasoning model itself.

The question in the title has no clean answer, and it’s worth saying that up front. Models solve multi-step problems that weren’t in training, which is more than memorisation. They also fail on trivial variants of problems they get right, which is less than robust reasoning. Picking a side of that debate pays less than understanding where it works and where it breaks.

What can be said with numbers is narrower and more useful. Writing the steps changes the hit rate, the size of that effect depends on the size of the model, and the chain on screen isn’t always what determined the answer. All three have been measured, and all three are below.

Why writing the steps changes the answer

The mechanism is simpler than it looks, and it doesn’t involve the model “thinking harder”.

An LLM generates one token at a time, and every generated token enters the context of the ones that follow. When the model writes “first, 47 × 3 = 141”, that 141 now exists in the context. The rest of the answer is conditioned on it, instead of having to recompute everything inside a single forward pass.

Every token comes out of a forward pass of fixed depth. The model has no way to spend more computation on a hard token than on an easy one, because the architecture has no such dial. What it can spend is more tokens. Writing ten steps is running ten times as many passes over the same problem. The technique doesn’t buy cleverness per step, it buys steps.

An order of 47 items at £3 each, with 12% off and £25 of shipping, shows the difference. The direct answer has to produce “149.08” in one go. With steps it produces 141, then 124.08, then 149.08, and each of those numbers sits somewhere the next one can read it.

The scratchpad becomes working memory. That’s why the technique helps on multi-step problems and does nothing for single-fact questions: there’s no intermediate result to hold onto. And it’s why a wrong scratchpad weighs as much as a right one, since the following step reads both the same way.

The work that popularised this measured one detail that matters: the gain shows up in large models and disappears in small ones¹. Below a certain size, asking for steps produces bad steps, and bad steps give worse answers than the direct one. At the top of the scale the effect was large: with eight exemplars containing steps, a 540-billion-parameter model reached state of the art on GSM8K, the benchmark of maths word problems, ahead of a GPT-3 fine-tuned with a verifier¹. Eight examples in a prompt beating a whole training run is what made the technique stick.

The five families

Linear is the straight chain of steps. Chain-of-thought and its variants. One attempt, one path, and every step comes out as response tokens you pay for and can read. It’s the sensible starting point for any problem whose steps you could write by hand given the time.

Branching explores several paths and picks. Self-consistency samples N independent chains above temperature zero and votes on the most frequent final answer². The original paper reported gains of 17.9% on GSM8K, 11.0% on SVAMP, 12.2% on AQuA, 6.4% on StrategyQA and 3.9% on ARC-challenge. The condition for it to work is having something to count: the answer has to be a number or a label, because in free text two correct chains never match word for word. Tree of thoughts goes further and prunes bad paths mid-way, which helps when the search space is wide and most branches die early.

Iterative produces an answer, critiques it and redoes it. Good for format and completeness. It has an honest limit: the model won’t find a factual error it can’t recognise in the first place. It has a second, less often mentioned: without an external signal the revisions converge fast, and from the third one on it’s adjusting style. With an external signal the family changes level. A compiler, a test suite or a schema validator says it’s wrong without depending on the model noticing, and iterating then becomes the strongest arrangement on this list.

Verified generates checking questions about its own answer and answers them in isolation before revising. The isolation is what makes it work: answering the verification question without looking at the draft stops the model from simply confirming what it already wrote. It costs three calls where there was one, and the third is the one that decides whether it was worth it.

Delegated doesn’t reason at all: it writes code and lets an interpreter run it. For arithmetic and data manipulation this is the most reliable family, because it swaps a weak model capability for a tool that doesn’t get sums wrong. “How many working days between 3 March and 12 September” is the kind of question where the most careful chain loses to four lines of Python. The price is infrastructure: you need a sandbox, and a sandbox is a system rather than a line of configuration.

Prompted chain-of-thought and trained reasoning

Both produce steps, and they differ enough to change what you write in the prompt.

With chain-of-thought you ask, and the steps appear in the response. You read them, audit them, strip what you don’t need and pay for them as output tokens. The control is entirely yours and it’s exercised in text: ask for more detail and you get more steps, ask for brevity and you get fewer.

In a reasoning model the process was trained to happen before the answer, and three things move. Control leaves the text and goes to a parameter, when the API exposes one: you turn effort up and down, not phrasing. The tokens spent in that process are billed as output even when the final answer is three lines long, so asking for a short answer doesn’t make the call cheap. And instructing it to “think step by step” on top usually interferes, because you’re stacking a second process over one that already exists.

There’s an auditing consequence worth recording. As of July 2026, most APIs that expose this reasoning hand back a summary of it, and a summary of a chain is not the chain. For pinpointing which step went wrong, an ordinary chain-of-thought still shows more.

Choosing between them follows the task rather than taste. Problems with a verifiable answer, maths and code among them, are where trained reasoning earns its keep. Extraction and formatting usually gain nothing, and that’s where the bill in the next section hurts most.

Where it breaks

Four failure modes, and recognising which one is at work decides whether the fix is a prompt or an architecture.

Unfaithful reasoning. The chain on display isn’t necessarily what determined the answer. A 2023 study inserted a biasing cue into the prompt, such as reordering the multiple-choice options so the correct one was always “(A)”, and measured accuracy dropping by as much as 36% across 13 tasks from BIG-Bench Hard. The models rarely mentioned the cue in their explanations: the chain justified the new answer as though it had come out of the reasoning³. The models tested were from 2023 and the magnitude has probably moved; the implication hasn’t. The chain helps locate where reasoning went off course, and doesn’t count as evidence of why the answer was what it was.

Error propagation. If step 2 comes out wrong, the next five will be coherent with it. The final answer arrives looking rigorous with an error baked in. It’s the most dangerous failure because the form looks right. It’s also why the branching family works: independently sampled chains rarely botch the same intermediate step the same way, and the vote discards the contaminated path without anyone having to find where it broke.

Overthinking. On a simple problem, asking for reasoning can make it worse. The model builds an elaborate justification for something it would have answered correctly straight away, and intuitive classification is where this shows up most: asking for the justification before the label gives the model room to talk itself into the wrong label. This matters most on reasoning models, where layering chain-of-thought over the internal process tends to interfere.

Out of distribution. Swap the numbers in a problem the model gets right and the accuracy sometimes vanishes. Swap the names, the order of the clauses or the units and the same thing happens. It’s the clearest sign that part of the performance comes from a memorised pattern rather than a generalised procedure, and it’s the test that separates an evaluation that means something from one that confirms what the benchmark already said.

What reasoning costs

Steps are output tokens, and an output token costs a multiple of an input token at essentially every provider. The numbers below are order of magnitude, with the assumptions in the open.

Say you’re classifying 100,000 support tickets into five categories. The direct answer is a label: five output tokens, 500,000 in total. With a short chain of 250 tokens before the label, that’s 25 million. Same task, fifty times the output bill. Turn on self-consistency with ten samples and multiply by ten again: 250 million tokens for the same work.

Latency moves with it. Generation runs in the order of tens of tokens per second, which puts a 250-token chain in the seconds range against a fraction of a second for the direct label. In a batch queue that’s irrelevant. In a search box where someone is waiting, it’s the whole product.

On a reasoning model part of the bill is invisible to you: the tokens spent before the answer are billed as output, and the count isn’t under your direct control. A three-line answer can have cost thousands of tokens.

Where it doesn’t pay is where the direct answer already gets it right: high volume, low value per item, interactive path. Where it pays is the reverse, low volume and a high cost of error, with a verifiable answer. The way to decide is to run both versions over the same fifty cases and compare accuracy against invoice. It takes an afternoon and it replaces the argument.

What to do with this

1. Start without asking for reasoning. Plenty of tasks don’t need it, and steps cost tokens and latency. If the direct version already gets your cases right, you’re done here.
2. If the problem has steps, ask for the steps. On a standard model that’s chain-of-thought. On a reasoning model, don’t: it already does it, and asking on top interferes.
3. If the answer varies between calls, sample several and vote. That’s self-consistency, and it needs a comparable answer, a number or a label. The accuracy gain costs N calls.
4. If arithmetic or dates are involved, delegate to code. Don’t spend effort making the model compute better than an interpreter.
5. Put the label before the justification on classification tasks. Reversing the order takes away the model’s chance to argue its way to the wrong label.
6. Test with the numbers swapped. If the accuracy vanishes, you’re measuring memorisation, not capability.

Step 6 is the most informative and the least practised. It takes five minutes and completely changes how you read any result.

Footnotes

1. Wei et al. (2022) measured the effect on arithmetic, commonsense and symbolic reasoning tasks, showed the gain depends on model scale, and reported state-of-the-art accuracy on GSM8K with a 540B model and eight chain-of-thought exemplars, ahead of a GPT-3 fine-tuned with a verifier. ↩ ↩²

2. Wang et al. (2022) introduced sampling multiple chains and taking a majority vote over the final answer, and reported gains across five arithmetic and commonsense benchmarks. ↩

3. Turpin et al. (2023) demonstrated chains that justify an answer without mentioning the factor that actually determined it, with accuracy dropping by as much as 36% across 13 tasks from BIG-Bench Hard under biasing cues in the prompt. ↩