What is self-consistency in prompting?

Self-consistency means asking the same question several times with sampling switched on, collecting the final answer from each response, and keeping whichever one came back most often. The route each chain took is thrown away; only the result is counted. In the 2022 paper that proposed it, this added 17.9 points of accuracy on GSM8K over ordinary chain-of-thought.

It isn’t a prompting technique. It is a decoding strategy: you swap the greedy pick of the most likely token for a sample of several routes, then decide between them. The prompt stays the same chain-of-thought you already had.

For it to work at all, generation has to vary between calls, which means temperature above zero. At temperature zero the five chains come back identical and the vote has nothing to decide. If what you want is to pick the best of N rather than count the most frequent, that is a different thing, compared in self-consistency or best-of-N.

The whole algorithm fits in four lines

Call the prompt N times at a temperature above zero. Extract the final answer from each response. Count. Return the most frequent one.

No library, no auxiliary model, no training. What is hard lives in step two: reliably extracting the answer from N different pieces of text. If your exemplars always end with the same line, that step is one line of code. If they don’t, that step is the whole project.

Worth noting what the method does not do. It doesn’t look at the reasoning, doesn’t judge quality, doesn’t ask the model to choose. A muddled chain that reaches 18 is worth exactly as much as an immaculate chain that reaches 18. The only information used is the final answer.

Why counting answers fixes anything

The authors’ intuition is in the name: a complex reasoning problem usually admits several valid routes to the same correct answer¹. Valid routes converge; wrong ones scatter.

It works because path errors are random. A chain that flipped a sign at step three produces a wrong number, and the next chain, sampled independently, is unlikely to flip the same sign at the same step. Five different mistakes spread across five values; three correct runs land on one. Counting separates the two groups without needing to know which is which.

Notice the asymmetry, because it is the heart of the thing. There is one right number and there are many wrong ones. Every correct chain falls into the same bucket whatever route it took; each wrong chain tends to get a bucket of its own. Even with a per-chain accuracy below half, the correct answer can still be the most frequent, because it is the only one that accumulates. It is the same reason a five-option multiple-choice question is easier to settle by consensus than an open question.

One consequence follows directly: chains that resemble each other don’t help. If the temperature is too low, the five samples walk nearly the same route, fail together, and the vote becomes an expensive copy of greedy decoding. Diversity is not a side effect of the method, it is the input to it.

And this is why the technique doesn’t touch knowledge errors. If the model believes the rate is 12% when it is 15%, all five chains believe it. The vote returns the same error with five confirmations, which is worse than returning it once, because now it arrives wearing consensus.

What was measured, and under what conditions

The comparison baseline is chain-of-thought with greedy decoding, meaning the exemplars-with-steps prompt from the 2022 paper². Against that, the reported gains were 17.9 points on GSM8K, 11.0 on SVAMP, 12.2 on AQuA, 6.4 on StrategyQA and 3.9 on ARC-challenge¹. The models were the 2022 crop — LaMDA-137B, PaLM-540B, GPT-3 and UL2 — and the sampling used was temperature 0.5 with top-k of 40 on LaMDA, temperature 0.7 with top-k of 40 on PaLM.

The authors also varied the sampling strategy, moving temperature, top-k and top-p, and the method kept delivering across every configuration they tried¹. That matters in practice: you don’t need to hunt for the exact temperature for this to work, you just need it not to be zero.

Four results from the same paper say more about the method than the headline numbers do.

It repairs a broken prompt. The authors deliberately corrupted the exemplars, replacing the numbers inside the reasoning with random values. On LaMDA-137B, GSM8K fell from 17.1% to 14.9% under greedy decoding. With 40 samples and a vote, it rose to 23.4%, above what the correct prompt delivered on its own¹.

It stacks with zero-shot CoT. Applied over the trigger phrase³, with no exemplars at all, GSM8K on PaLM-540B went from 43.0% to 69.2%¹. Anyone who doesn’t want to build exemplars has the simplest working combination right here: one phrase, five calls and a counter.

It survives out-of-distribution input. On the symbolic tasks, the authors set up a case where the exemplars showed two-letter or two-flip problems and the test asked for four. That is the setup where the model can’t copy the exemplar’s pattern, and where most prompting gains evaporate. The vote kept paying there, given a large enough model¹. That is the result that justifies using the technique on real input, which rarely looks like your example.

It beats prompt ensembles. Compared against voting across 40 permutations of exemplar order, or across 40 different prompts, voting across 40 sampled chains won¹. The diversity that counts is diversity of reasoning route, not of prompt wording.

A later paper refined the counting rule: instead of weighing every chain equally, prefer the ones with more steps. Complexity-based selection applied to decoding returned an average gain of 5.3 points, up to 18, on the benchmarks they tested⁴.

The condition that sinks most attempts

The method counts identical answers. That requires “identical” to be an answerable question, and this is where most implementations stall.

Numbers, classification labels, multiple-choice options, extracted fields, booleans: all count cleanly. Two chains that land on 4,968 landed in the same place, and the comparison is ==.

Summaries, prose, explanations, code: don’t count. Two correct answers never match character for character, and the vote degenerates into five buckets of one. That isn’t an implementation limitation, it is what the method is. Free-form output needs a different aggregation rule, at which point it isn’t self-consistency.

Even with countable answers there is a normalisation step nobody mentions. “18”, “18.0”, “$18” and “18 boxes” are the same answer to a person and four different answers to a counter. Cast to a canonical form before counting — parse the number, lowercase the label, strip the currency — or you will get a three-way tie between three spellings of the same correct result.

There is a middle case worth knowing. If your output is free-form but contains one decisive field — the classification inside the assessment, the figure at the end of the analysis — you can vote on the field and generate the prose once, conditioned on the vote’s result. That costs N short calls plus one long one, instead of N long ones.

Agreement as a confidence signal

This is the most useful by-product and the least used.

The authors measured that the share of chains agreeing with the winning answer correlates with the accuracy of that answer¹. Five out of five agreeing is a different case from two out of five, and you already hold that number at no extra cost: it falls out of the counter.

In practice it gives you an escalation gate. High agreement, answer. Low agreement, route to a human, or ask for explicit verification, or return “not sure”. It is the closest thing to a free uncertainty measure available in an LLM system.

The caveat: it isn’t a calibrated probability. Three out of five doesn’t mean a 60% chance of being right. It means “less reliable than five out of five”, and the threshold separating answer from escalate comes from your data, not from the paper. At N of 5 the granularity is coarse too: six possible agreement values, enough to separate consensus from doubt and not enough to rank doubtful cases against each other.

Where it breaks

Systematic error passes and wins votes. This is the central failure, and it is worth restating: the method assumes errors are independent across samples. When the error comes from the model’s training, they aren’t.

The right answer lands in the minority. On hard problems, the correct route can be the least likely one. A majority vote picks the most frequent, which is sometimes the wrong shortcut the model prefers. Raising N doesn’t fix that; it confirms the majority with more confidence.

The parser becomes a single point of failure. A bad extraction from one chain moves one vote. At N of 5, one vote can decide. Measure the parser’s error rate separately from the model’s accuracy, or you will blame the model for a regex problem.

Small models gain little. The measured gain was smaller on the smaller models of the series tested, because the ability to get the sum right on some route has to exist before a vote can find it¹. Voting across five bad chains returns the most popular bad answer.

Latency multiplies unless you parallelise. The N calls are independent, so fire them together. Done in series, the user waits five times longer.

On a reasoning model the bill changes tier. The internal process already spends tokens before the answer, and multiplying that by five multiplies the part of the invoice you can’t see. I found no public measurement comparing external voting against those models’ internal effort, so the honest advice is to measure: run the same task with one high-effort call and with five low-effort ones, and compare accuracy against the bill.

What it costs

The cost is exactly N times, and there is no subtlety here. Five samples cost five calls, five times the output tokens and five times the input tokens, unless prompt caching covers the fixed part.

Accuracy doesn’t grow that way. The authors tested 1, 5, 10, 20 and 40 samples and described the behaviour as saturating quickly; their own recommendation is to start at 5 or 10 to capture most of the gain without incurring too much cost¹. Between 10 and 40 there is still gain, and it is small for four times the price.

Some arithmetic to give it shape. A task with a 250-token output chain, at 10,000 questions a month, spends 2.5 million output tokens. At five samples, 12.5 million. At forty, 100 million. Choosing between 5 and 40 is a budget decision dressed as a technical one.

Where it pays: high-value items, expensive errors, low volume, verifiable answers. Tax calculations, risk triage, contract clause extraction. Where it doesn’t: interactive paths, high volume, cheap items — which is where most applications live.

How to use it

1. Confirm the answer is comparable. A number, a label or a field. If it isn’t, stop here: the method doesn’t apply and adapting it means rewriting it.
2. Turn the temperature up. Between 0.5 and 0.8 is the range from the original experiments. At zero you pay N times for N copies of the same chain.
3. Start at N of 5. Measure. Move to 10 only if the curve on your data is still climbing.
4. Fire the calls in parallel. They are independent. In series you multiply latency by N for no reason.
5. Keep the count, not just the winner. Three out of five and five out of five are different answers as far as your system is concerned, even when the value is the same.
6. Decide what low agreement triggers. Escalate, abstain or verify. If you don’t decide, the method’s most useful signal goes in the bin along with the losing chains.
7. Measure the parser separately from the model. A bad extraction becomes a bad vote, and at small N one vote decides.

Step 6 is what separates teams using self-consistency from teams paying five times over for a slightly better answer.

Footnotes

1. Wang et al. (2022) proposed replacing greedy decoding with sampling several chains and voting on the most frequent final answer; reported +17.9 on GSM8K, +11.0 on SVAMP, +12.2 on AQuA, +6.4 on StrategyQA and +3.9 on ARC-challenge; showed robustness to a corrupted prompt (17.1 to 14.9 under greedy decoding, 23.4 with the vote), a gain over zero-shot CoT (43.0 to 69.2 on PaLM-540B), superiority over prompt ensembles, quick saturation with sample count, and a correlation between agreement and accuracy. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰

2. Wei et al. (2022) established chain-of-thought with exemplars, the prompt self-consistency runs on top of, and showed the gain depends on model scale. ↩

3. Kojima et al. (2022) showed that a single trigger phrase produces a reasoning chain with no exemplars, which is the configuration used in the zero-shot CoT voting experiment. ↩

4. Fu et al. (2022) extended complexity-based selection from picking exemplars to decoding, voting preferentially among the chains with more steps, for an average gain of 5.3 points and up to 18. ↩