What is zero-shot prompting?

Zero-shot prompting is asking for the task with no worked example attached: just the instruction and the input. Everything you don’t write, the model infers — the shape of the answer, which labels exist, how granular to be. It’s the right starting point for almost any task: it’s the shortest prompt, the cheapest one, and the only one that doesn’t tilt the output toward one particular case.

The name comes from “shot”, which here means example, not attempt. Zero-shot isn’t asking once and not pushing; it’s supplying no solved case. It’s the first of the example-based prompting modes, and the others are defined against it: one example and several. The comparison with numbers attached lives in zero-shot, one-shot or few-shot.

It’s worth saying what zero-shot doesn’t mean. It doesn’t mean a short prompt or a lazy one. The instruction in a well-written zero-shot prompt is often longer than in a few-shot one, because everything an example would have conveyed for free has to be spelled out in words.

What the model has to infer on its own

Take a concrete task: classifying the severity of a monitoring alert.

Classify the severity of this alert.

Alert: “checkout p99 latency at 3.4 s for 8 minutes”

The model answers. The answer is plausible and, for your system, wrong: “High — significant performance degradation in the purchase flow.” You wanted P2.

Three decisions left your hands in the same call.

The format: one word, one sentence, JSON? You didn’t say, so the model picked what usually pleases, which is a sentence with a justification.

The label space: high, medium, low? critical, warning, info? P1, P2, P3? Every company has its own, and none of them is more obvious than the others from where the model is sitting.

The granularity: the label alone, or the label plus the reasoning? When in doubt it explains, because explaining looks more helpful.

Writing a zero-shot prompt is precisely the work of turning those three inferences into text. That isn’t a limitation of the technique; it is the technique.

There’s a fourth thing the model infers and almost nobody writes down: where your instruction ends and the data begins. If the alert text happens to contain the word “classify”, or a line break in an awkward place, that boundary dissolves. Marking it with an explicit delimiter, a ### Alert line or a tag, costs five tokens and removes an entire class of failure that only shows up on day 40, on the weird input.

Why zero-shot got good

In 2020, zero-shot was the weak baseline. The GPT-3 paper measured the same task across all three modes and the result depended heavily on the dataset: on TriviaQA, the 175B model scored 64.3% with no examples and 71.2% with 64 of them; on Natural Questions it went from 14.6% to 29.9%¹. Seven points in one case, double the score in the other, same model.

What changed afterwards wasn’t size, it was training. A 2021 paper tuned a 137B model on more than 60 NLP tasks phrased as natural-language instructions and showed that this substantially improves performance on unseen task types: the resulting model beat zero-shot 175B GPT-3 on 20 of the 25 tasks evaluated². That same year another group converted a large set of supervised datasets into readable prompted form and trained an encoder-decoder on the mixture; it reached strong zero-shot performance, often outperforming models up to 16 times its size³.

The practical consequence is direct. The models you call today were explicitly trained to obey an instruction with no example attached. Zero-shot stopped being the degraded mode and became the normal one — and a fair share of the 2021 advice about “always give examples” aged along with it.

Where it’s enough and where it predictably fails

Zero-shot is enough when the output has no mandatory shape. Summarising, rewriting, translating, answering a question about a text that’s already in the prompt: in all of those there’s a wide band of acceptable answers, and the model lands inside it unaided.

It fails in a predictable way in three situations, and they share a root.

When the exact format matters. You need bare JSON, no code fence around it, with a numeric value carrying no currency symbol and no thousands separator. You can describe that in words, but it takes a paragraph and stays ambiguous anyway.

When the label vocabulary is yours. P1, P2 and P3 live in your runbook. A category spelled billing_dispute exists because a column in your table is spelled that way. None of it is deducible.

When the boundary between two categories is your convention. Is a latency alert lasting 8 minutes a P2 or a P1? The answer depends on your SLA, not on the world. This is where zero-shot loses the most and examples win the most, because the boundary is exactly what an example shows and what a textual definition rarely manages to pin down.

How to write a zero-shot prompt that works

The reflex when zero-shot fails is to add an example. Before that, it’s worth ten minutes writing down what was implicit.

1. Enumerate the answer space. “Answer with P1, P2 or P3 only.” One sentence removes the entire label problem, and costs fewer tokens than any example.
2. Describe the format with the precision you’d demand of a schema. If the output feeds code, state the number of fields, the type of each and that nothing may appear before or after.
3. Define the boundary with an operational criterion. “P1 if the purchase flow is unavailable; P2 if it is degraded” pins the line far better than “classify by how serious it is”.
4. Say what to do with an input that doesn’t fit. Without that, the model forces the input into some category instead of saying it doesn’t know. Abstention has to be written down.
5. Cut the decorative persona. “You are a senior SRE with 15 years of experience” burns context and defines nothing checkable. What moves the result is the constraint, not the badge.

Step 3 pays the most and is the one people skip. Most classification prompts list the categories and define none of them.

Zero-shot for classification

Classification is where the question “is zero-shot enough?” comes up most, so it’s worth being specific.

It’s enough when the categories exist in the world: positive or negative sentiment, the language a text is written in, spam or not spam. The model saw millions of instances of those distinctions in training and doesn’t need yours.

It isn’t enough when the categories are internal. P1 and P2 aren’t concepts, they’re conventions written in a document the model never read. The same goes for any taxonomy of product, department or cancellation reason that somebody drew up in a meeting.

There’s also an effect that interferes quietly: the model doesn’t treat the labels as equally likely before it sees the input. A 2021 paper measured this bias in the few-shot setting and found a systematic preference for answers common in pretraining and for labels appearing near the end of the prompt; calibrating the output against a content-free input recovered up to 30 absolute points of average accuracy⁴. The experiment used examples, so don’t carry the number over to zero-shot, but the cause is the same and it’s still there: critical and high are not equally frequent words, and the difference shows up in the distribution before any reasoning about the alert happens.

In practice that has two exits. Pick label names that are neutral and comparably frequent, or ask the model for a natural label and map it to your internal code in your own code, where a switch settles it without ambiguity.

Zero-shot with reasoning

One variant deserves its own note. In 2022, a paper showed that appending “let’s think step by step” before the answer, with no examples at all, moves reasoning performance substantially: on text-davinci-002, MultiArith went from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%⁵.

Two caveats before you paste that phrase everywhere. The first is the date: that is a 2022 model, and today’s reasoning models produce the intermediate steps on their own, so imposing a thinking format can hurt rather than help. The second is scope: the gain was measured on tasks with a verifiable correct answer, arithmetic and symbolic reasoning. On classification or extraction, asking for step-by-step usually just buys you a longer answer. The subject has its own cluster in reasoning in LLMs.

Where it breaks

The output drifts between calls. With no example pinning the format, two identical requests can return P2 and Severity: P2. If a parser sits on the other end, one of the two breaks it.

The model doesn’t ask for clarification. An ambiguous input produces a confident answer, not a question. Nothing in zero-shot signals uncertainty unless you ask for it.

Wording matters more than we’d like. A 2023 study measured meaning-preserving format variations — separator, spacing, casing — and found up to 76 accuracy points of spread on LLaMA-2-13B. The experiment ran in a few-shot setting, so don’t carry the number over to zero-shot; what carries over is the habit of treating format as a variable and measuring it⁶.

It doesn’t substitute for missing knowledge. If the answer depends on your runbook, no instruction makes the model know the runbook. Either that text goes into the prompt or there’s no task.

Zero-shot learning is a different thing. The name collision wrecks entire searches. Zero-shot learning is a machine learning line about recognising classes never seen in training from a description or attributes, and training is involved. Zero-shot prompting updates no weights at all.

What it costs

Order of magnitude, using the severity task as the base.

The full zero-shot instruction described above — label enumeration, boundary criterion, format and abstention rule — lands around 120 tokens. Five worked alert-and-label examples add something like 250 more. The per-call arithmetic is irrelevant; the monthly arithmetic isn’t.

A system classifying 1 million alerts a month pays 120 million input tokens with zero-shot and 370 million with five-shot. If few-shot removes 4 percentage points of error that cost somebody a 3 a.m. page, it pays for itself easily. If it removes 0.3 of a point, it doesn’t. The question is identical at any volume, and it only has an answer with measurement.

When to escalate

1. Write the zero-shot version with the five items from the section above. No examples.
2. Run it on 30 real inputs with the expected output written next to each. Count the hits. That number is your baseline.
3. Look at the errors, not at the rate. A format error asks for one example. A boundary error asks for examples on both sides of the boundary. A knowledge error asks for no example at all: it asks for the document in the context.
4. Escalate one step at a time and measure again. If one example fixed it, stop at one. The natural path is zero-shot, one example, few-shot, and every step costs context on every call.

Step 3 is what prevents the unnecessary purchase. Adding five examples because zero-shot “didn’t feel reliable” is the most common way to pay 250 tokens per request, forever, without knowing what you bought.

Footnotes

1. Brown et al. (2020) report, in the closed-book question answering table, 64.3% zero-shot and 71.2% few-shot on TriviaQA, and 14.6% against 29.9% on Natural Questions, with the 175B model and 64 examples. ↩

2. Wei et al. (2021) tuned a 137B model on more than 60 tasks phrased as instructions and measured gains on unseen task types, beating zero-shot 175B GPT-3 on 20 of 25 tasks. ↩

3. Sanh et al. (2021) converted supervised datasets into prompted form and trained an encoder-decoder on the mixture, reaching zero-shot performance above models up to 16 times larger on several datasets. ↩

4. Zhao et al. (2021) describe the model’s bias toward answers common in pretraining and toward labels near the end of the prompt, and recover up to 30 absolute points of average accuracy by calibrating with a content-free input. ↩

5. Kojima et al. (2022) measured the effect of “let’s think step by step” with no examples at all on arithmetic and symbolic reasoning benchmarks. ↩

6. Sclar et al. (2023) measured meaning-preserving format variations in a few-shot setting and found up to 76 points of accuracy difference on LLaMA-2-13B. ↩