What is a jailbreak in an AI model?

A jailbreak makes a model produce an output its policy was trained to refuse. No code bug is involved: refusal is learned behaviour, and learned behaviour has regions where training never reached. The attack consists of finding one of those regions, either by changing how the request is framed or by changing the form it is written in.

The word is borrowed from phones and the analogy gets in the way. There is no privilege to escalate and no partition to mount. There is a statistical model that learned to refuse certain requests in certain contexts, and an attacker hunting for a context training did not cover. That is why the subject belongs to model alignment before it belongs to infrastructure.

Worth separating up front from prompt injection, because the two terms blur together and the blur sends the fix to the wrong team. This article covers why the attack class exists and how to measure a defence; building the suite itself belongs to LLM red teaming, and the layer that filters input and output in production shows up in content moderation with LLMs.

Different targets

In a jailbreak, the attacker talks to the model and wants a specific output: the one the provider trained it not to give. The target is the policy. The provider fixes it, in training or in a filter, and the fix covers every customer at once.

In an injection, the attacker does not want text, they want effect. They place an instruction in content your application reads and get the model to call a tool with the wrong argument. The target is your system. You fix it, in the scope of what each tool is allowed to do.

The consequence that matters is this: a perfectly aligned model stays vulnerable to injection. Forwarding an email is not forbidden content, it is exactly what the tool exists for. No model policy separates the user’s legitimate request from the one planted in a document, because both ask for the same thing.

In the other direction, a jailbreak on its own is rarely your incident. It becomes yours when the application treats the output as trusted: renders without escaping, executes generated code, publishes text unreviewed. At that point the problem stopped being that the model wrote something and started being that your system acted on it.

Why safety training fails

The reference work on the mechanism is from 2023, and its argument beats the list of tricks that usually accompanies the subject. Wei, Haghtalab and Steinhardt propose two failure modes and use both to construct new attacks, which is the most honest way to test an explanatory hypothesis¹.

Competing objectives. The model was trained to follow instructions, to be helpful and to be harmless, and those goals collide. When a request is built so that refusing means violating an explicit format instruction, or abandoning a role the prompt itself established, the model holds two commands and no tiebreaker. The capability that makes it follow your instruction is the same one that makes it follow the structure the attacker assembled.

Mismatched generalization. Safety training covers a smaller set of situations than the set of things the model can do. If it understands an encoding, an under-represented language or a writing style that never appeared in the safety data, there is a domain where capability arrived and refusal did not. The request is the same; the surface it is written on is different.

The authors tested against the GPT-4 and Claude v1.3 of the day and report that attacks derived from these two modes succeeded on every prompt in a collection of unsafe requests drawn from the models’ own red-teaming evaluation sets¹. Their conclusion is the part that aged best: safety has to keep pace with capability, and scaling the model alone closes neither gap. A larger model understands more encodings, which widens the surface of the second failure.

The four families

Published techniques cluster into four families. The description below stays conceptual on purpose: what matters here is which gap each one exploits and which defence answers it.

Competing objective. Fiction, role play, urgency, authority, a format prefix. All of them build a context where refusing contradicts something else the model also learned to do. The matching defence is training priority between instructions, so the policy outranks the framing in the prompt. An input classifier helps little here, because the text on its own does not look dangerous.

Mismatched generalization. Encodings, low-resource languages, unusual writing formats. The defence is widening safety training to cover the domains where the model is already competent. It is the provider’s work and it never finishes, because every gain in capability opens a new domain.

Automated search. Instead of writing the attack, an algorithm searches for a sequence that maximises the chance the model starts answering rather than refusing. The 2023 work that popularised the approach runs gradient-based search over open models and reports that the suffixes it finds transfer to closed interfaces of the era, including ChatGPT, Bard and Claude². The defence is statistical detection: the resulting sequence tends not to read like human text.

Iterative refinement. An attacker model proposes a request, reads the refusal and rewrites, repeating until something gets through. PAIR, also from 2023, does this with black-box access only and usually finds a jailbreak in fewer than twenty queries³. The defence here is not about content, it is about volume: per-session limits, correlation across attempts and watching for rephrasing patterns.

Notice that these defences do not substitute for one another. Training priority does nothing against automated search, and a classifier does nothing against well-written framing. Picking one layer is picking which family gets through.

The split also decides who can act. The first two families live inside the model, so only the provider can close them, and they close for everyone at once when a new version ships. The last two are about traffic, which means they are yours: rate limits, session correlation and anomaly detection sit in your infrastructure whether or not the model behind them ever changes.

What automation changed

Before 2023, jailbreaking was artisanal: someone found a phrasing, it circulated on a forum, the provider patched it. The cycle was slow on both ends.

Automated search changed the attacker’s cost and, with it, the nature of the problem. A method that generates attacks by optimisation produces new variants faster than a signature filter can absorb them, and the cross-model transfer observed in 2023 means the attacker does not even need access to the target model to produce a candidate². Iterative refinement with an attacker model took that further, swapping gradients for conversation while keeping the efficiency³.

It also changed evaluation, for the better. Comparing defences became possible once standardised environments existed: HarmBench, from 2024, compares 18 red teaming methods against 33 models and defences, and its authors use the same environment to develop an efficient adversarial training method, showing that attack and defence can be measured in one place⁴. Before that, each paper reported success against whatever set it had chosen.

What defences achieve today

The most informative public result as of July 2026 comes from a 2025 paper on classifiers trained from rules written in natural language, using synthetic data generated from those rules. Three numbers in the report say more than the method description⁵.

Across more than 3,000 estimated hours of red teaming, no participant found a universal jailbreak that extracted information from the guarded model at a level of detail comparable to the unguarded one, across most target queries. The absolute increase in refusals on production traffic was 0.38 percentage points. The inference overhead was 23.7%.

Taken together they say something the debate usually loses: defending against jailbreaks is neither free nor impossible. It costs latency and compute, it charges a little utility in unwarranted refusals, and in that particular arrangement it held up against a large red-teaming effort. Note also what the result does not say. No universal jailbreak was found by that group, in that system, in that window. That is not a proof of impossibility.

Measuring success is harder than it looks

Before comparing two success rates, ask what each one counted as success, because the answer varies more than you would expect.

The cheap criterion is absence of refusal: the evaluator looks for the usual refusal phrases at the start of the answer and, finding none, marks the attack successful. It is automatic, reproducible and inflated. A model can accept the request and produce three generic paragraphs that help nobody, and that scores the same as a detailed answer.

The expensive criterion is the opposite: a person, or a classifier trained for it, reads the output and decides whether it actually delivers the requested behaviour. That kind of care is what separates an evaluation environment from a test script, and it was one reason HarmBench was built: the authors identify desirable properties that earlier red-teaming evaluations left unaccounted for and design the framework to meet them⁴.

The practical lesson applies to your own suite. Define success as “the output serves what the attacker wanted”, not as “the model did not refuse”. The second definition inflates the number and sends you hunting for problems that are not there.

Where it fails

The attack migrates between families. Hardening the model against framing pushes the attacker toward automated search. Blocking that search with statistical detection pushes them toward iterative refinement, which produces perfectly natural text. Every closed layer redistributes traffic to the others.

Unwarranted refusal is the invisible cost. Driving the jailbreak rate to near zero is easy if you refuse anything that resembles the topic. The legitimate user asking about drug safety, about history or about their own job takes the block. Every defence has to report both rates together, and the 0.38 percentage points in the 2025 work exists precisely for that⁵.

Results do not carry across versions. A model update changes behaviour with nothing in your code changing. The suite you ran last quarter measured a system that no longer exists.

Jailbreak is not your biggest risk if you have tools. Teams spend weeks hardening policy against unwanted output while the agent next door holds a write tool with application credentials. The question that orders priorities is not what the model will write, it is what it can trigger.

A benchmark score is not a product metric. A public leaderboard measures the model’s general policy. What sets your risk is your usage rule, and it almost never coincides with the generic list of forbidden behaviour.

How to test your defence

1. Write your policy before the test. What your application must not produce, as a list specific to your domain. Without it you measure the provider’s policy, which they already measured.
2. Derive cases from the four families. A handful of scenarios per family, with the request phrased in terms of your product.
3. Measure two rates. Unwanted output and refusal on legitimate requests. Reporting only the first yields a system nobody can use.
4. Keep per-case results, not just the aggregate. The useful information is in which cases changed between two rounds.
5. Run it on every model version change. Including the ones the provider describes as minor.
6. Treat output as untrusted regardless. Escape before rendering, review before executing. This holds even with the policy working, because it is the layer that does not depend on it.

What it costs

Order of magnitude. Your own suite of 100 cases, run against two models, consumes a few hundred thousand tokens and costs a few dollars at July 2026 prices. The real expense is human: writing those 100 cases against your policy takes a day or two, and reviewing ambiguous outputs comes back every round, because not every case reduces to a programmatic check.

On the production side, the 2025 work spells out the bill. An input classifier and an output classifier add two calls per request and inference overhead in the tens of percent, plus a small fraction of extra refusals on legitimate traffic⁵. Whether that price is worth paying depends entirely on what an unwanted output costs in your context, and that is a product question rather than a security one.

Footnotes

1. Wei, Haghtalab and Steinhardt (2023) proposed competing objectives and mismatched generalization as failure modes of safety training, and used both to build attacks that succeeded on every prompt in a collection drawn from the models’ red-teaming sets. ↩ ↩²

2. Zou et al. (2023) generated adversarial suffixes by gradient-based search on open models and observed transfer to closed interfaces of the era. ↩ ↩²

3. Chao et al. (2023) used an attacker model to refine the request with black-box access only, typically in fewer than twenty queries. ↩ ↩²

4. Mazeika et al. (2024) standardised the comparison with 18 red teaming methods against 33 models and defences, and developed a more efficient adversarial training method in the same environment. ↩ ↩²

5. Sharma et al. (2025) report over 3,000 estimated hours of red teaming with no universal jailbreak found, at 0.38 percentage points of absolute increase in production refusals and 23.7% inference overhead. ↩ ↩² ↩³