What is context engineering?

Context engineering is deciding what fills the model’s context window on every call: system instructions, tool definitions, session history, retrieved documents and the question itself. Prompt engineering covers one of those parts, the text you write. Context engineering covers the whole set and how the space gets divided between the parts.

The model doesn’t see the division. It receives a sequence of tokens and answers. What decides the contents of that sequence is your code.

The five slices

The line between “system instruction” and “tool result” lives in your code and in the API’s delimiters, not in the model’s attention. What exists is a token ceiling and five kinds of content competing for the room beneath it.

System instructions. Role, rules, output format, refusal policy, criteria for when to reach for each tool. In production this is rarely short: an agent system prompt clears 2,000 tokens on rules alone. It’s the slice that stays stable across calls, which is why it’s the one caching pays off on.

Tool definitions. Each tool arrives as a schema with a name, a description and parameters. A well-described tool runs a few hundred tokens; fifteen of them run a few thousand, on every call, including the iterations where the model calls none of them.

History. Everything said so far in the session, including what the model answered and what the tools returned. It grows on every iteration and never shrinks on its own. It’s the slice nobody budgets for and the one that blows the ceiling first.

Documents and results. Passages pulled by search, files read, command output, API responses. A large git diff, a failing test log, the HTML of a page: any of those clears 10,000 tokens comfortably.

The question. What the user just typed. It’s almost always the smallest slice in the window, and the only one most teams review carefully.

The sum has a ceiling. Commercial models as of July 2026 advertise windows between 200 thousand and 1 million tokens. The advertised limit is the easy half of the problem.

Look at how the slices behave over a session. Two are fixed, and you wrote both. One grows on its own with every iteration. One varies with whatever the model decides to fetch. The last one is tiny. Context engineering happens almost entirely in the middle two, which are exactly the ones nobody wrote.

Why the term showed up in 2025

The name is recent. It circulated in June 2025 in posts by Tobi Lütke and Andrej Karpathy, and got a systematic treatment when Anthropic published an engineering guide on it that September¹. The practice is older than the label. What changed was the proportion.

In a single chat call, what you write is practically the whole context. You control what the model reads, and improving the answer means improving that text. That’s prompt engineering, and the entire discipline fits inside it.

In an agent running twenty iterations with fifteen tools, what you write is the system prompt. Everything else lands at runtime: the model decides to call a search, the search returns 8,000 tokens, that goes back into the window, the model reads it and decides to call something else. By iteration ten, most of the window is content no person wrote or reviewed.

The useful question stops being “how do I word this instruction” and becomes “what belongs in the window right now, and what no longer does”. Both disciplines still exist and solve different problems, which earns a comparison of its own in context engineering or prompt engineering.

Long context degrades before the limit

Filling the window because it’s big isn’t a strategy. Three results explain why.

A 2023 study measured the effect of position. On multi-document question answering and on key-value retrieval, accuracy is highest when the relevant information sits at the start or the end of the input, and drops when it sits in the middle. The effect shows up even in models sold as long-context². This is the phenomenon that became known as lost in the middle.

In 2024, the RULER benchmark tested 17 models across 13 long-context tasks, going past needle-in-a-haystack to include multi-hop tracing and aggregation. All 17 advertised a window of 32 thousand tokens or more. Half of them failed to hold acceptable performance at 32 thousand³.

In 2025, NoLiMa went after a bias in the earlier tests: when the question and the target information share wording, the model can solve it by literal matching and the test gets too easy. With minimal lexical overlap, 11 of 13 models advertising at least 128 thousand tokens dropped below half their own short-context score by 32 thousand. GPT-4o, one of the strongest in the group, went from 99.3% to 69.7%⁴.

The reasonable reading is that the advertised window is an allocation limit, not a quality guarantee. The number that matters for you is where your task starts to degrade, and it’s always lower.

The five questions

How much fits. The physical limit of the window, what it costs, and how the model behaves near it. The only one of the five with a number the provider hands you.

What degrades. Where quality falls before the limit, and why. Position, distractor volume and accumulated irrelevant history are different causes with different fixes.

How to split. How much room goes to instructions, tools, history and retrieval. Without an explicit budget the split happens by accident, and whichever slice grows fastest takes everything.

What to drop. Every long session reaches a point where something has to be summarised or discarded. Compaction is the central technique, and the idea isn’t new: a 2023 system already treated the window as paged memory, moving content between the window and external storage the way an operating system does⁵.

In what order. Where each block sits in the window, given that the ends read better than the middle, and how to delimit blocks so the model can tell instruction from data.

Fetch on demand or load up front

The most concrete decision in the discipline, and the one that comes up first when someone is assembling their first agent, is what to put in the window before anything starts.

Loading up front means gathering everything that might be useful and sending it in one go: the database schema, the five policy documents, the config file. It’s predictable, it sits in the stable prefix and it’s cheap to keep there with caching. The price is spending room on what won’t be consulted this run, and that room comes out of what’s left for the actual work.

Fetching on demand means handing the model a read tool and passing only identifiers: the table names, the policy titles, the directory tree. The model asks for what it needs. You spend less window and pay in latency, since each read is another round trip, and you take on the risk that the model asks for the wrong thing or doesn’t ask at all.

The rule of thumb that survives contact with production: preload what gets used on nearly every run, put the rest behind a tool, and keep the index down to a few hundred tokens. A list of twenty file paths costs almost nothing and removes most of the cases where the model would have to guess.

The common mistake is treating this as one decision for the whole agent. It’s usually a decision per content type: the schema goes in every time, the customer’s ticket history goes in on demand, and the refund policy depends on the route. The detail of each strategy lives in fetch on demand or preload.

Where this fails

The part introductory write-ups tend to leave out.

You don’t know what compaction cost you. A history summary looks fine until iteration 30, when the agent redoes a decision it had already made because the reasoning behind it didn’t survive the summary. The failure surfaces late, and it never surfaces in short tests.

The evidence on ordering is thin. That the ends read better than the middle is measured. That putting instructions after documents beats putting them before is a heuristic that varies by model and evaporates when the model family changes. Treat it as something to measure, not as a rule.

Synthetic benchmarks don’t predict your case. A model that nails needle-in-a-haystack at a million tokens can fail at 32 thousand once the task requires aggregating scattered information. That’s exactly the RULER result, and it’s why “supports 1M context” answers nothing about your application.

Sometimes context isn’t the problem. A fair share of context engineering effort gets spent on tasks the model simply can’t do. Before optimising the window, run the task with the ideal context assembled by hand. If it fails there, you’re fixing the wrong thing.

Too many tools cost twice. Fifteen tools burn tokens on every call and raise the odds of the wrong pick. Cutting five rarely-used tools usually buys more than any prompt tweak.

Caching pushes the wrong way. Because the stable prefix bills cheaper when reused, there’s an incentive never to rewrite the start of the window. That preserves the cache, and it preserves whatever junk is already sitting there.

Where to start

1. Measure per category. Log, on every call, how many tokens went to the system prompt, tool definitions, history and documents. Without that you’ll optimise the wrong slice.
2. Attack the biggest one. In nearly every agent it’s history or tool output, and almost never the system prompt, which is where teams look.
3. Cut tools. Drop the ones called less than once per hundred runs and shorten the descriptions of the ones left.
4. Truncate at the source. Tool output should come back trimmed by whoever calls the tool, with an identifier for fetching the rest. A 40,000-token log becomes 2,000 relevant lines and a file path.
5. Compact on a threshold, not on overflow. Pick a point, say 60% of the window, and summarise old history there, keeping the decisions made and the questions still open.
6. Put what matters at the ends. Instructions at the start, current question at the end, bulk in the middle.
7. Redo the measurement whenever you change models. The point where degradation starts moves with the model, and your setup was tuned for the previous one.

The token arithmetic

Context cost in an agent isn’t linear, and that catches nearly everyone out. Every iteration resends everything that came before. If history grows by 5,000 tokens per iteration, the twentieth call processes 100,000 input tokens, and the total across the whole task is the sum of the series: roughly 1 million tokens, not the 100,000 of the last call.

Bringing per-iteration growth down from 5,000 to 1,500 drops that total to around 315,000. Same agent, same model, same task, for under a third of the price, and usually with better accuracy, because the window got cleaner.

Prompt caching changes the stable half of that arithmetic. The system prompt and tool definitions get resent on every iteration, and providers charge far less for a token read from cache than for a new one: as of mid-2026, somewhere between a tenth and a quarter of the input price, depending on the provider. Check the current ratio before sizing anything, since it’s the number that moves most.

What caching won’t fix is history, which is new on every iteration by definition. That part only improves by cutting content.

Footnotes

1. Anthropic’s engineering guide, published in September 2025, frames the discipline as curating the set of tokens available during inference and describes four techniques: compaction, structured note-taking outside the window, sub-agents and just-in-time retrieval. ↩

2. Liu et al. (2023) measured performance on multi-document question answering and key-value retrieval while varying the position of the relevant information. The mid-context drop appears even in long-context models. ↩

3. Hsieh et al. (2024) evaluated 17 models across 13 tasks, including multi-hop tracing and aggregation, which go beyond plain retrieval. ↩

4. Modarressi et al. (2025) built a test set with minimal lexical overlap between question and answer, forcing latent association instead of word matching. ↩

5. Packer et al. (2023) proposed managing the window as virtual memory, moving content between the window and external storage under the model’s own control. ↩