What is a model's context window?

A context window is the maximum number of tokens a model processes in a single call. Everything competes for that room: system instructions, tool definitions, conversation history, pasted documents, the question — and the answer it is about to generate. None of it persists after the call. On every request, the model receives the whole sequence again.

That last sentence is what confuses people most at the start. The window is not memory. It is the size of the sheet you hand the model, and the sheet is rewritten from scratch on every question. What looks like memory in a chat is your own code resending the history, and in a long conversation that history is what occupies most of the window.

Because the space is finite and contested, deciding what goes in becomes a problem in its own right, which is what context engineering is about. The unit being counted is the token, and the ceilings each model family actually ships are in windows from 8k to 1M.

Input and output share one ceiling

The model does not see two areas. It receives a sequence of tokens and produces the next one, then the next, each becoming part of the sequence it reads to produce the one after. Generating is continuing the same sequence.

From that comes the most-forgotten practical consequence: the limit applies to the sum. If the window is 128,000 tokens and you reserve 8,000 for the answer, 120,000 remain for everything you send. Asking for a long answer shrinks the room available for the document.

Check the documentation for the model you use, because this has changed shape. In 2026, some providers publish a separate output ceiling — smaller than the rest of the window — and in that case both limits hold at once: the sum cannot exceed the overall ceiling, and the output cannot exceed its own.

The detail that breaks production systems is the maximum-output-tokens parameter. It gets treated as “how much I let the model say” and is really a reservation: that space leaves the budget before the call begins. Leaving the value high just in case reserves room that is almost never used and could have been carrying context.

Why there is a limit

Three independent reasons, all of which still hold with today’s windows.

Attention gets expensive with length. In a transformer, every token attends to every earlier one, which makes the attention matrix grow with the square of sequence length¹. Doubling the context quadruples that work. Implementation techniques improved the memory behaviour of the step enormously, but the amount of computation still grows the same way.

Positions were trained up to a point. The model needs to know the order of tokens, and the most common way to encode it is rotary positions, which rotate the query and key vectors by an angle proportional to position². Past the length seen in training, those angles enter territory the model never saw, and quality drops. A 2023 paper fixed that by compressing position indices rather than extrapolating them, which extended LLaMA-family models to 32,768 tokens in under a thousand tuning steps — and the authors argue plain extrapolation produces catastrophically high attention scores that ruin the mechanism³.

The key-value cache takes memory. To avoid recomputing everything for each generated token, the server stores the keys and values of every token already processed. That cache grows linearly with context and eats VRAM: on an 8B model with grouped-query attention it costs roughly 128 KB per token, which puts 100,000 tokens of context around 13 GB — per request. That problem is what motivated treating the cache as paged memory, on the same idea as operating-system paging⁴.

The three stack. That is why a large window is expensive even when the input token price is low, and why serving long context to many concurrent users is an infrastructure problem rather than a configuration one.

It is worth noting what changed and what did not. Windows grew from 2,000 to a million tokens in a few years because all three constraints were attacked on separate fronts: attention kernels that avoid materialising the full matrix in memory, position schemes that extend without retraining from scratch, grouped-query attention and paged management to hold the cache. None of that removed the underlying constraint. They pushed out the point where it hurts.

What happens on overflow

On the provider’s side the answer is simple and unhelpful: the call fails. You get an error saying the token count exceeds the model’s maximum, and the whole request does not happen. No automatic truncation, no silent dropping of the start of the conversation, no warning on the previous call.

That is the right behaviour, and it is worth understanding why. The provider has no way to know what is disposable in your prompt. Cutting the start would erase the instruction defining the task; cutting the end would erase the question. Any automatic choice would be wrong in some case, and wrong in silence.

On the application’s side there are three answers, in increasing order of quality.

Dropping the oldest message is the most common and the worst. The oldest message is usually the one establishing what is being done. The agent loses the task and keeps working, which produces a kind of error that is hard to diagnose because nothing failed.

Truncating at the source is cheap and handles a good share. Tool results should arrive already cut by whoever calls the tool, with an identifier for fetching the rest. A 40,000-token log becomes two thousand relevant lines and a file path.

Compacting at a threshold is the mature answer. You set a point — something like 60% of the window — and on crossing it you summarise the old history while preserving decisions taken and open items, keeping the detail outside the window to reload on demand. How to do that without losing what matters is in context compaction.

The difference between the three is not sophistication. It is who decides: in the first case, chance decides.

There is a fourth answer worth naming because it is the one teams reach for last and should reach for first: do not put it in the window. A tool that returns a summary and a handle beats a tool that returns the whole payload, and the decision belongs to whoever wrote the tool, not to whoever writes the prompt. Most overflow incidents trace back to a single integration that returns everything it has because returning everything was the easiest thing to write.

How much fits, in real things

Order-of-magnitude numbers, to turn the spec into something you can reason about. All of them vary with language and content type.

A page of running English prose lands around 500 to 750 tokens. A 30-page contract somewhere near 20,000. A 200-page product manual, about 130,000 — already outside a 128,000 window once you add everything else.

Code costs more. A 500-line file usually clears 6,000 tokens, because identifiers, indentation and punctuation become many short tokens. A git diff for a medium-sized feature easily reaches 15,000. A whole package-lock.json clears 100,000 and should never be in the window at all.

JSON API responses are the worst case of all: keys repeated in every object, long identifiers, quotes and commas. A listing of 200 records with fifteen fields each can cost 30,000 tokens to carry information that would fit in a 3,000-token table.

The pattern in those numbers: what overflows the window is almost never text a person wrote. It is machine content going in whole because nobody decided otherwise.

The advertised limit is not the useful limit

This is the part the spec sheet leaves out.

In 2024, a benchmark tested 17 models across 13 long-context tasks, going beyond the needle-in-a-haystack test to include multi-hop tracing and aggregation. All 17 advertised windows of 32,000 tokens or more. Half of them failed to hold acceptable performance at 32,000⁵.

The detail that matters most in that result: nearly all of them scored close to perfectly on the simple retrieval test. The degradation appears when the task requires combining scattered information, which is exactly what a real application asks for. “Supports 1M of context” answers a question about memory allocation and no question about your task.

The reasonable reading is to treat the advertised window as an allocation ceiling and find empirically where your task starts to degrade. That number is always lower, moves by model, and is the only one useful for sizing a system.

Where the window misleads

Context is not memory. Long sessions look like they remember because the history gets resent. Restart the process without persisting that history and the model knows nothing. Memory across sessions is your infrastructure, not the model’s.

Tokens are not words. The conversion varies by language and content. Code, JSON and identifiers produce far more tokens per character than prose does. Estimating the window in pages is a mistake that only surfaces once the system is in production.

The window is not the most frequent cost bottleneck. In an agent, the same history is resent on every iteration. If it grows 5,000 tokens per turn, the twentieth call processes 100,000 input tokens — and the total processed across the whole task is the sum of the series, close to a million. The window ceiling was never hit and the bill arrived anyway.

Prompt caching changes the bill, not the limit. Resending the stable prefix gets cheaper when it is read from cache, but it still occupies the same room in the window. And the cache creates a perverse incentive: because rewriting the start invalidates the cache, there is pressure never to clean up what is already there.

Latency grows before the error does. Long before overflow, a full window shows up as slowness. Time to first token depends on processing the whole input, and it climbs as context grows. A session answering in two seconds at message five and taking twelve at message forty has no bug: it has history. The symptom is performance and the cause is context budget, which sends teams optimising the wrong thing for weeks.

Sizing it in practice

1. Measure per category. Log on every call how many tokens went to system prompt, tool definitions, history and documents. The dominant slice is almost never the one the team assumes.
2. Find your degradation point. Run your real task with context at 10%, 25% and 50% of the window and compare accuracy. Where the curve drops is your useful limit.
3. Budget the output explicitly. Set the answer token ceiling from what the task needs, not from the maximum allowed.
4. Truncate tool results at the source, before they enter the window.
5. Compact at a threshold, not on overflow. Overflowing is already too late: the user lost the session.
6. Put the critical parts at the edges. Instructions at the start, current question at the end, bulk in the middle.
7. Redo the measurement on every model change. The degradation point moves, and your configuration was calibrated for the previous model.

Step 2 is what separates sizing from guessing. It costs an afternoon and replaces the vendor’s spec with a number that holds for your case.

Footnotes

1. Vaswani et al. (2017) defined the attention in which every token attends to every other, whose cost grows with the square of sequence length. ↩

2. Su et al. (2021) proposed encoding position by rotating query and key vectors by an angle proportional to position, the scheme most current open models adopted. ↩

3. Chen et al. (2023) extended LLaMA-family models to 32,768 tokens by linearly compressing position indices, in under a thousand tuning steps, and show that plain extrapolation produces attention scores high enough to wreck the mechanism. ↩

4. Kwon et al. (2023) treated the key-value cache as paged memory to cut fragmentation and waste, which were what limited batch sizes on long-context servers. ↩

5. Hsieh et al. (2024) evaluated 17 models on 13 tasks including multi-hop tracing and aggregation: half of those advertising at least 32,000 tokens failed to hold acceptable performance at that length. ↩