LLM Context Window Explained: Long Context vs RAG

Who this is for: developers and technical founders building anything on top of Claude, GPT, or Gemini who keep asking "should I just dump the whole document in the prompt, or build a RAG pipeline?" This article gives you the actual numbers, the failure modes, and a decision framework — not a marketing pitch for either approach.

What an LLM context window actually is

An LLM context window is the maximum amount of text - measured in tokens, not characters or words - that a model can hold in a single request, covering your system prompt, conversation history, any documents you paste in, tool definitions, and the response it's about to generate. Every one of those pieces competes for the same budget. A 200K-token context window sounds enormous until you remember that a single tool definition schema can eat 500-2,000 tokens, and a 40-page PDF can run 25,000-30,000 tokens once you account for tables and formatting overhead.

As of mid-2026 the practical range looks like this:

A token is roughly 3/4 of an English word, and roughly 1-2 Hebrew characters depending on the tokenizer, which matters if you're building for Israeli clients - Hebrew text often costs 30-50% more tokens than the equivalent English for the same content.

Why "just make the context bigger" doesn't solve retrieval

The tempting shortcut is: skip building a retrieval pipeline, paste the entire knowledge base into the prompt, let the model figure it out. This works for small corpora and fails predictably at scale, for three concrete reasons.

1. Needle-in-a-haystack degradation

Anthropic, OpenAI, and independent researchers have all published needle-in-a-haystack benchmarks: hide one fact somewhere in a long document and ask the model to retrieve it. The consistent finding is a "lost in the middle" effect - recall is near-perfect for facts at the very start or very end of the context, and measurably worse for facts buried in the middle third, even well within the model's stated limit. Newer models have narrowed this gap significantly (Claude and Gemini 3 both handle single-needle retrieval at 90%+ across most of a 200K window now), but multi-needle and reasoning-over-scattered-facts tasks still degrade well before you hit the token ceiling. If your task is "find this one fact," long context works. If it's "synthesize five facts scattered across 80 pages," it's still shakier than a model working over 5 well-chosen paragraphs.

2. Cost scales linearly, latency scales worse

Every token in your context gets re-processed (or pulled from cache) on every call. A 150K-token context at Claude Opus pricing (roughly $15 per million input tokens as of writing) costs about $2.25 per request before you've generated a single word of output - and if this is a chat product with multi-turn history, that cost repeats and grows every turn unless you're aggressively trimming or caching. Time-to-first-token also increases with context size; a 150K-token prompt can add 3-8 seconds of prefill latency compared to a 2K-token prompt, which is the difference between a snappy assistant and one that feels broken.

3. Irrelevant context actively hurts output quality

This is the counterintuitive one. Feeding a model 50 pages of loosely-related material "just in case" doesn't just waste tokens - it can measurably dilute the answer, because the model has to weigh irrelevant passages against relevant ones during generation. A tightly curated 2,000-token context with exactly the right three paragraphs regularly outperforms a 100,000-token dump of "everything that might be relevant." This is the single biggest reason RAG still matters in an era of million-token windows.

Long context vs RAG: the real trade-off

RAG (retrieval-augmented generation) means storing your knowledge base in a vector database or search index, retrieving only the top-K relevant chunks at query time, and injecting just those into the prompt. See our deep dive on RAG for SMBs turning PDFs into assistants for the implementation details. Here's how the two approaches actually compare in production.

The honest answer for most small-business and internal-tool projects: use long context for a single document or a handful of related files (a contract, a product spec, a customer's own uploaded PDF), and use RAG the moment you're dealing with a knowledge base that grows over time or spans dozens of documents nobody wants to re-paste on every deploy.

Rule of thumb we use for client projects: if the corpus fits in under 30,000 tokens and doesn't change more than once a week, skip RAG entirely. Below that threshold, the engineering cost of a retrieval pipeline isn't worth it.

Prompt structure inside the window: order matters

Given a fixed token budget, how you arrange the prompt changes output quality more than most people expect.

• Put static, reusable content (system instructions, reference documents) first, and put the user's actual question last. Models attend more reliably to content near the end of the context, right before generation starts.
• If you're combining multiple documents, put the most important one closest to the question, not first.
• Use clear delimiters (XML-style tags like <document>...</document> work well with Claude specifically) so the model doesn't blend content from different sources.
• Explicitly ask the model to quote the relevant passage before answering - this measurably improves grounding and gives you a debuggable trace of what it actually used.

# Anthropic-style prompt structure for long-context Q&A
prompt = f"""
<documents>
<document index="1">
<source>contract_v3.pdf</source>
<content>{doc_text}</content>
</document>
</documents>

Instructions: Answer the question using only the documents above.
First quote the exact sentence(s) you're relying on, then answer.
If the answer isn't in the documents, say so explicitly.

Question: {user_question}
"""

That last instruction - "if the answer isn't in the documents, say so" - is the cheapest hallucination guard available to you, and it's astonishing how often it's left out of production prompts.

Prompt caching: the cost lever most teams miss

If you're repeatedly sending the same large context (a system prompt, a reference document, a codebase snapshot) across multiple calls, prompt caching cuts input costs by up to 90% on the cached portion and cuts latency substantially, because the model doesn't re-process tokens it's already seen. Claude, Gemini, and GPT all support some form of this now, with cache lifetimes typically 5 minutes to 1 hour depending on provider and tier. For a deeper walkthrough see our prompt caching guide. The practical implication for context-window decisions: caching makes long-context approaches meaningfully cheaper than the raw per-token math suggests, which shifts the break-even point toward "just use long context" for stable, reusable documents - but it does nothing for the lost-in-the-middle recall problem, which is a model behavior issue, not a cost issue.

Multi-turn chat: the silent context killer

The most common way teams blow their context budget isn't a huge document - it's conversation history. Every turn in a chatbot appends to what gets sent next time, and by turn 20 of an unmanaged conversation you can be re-sending 40,000+ tokens of history just to ask a follow-up question. Three practical fixes:

1. Summarize and truncate. After N turns, collapse older history into a running summary instead of keeping verbatim transcripts.
2. Cap tool results. If your agent calls tools (search, database queries), truncate raw results before they enter history - keep the answer, drop the raw payload.
3. Separate system context from chat history. Static instructions and RAG-retrieved chunks shouldn't accumulate turn over turn; only the actual dialogue should.

This matters especially for products like a WhatsApp AI agent or an AI chatbot on your website, where conversations can run long and users never see the token bill piling up behind the scenes.

When big context genuinely helps

It's not all downside. Long context is the right call when:

• You're doing a single-pass analysis of one large artifact - reviewing an entire codebase, auditing a full contract, summarizing a long transcript. There's no retrieval step to get wrong.
• The task requires holistic understanding, not fact lookup - detecting inconsistencies across an entire document, matching tone throughout a long piece of writing, or code review where cross-file context matters (this is core to how Claude Code's subagents manage context across a codebase).
• The corpus is small enough that retrieval overhead isn't worth it - as noted above, under roughly 30K tokens, just send it all.
• You need verbatim fidelity - RAG chunk boundaries can accidentally split a table or a clause mid-sentence; a full document avoids that failure mode entirely.

When RAG is the better call

• The corpus is large and growing - a support knowledge base, a product catalog, years of internal documentation.
• Freshness matters - RAG lets you update one document without re-processing everything downstream.
• You need source attribution - RAG naturally gives you "this came from document X, section Y," which long-context stuffing makes harder to reconstruct reliably.
• Cost at scale is a constraint - if you're serving thousands of queries a day against the same large corpus, RAG's flat retrieval cost beats re-sending huge contexts even with caching.
• You want structured evaluation - RAG pipelines are easier to eval (precision/recall on retrieval) than "did the model find the right needle in 100K tokens," which ties into broader eval-driven AI development practice.

FAQ

What is a context window in simple terms?

It's the total amount of text an LLM can process in one request, counted in tokens rather than words. It includes your instructions, any documents or history you provide, and the space reserved for the model's response - all sharing the same fixed budget.

Is a bigger context window always better?

No. Bigger windows let you fit more information, but they don't guarantee the model uses all of it well - recall degrades for facts buried in the middle of very long inputs, cost and latency both increase, and irrelevant content can dilute answer quality even when it technically fits.

Should I use RAG or just paste my documents into the prompt?

For a single document or corpus under roughly 30,000 tokens that doesn't change often, pasting it in directly is simpler and usually just as accurate. For a growing knowledge base spanning dozens of documents, or when you need source attribution and predictable cost per query, build a RAG pipeline instead.

How many tokens is a typical PDF or web page?

A dense text page runs roughly 500-800 tokens; a 40-page PDF with tables and formatting often lands around 25,000-30,000 tokens. Hebrew text typically costs 30-50% more tokens than equivalent English content due to tokenizer differences.

Does prompt caching change the long-context vs RAG decision?

Yes, partially. Caching cuts the cost of repeatedly sending the same large context by up to 90%, which makes long-context approaches more affordable for stable, reusable material. It doesn't fix the lost-in-the-middle recall problem, so it shifts the cost calculus but not the accuracy calculus.

What is the "needle in a haystack" test?

It's a benchmark where a specific fact is hidden somewhere inside a long document and the model is asked to retrieve it. It's used to measure how reliably a model can find and use information at different positions within its context window, and it's the standard way vendors report long-context performance.

Get this built right the first time

If you're deciding between long-context prompting and a RAG pipeline for a real product - a support bot, an internal assistant, a document Q&A tool - it's worth 30 minutes to map your actual corpus size, update frequency, and query volume before you commit to an architecture. Book a free call through the contact form or message us directly on WhatsApp.

Sources

• Anthropic - Context windows documentation
• Anthropic - Prompt caching
• OpenAI - Models and context length
• Google - Gemini long context documentation
• Lost in the Middle: How Language Models Use Long Contexts (arXiv)