Context Window
The maximum range of data (tokens) that an AI model can process and remember at one time in a single request (prompt). It refers to the model's 'short-term memory' and workspace size.
Detailed explanation
Why it matters in tool selection
The context window determines the 'depth of tasks' the AI can perform. A small window leads to the 'goldfish effect', where the model forgets earlier parts when analyzing dozens of emails or summarizing long reports. On the other hand, a large window enables high-level tasks such as understanding tens of thousands of lines of a codebase entirely or analyzing hours of transcripts.
What to check
- Needle in a Haystack performance: How well does it find specific information in massive datasets?
- Cost efficiency: Input costs spike as the context gets longer, so check if there is a 'context caching' feature
- Latency: Filling up the window may slow down the response generation speed
Real-world use example
If a legal expert uploads five 200-page contracts at once and instructs to find 'toxic clauses commonly found in all contracts', a model with a context window of at least 200,000 tokens is required.
Confusing terms
Token
The unit of measurement for the size of a context window (a fragment of a word).
RAG (Retrieval-Augmented Generation)
A complementary technology that selects only the necessary parts out of tens of thousands of documents to deliver to the AI, overcoming the limitations of the context window.