What is a context window in LLMs?

The context window is the maximum amount of text an LLM can see at once - it includes the system prompt, conversation history, retrieved documents, and the response being generated.

AILLM

Overview

An LLM's context window is its working memory - the total number of tokens it can process in a single call, covering everything: your system prompt, chat history, injected documents, and the response it generates. Unlike human memory, LLMs have no persistent memory between calls unless you explicitly pass prior messages back in. Managing the context window is one of the core engineering challenges in LLM applications.

Key Points

•Context window sizes (2025): GPT-4o: 128k tokens; Claude 3.5 Sonnet: 200k tokens; Gemini 1.5 Pro: 1M tokens.
•Everything in a single API call (system prompt + history + documents + response) must fit within the limit.
•Models can attend to the full window but performance degrades in the middle of very long contexts ("lost in the middle" problem).
•Larger context = higher cost per call (you pay for all input tokens) and slower time-to-first-token.
•Token budget: system prompt (~500t) + history (~2000t) + RAG context (~4000t) + response (~1000t) = ~7500 tokens.
•Context compression strategies: summarise old messages, use a sliding window, or use RAG to replace large documents with relevant chunks.

Examples

Implementing a sliding window to manage chat history

typescript

import { encoding_for_model } from 'tiktoken';

const enc = encoding_for_model('gpt-4o');

function countTokens(messages: { role: string; content: string }[]): number {
  return messages.reduce((total, msg) => {
    return total + enc.encode(msg.content).length + 4; // +4 for role/formatting overhead
  }, 3); // +3 for conversation framing
}

function trimHistory(
  messages: { role: string; content: string }[],
  maxTokens = 6000
): { role: string; content: string }[] {
  // Always keep the system message (index 0)
  const system = messages.slice(0, 1);
  let history = messages.slice(1);

  // Remove oldest messages until we're under the limit
  while (countTokens([...system, ...history]) > maxTokens && history.length > 2) {
    history = history.slice(2); // remove oldest user+assistant pair
  }

  return [...system, ...history];
}

Summarise old messages to compress context

typescript

async function summariseHistory(
  oldMessages: { role: string; content: string }[]
): Promise<{ role: string; content: string }> {
  const summary = await client.chat.completions.create({
    model: 'gpt-4o-mini',    // use cheap model for summarisation
    temperature: 0,
    messages: [
      {
        role: 'system',
        content: 'Summarise the following conversation in 3–5 bullet points, preserving key decisions, facts, and user preferences.',
      },
      {
        role: 'user',
        content: oldMessages.map(m => `${m.role}: ${m.content}`).join('\n'),
      },
    ],
  });

  return {
    role: 'system',
    content: `Previous conversation summary:\n${summary.choices[0].message.content}`,
  };
}

Replace 20 old messages (~4000 tokens) with a summary (~200 tokens) when history grows too long.

Context budget planning

typescript

const MODEL_LIMIT = 128_000; // GPT-4o

const BUDGET = {
  system:   1_000,   // system prompt
  history:  4_000,   // recent chat messages
  rag:      8_000,   // retrieved document chunks
  response: 2_000,   // model output budget
} as const;

const TOTAL = Object.values(BUDGET).reduce((a, b) => a + b, 0); // 15,000
// Leaves 113k tokens of headroom

// Enforce response token limit
await client.chat.completions.create({
  model: 'gpt-4o',
  max_tokens: BUDGET.response,   // cap model output
  messages,
});

Pro Tip

A larger context window does not always mean better results. LLMs tend to pay most attention to the beginning and end of the context window ("lost in the middle" effect). Place your most important instructions at the top of the system prompt and the most relevant context immediately before the user's question.