Memory Patterns: Context Window Management
Intent
Manage the agent's limited working memory dynamically to control token costs and prevent information loss.
Introduction
In autonomous agent design, the context window is the model's immediate working memory. Unlike human memory, which compresses and digests information naturally, a language model's context is rigid and finite. Every tool call, environment response, and reasoning chain consumes tokens. If left unmanaged, the context window fills with raw logs, causing exponential latency increases, massive token bills, and attention degradation. Context Window Management is the architectural practice of selecting, summarizing, and truncating information to present the model with a highly optimized working memory.
Core Context Management Techniques
Rather than leaving context growth to chance, modern orchestration layers implement three standard techniques:
- Trajectory Compaction: Periodically summarizing older segments of the dialogue transcript. The harness prompts an LLM to distill past steps into a concise summary, replacing many pages of chat history with a few sentences, while preserving critical decisions and active constraints.
- Observation Truncation: Restricting the raw size of tool outputs before they enter the prompt. For example, if a tool outputs a 50KB code file or database log, the harness truncates the text, saves the full content externally, and injects only a small representative excerpt into the active window.
- Structured Scratchpads: Requiring the agent to maintain its own state summary (such as a running tasks file or a list of discovered bugs) in a dedicated text file. The agent reads and rewires this scratchpad at each step, maintaining high-level focus independently of the chat log.
The Retention vs. Compaction Trade-off
Managing the context window is a balancing act between information density and attention clarity. If the orchestration layer compacts the history too aggressively, it risks throwing away subtle details—such as warnings from tools or earlier user constraints—which causes the model to repeat errors. Conversely, if it retains too much detail, token costs rise and the model's attention is distracted by noise, leading to hallucinations.
Empirical evidence in production coding environments indicates that preserving decisions, structural boundaries, and unresolved errors verbatim while aggressively summarizing raw data observations yields the best results.
Trade-offs
This pattern is essential for long-horizon agent stability, but introduces specific trade-offs:
Token Cost Control
Prevents exponential cost scaling by keeping the active context window size bounded, even over hundreds of execution steps.
Improved Instruction Focus
Clearing out irrelevant tool output logs helps the model focus on core instructions and the next active task.
Information Loss
Summarization can drop crucial context, causing the agent to repeat failed commands or lose track of specific user instructions.
Summarization Overhead
Running separate compaction passes requires extra model calls, introducing latency and computational costs that offset token savings.
Known Uses
Context window engineering is a prerequisite in production-grade agents:
- MemGPT: Manages agent memory as a structured operating system, paging information in and out of active context via tool calls [1].
- Reflexion: Distills past mistakes and lessons learned into verbal summaries stored in episodic memory to condition future attempts [2].
- SWE-bench coding agents: Production agents maintain interactive scratchpads (e.g. TODO lists, file changes) in external text files, re-reading them to prevent long git diffs from overloading their context.
References
-
[1]
Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv preprint arXiv:2310.08560.
https://arxiv.org/abs/2310.08560 -
[2]
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. Advances in Neural Information Processing Systems (NeurIPS).
https://arxiv.org/abs/2303.11366