Chapter 7: Memory and Multi-Turn Conversations
7.4 Context Limit Workarounds
As powerful as OpenAI models like GPT-4o are, they still operate within a context window—a hard limit on how many tokens the model can "remember" in a single interaction. Think of this context window like a conversation buffer: it's the maximum amount of back-and-forth dialogue the AI can consider at once. For GPT-4o, this can be up to 128K tokens, which is massive—but not infinite. To put this in perspective, 128K tokens is roughly equivalent to a 100-page book, allowing for extensive conversations but still requiring careful management.
When your app reaches that limit, the model begins to "forget" earlier parts of the conversation, much like how a person might forget the beginning of a very long conversation. This "forgetting" happens automatically unless you explicitly manage the context through techniques like summarization, selective trimming, or clever engineering solutions.
The model will always prioritize the most recent content, dropping older messages from the beginning of the conversation when new ones are added. This behavior makes it crucial to implement proper context management strategies. In this section, we'll explore effective workarounds that help you keep long, meaningful interactions flowing — even beyond the model's token budget. These strategies ensure your AI maintains coherent, contextually aware conversations while efficiently managing its memory limitations.
7.4.1 The Challenge of the Context Window
A token is the fundamental building block in how language models process and understand text. Think of tokens as the individual pieces of a puzzle that make up the whole text. These tokens can vary significantly in size and complexity:
- Single Characters: The smallest tokens might be just one character, such as:
- Individual letters ("a", "b", "c")
- Punctuation marks (".", ",", "!")
- Special characters ("@", "#", "$")
- Word Fragments: Many tokens are actually parts of words:
- Common prefixes ("pre-", "un-", "re-")
- Common suffixes ("-ing", "-ed", "-tion")
- Word stems and roots that form larger words
- Complete Words: Some tokens represent entire words, particularly:
- Common English words ("the", "and", "but")
- Simple nouns ("cat", "house", "tree")
- Basic verbs ("run", "jump", "sleep")
For example:
- "ChatGPT is amazing." → roughly 5 tokens, where "Chat" and "GPT" are often processed as separate tokens, while common words like "is" are typically single tokens. This example shows how even a simple sentence can be broken down into multiple distinct tokens.
- "Once upon a time in a distant kingdom…" → might be 10–12 tokens, as common phrases like "upon a" are often broken into individual tokens, and punctuation marks like "…" can be counted as separate tokens. This demonstrates how longer phrases get divided into their constituent parts.
Understanding token counting is absolutely crucial for developers and users because it directly impacts how AI models process and respond to text. Your conversation includes several key components:
- System prompts: The instructions that define how the AI should behave
User queries: The questions and inputs you provide
Assistant replies: The responses generated by the AI model
As these components accumulate, your conversation's token count grows rapidly, much like filling up a container with water. Each new message, whether it's a question, response, or instruction, adds more tokens to this total.
When your conversation approaches the model's token limit, an important process occurs: the system begins to drop older messages automatically from the beginning of the conversation. This is similar to how a full container might overflow - the oldest content gets pushed out to make room for new information. This automatic truncation process can have significant consequences:
- Loss of Context: Important earlier details might be forgotten
- Disconnected Responses: The AI might not reference previous important information
- Confusion: Both the model and user might lose track of the conversation's thread
- Broken Continuity: The natural flow of dialogue can become disrupted
This limitation makes it essential to manage your conversation's token usage carefully and strategically to maintain coherent, contextual interactions.
7.4.2 Strategy 1: Summarize Past Dialogues
One of the most reliable workarounds is to summarize older messages and keep only the key information. This powerful technique involves carefully analyzing previous conversation turns and condensing them into concise summaries that capture essential points, decisions, and context. The process works by identifying the most important elements of each conversation segment and creating a condensed version that retains the crucial information while eliminating redundant or less relevant details.
For example, several messages discussing project requirements could be compressed into a single summary stating "User needs a Python-based data processing tool with CSV export capability." This compression might represent multiple messages that included technical discussions, feature requests, and implementation details, all distilled into one clear, actionable statement.
This approach preserves context while dramatically reducing token usage, often compressing dozens of messages into a single, information-rich summary that maintains conversational coherence while freeing up valuable context window space for new interactions. The summarization process can be implemented either automatically using AI-powered tools or manually through careful human review. The key is to maintain the essential meaning and context while significantly reducing the token count, allowing for longer, more meaningful conversations without hitting context limits. This is particularly valuable in scenarios where historical context is crucial, such as complex technical discussions, ongoing project management, or detailed customer support interactions.
Example: Auto-Summarization with OpenAI
def summarize_messages(messages, max_summary_length=120, temperature=0.3):
"""
Summarize a list of conversation messages using OpenAI's API.
Args:
messages (list): List of message dictionaries with 'role' and 'content' keys
max_summary_length (int): Maximum tokens for the summary (default: 120)
temperature (float): Creativity of the response (0.0-1.0, default: 0.3)
Returns:
dict: A system message containing the conversation summary
"""
# Format messages into a readable string
formatted_messages = []
for msg in messages:
# Skip system messages in the summary
if msg["role"] == "system":
continue
# Format each message with role and content
formatted_messages.append(f'{msg["role"].capitalize()}: {msg["content"]}')
# Create the summarization prompt
prompt = [
{
"role": "system",
"content": """You summarize conversations clearly and concisely.
Focus on key points, decisions, and important context.
Use bullet points if multiple topics are discussed."""
},
{
"role": "user",
"content": "Please summarize the following dialogue:\n\n" +
"\n".join(formatted_messages)
}
]
try:
# Call OpenAI API for summarization
response = openai.ChatCompletion.create(
model="gpt-4o",
messages=prompt,
max_tokens=max_summary_length,
temperature=temperature
)
summary = response["choices"][0]["message"]["content"]
# Return formatted system message with summary
return {
"role": "system",
"content": f"Summary of earlier conversation: {summary}"
}
except Exception as e:
# Handle potential API errors
print(f"Error during summarization: {str(e)}")
return {
"role": "system",
"content": "Error: Could not generate conversation summary."
}Code Breakdown:
- Function Definition and Documentation
- Added comprehensive docstring explaining purpose and parameters
- Added configurable parameters for summary length and temperature
- Message Formatting
- Filters out system messages to focus on user-assistant dialogue
- Capitalizes roles for better readability
- Creates a clean, formatted conversation string
- Enhanced Prompt Engineering
- Expanded system instructions for better summaries
- Suggests bullet point format for multi-topic discussions
- Error Handling
- Added try-except block to handle API failures gracefully
- Returns informative error message if summarization fails
- Best Practices
- Uses type hints and clear variable names
- Follows PEP 8 style guidelines
- Implements modular, maintainable code structure
You can call this every time your conversation reaches a certain length (e.g., 80% of the context limit), then replace earlier messages with this summary before continuing.
7.4.3 Strategy 2: Trim Irrelevant Messages
Rather than including the entire conversation history in every interaction, it's crucial to be selective about what context you maintain. This strategic approach helps optimize token usage and maintain relevant context while ensuring the AI can provide meaningful responses. By carefully selecting which information to keep, you can significantly improve the efficiency of your conversations while maintaining their quality. Here's a detailed breakdown of what you should prioritize keeping:
- The system prompt: This contains the fundamental instructions and personality settings that guide the AI's behavior. Without it, the AI might lose its intended role or purpose. The system prompt typically includes critical information like:
- Behavior guidelines and tone of voice
- Specific capabilities or limitations
- Domain-specific knowledge requirements
- The last few user–assistant exchanges: Recent interactions often contain the most relevant context for the current conversation. Usually, the last 3-5 exchanges are sufficient to maintain coherence. This is important because:
- Recent context is most relevant to current questions
- It maintains the natural flow of conversation
- It helps prevent repetition or contradictions
- Any core instructions or facts: Keep any critical information that was established earlier in the conversation, such as user preferences, specific requirements, or important context that influences the entire interaction. This includes:
- User-specified preferences or constraints
- Important decisions or agreements made during the conversation
- Key technical details or specifications that affect the entire discussion
Code Snippet: Trimming Logic
def trim_messages(messages, max_messages=6, model="gpt-4"):
"""
Trim conversation history while preserving system prompts and recent messages.
Args:
messages (list): List of message dictionaries with 'role' and 'content'
max_messages (int): Maximum number of non-system messages to keep
model (str): OpenAI model to use for potential follow-up
Returns:
list: Trimmed message history
"""
try:
# Separate system prompts and conversation
system_prompt = [m for m in messages if m["role"] == "system"]
conversation = [m for m in messages if m["role"] != "system"]
# Calculate tokens (approximate)
def estimate_tokens(text):
return len(text.split()) * 1.3 # Rough estimate
# Get recent messages while staying under limit
trimmed_conversation = conversation[-max_messages:]
# Add a system note about trimming if needed
if len(conversation) > max_messages:
system_prompt.append({
"role": "system",
"content": f"Note: {len(conversation) - max_messages} earlier messages were trimmed for context management."
})
# Combine and validate against OpenAI's limits
final_messages = system_prompt + trimmed_conversation
# Optional: Verify token count with OpenAI
total_tokens = sum(estimate_tokens(m["content"]) for m in final_messages)
if total_tokens > 8000: # Conservative limit for GPT-4
raise ValueError(f"Combined messages exceed token limit: {total_tokens}")
return final_messages
except Exception as e:
print(f"Error trimming messages: {str(e)}")
# Return last few messages as fallback
return system_prompt + conversation[-3:]
# Example usage:
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hi there!"},
# ... many messages later ...
]
trimmed = trim_messages(messages)
# Use with OpenAI API
response = openai.ChatCompletion.create(
model="gpt-4",
messages=trimmed,
temperature=0.7
)Let's break down this code that manages conversation history and token limits:
Main Function Purpose:
The trim_messages function efficiently manages conversation history by preserving system prompts while limiting regular messages.
Key Components:
- Parameters
- messages: List of message dictionaries
- max_messages: Maximum non-system messages to keep (default 6)
- model: Specifies the OpenAI model
- Message Separation
- Separates system prompts from regular conversation
- Preserves all system messages while trimming regular messages
- Token Management
- Implements a simple token estimation (1.3 tokens per word)
- Enforces an 8000 token limit for GPT-4
- Raises an error if the limit is exceeded
- History Tracking
- Keeps track of trimmed messages
- Adds a system note about how many messages were removed
- Maintains the most recent messages within the specified limit
Error Handling:
If an error occurs, the function falls back to returning the system prompts plus the last three messages of conversation.
Usage Example:
The example shows how to use this with OpenAI's API, maintaining a clean conversation history while preventing token overflow.
This implementation ensures that the most recent and relevant information stays in context while minimizing token overload.
7.4.4 Strategy 3: Offload to External Memory (Hybrid Retrieval)
If you're simulating long-term memory, you can store all interactions externally and retrieve only the relevant ones at runtime. This powerful approach uses an external database or storage system to maintain a comprehensive history of all conversations, messages, and information. Instead of burdening the immediate context with excessive data, this method allows for intelligent and selective retrieval of historical information when needed. For example, in a customer service context, the system could instantly access previous interactions with the same customer about similar issues, providing more personalized and informed responses.
Using embeddings, you can transform conversations into mathematical representations that capture their meaning and context. This sophisticated technique enables semantic search capabilities that go far beyond simple keyword matching.
Here's a detailed breakdown of how this system works:
- Each message is transformed into a high-dimensional vector using embedding models
- These vectors capture the semantic meaning of the text by converting words and phrases into numerical representations
- Similar concepts end up closer together in the vector space, enabling intuitive relationship mapping
- The embedding process considers context, synonyms, and related concepts, not just exact matches
- When new queries come in, the system can:
- Convert the new query to a vector using the same embedding model
- Find the most similar stored vectors using efficient similarity search algorithms
- Retrieve only those relevant pieces of context, prioritizing the most semantically related information
- Dynamically adjust the amount of context based on relevance scores
This sophisticated approach allows for efficient and relevant context retrieval without overwhelming the token limits. The system can maintain a virtually unlimited memory while only pulling in the most pertinent information for each interaction. This is particularly valuable in applications requiring deep historical context, such as long-term customer relationships, educational platforms, or complex project management systems.
Essential Tools for Implementation:
openai.Embedding- OpenAI's embedding API that converts text into numerical vectors, capturing semantic meaning and relationships between different pieces of text. This is fundamental for creating searchable vector representations of your conversation history.FAISS- Facebook AI's powerful similarity search library, optimized for searching through millions of high-dimensional vectors quickly. OrPinecone- A managed vector database service that handles vector storage and similarity search with automatic scaling and real-time updates.- Vector search frameworks:
chromadb- An open-source embedding database that makes it easy to store and query your vector embeddings with additional metadataWeaviate- A vector search engine that combines vector storage with GraphQL-based queries and automatic classification capabilities
As we discussed in Chapter 6, section 6.4 about RAG (Retrieval-Augmented Generation), the key principle remains straightforward: store more, inject less. This means maintaining a comprehensive external knowledge base while selectively retrieving only the most relevant information for each interaction, rather than trying to stuff everything into the immediate context window.
class ConversationManager:
def __init__(self, openai_api_key):
self.api_key = openai_api_key
self.summary = ""
self.messages = []
self.summary_interval = 5 # Summarize every 5 messages
self.message_count = 0
def add_message(self, role, content):
"""Add a new message to the conversation."""
self.messages.append({"role": role, "content": content})
self.message_count += 1
# Check if it's time to create a summary
if self.message_count % self.summary_interval == 0:
self.update_summary()
def update_summary(self):
"""Create a summary of recent conversation."""
try:
# Create prompt for summarization
summary_prompt = {
"role": "system",
"content": "Please create a brief summary of the following conversation. "
"Focus on key points and decisions made."
}
# Get last few messages to summarize
recent_messages = self.messages[-self.summary_interval:]
# Request summary from OpenAI
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[
summary_prompt,
*recent_messages,
{"role": "user", "content": "Please summarize our discussion."}
],
temperature=0.7
)
# Update the running summary
new_summary = response.choices[0].message.content
if self.summary:
self.summary = f"{self.summary}\n\nUpdate: {new_summary}"
else:
self.summary = new_summary
# Trim old messages but keep the summary
self.messages = [
{"role": "system", "content": f"Previous context: {self.summary}"},
*recent_messages
]
except Exception as e:
print(f"Error updating summary: {str(e)}")
def get_current_context(self):
"""Get current conversation context including summary."""
if self.summary:
return [
{"role": "system", "content": f"Previous context: {self.summary}"},
*self.messages
]
return self.messages
# Example usage
conversation = ConversationManager("your-api-key")
# Add some messages
conversation.add_message("user", "Can you help me learn Python?")
conversation.add_message("assistant", "Of course! What specific topics interest you?")
conversation.add_message("user", "I'd like to learn about functions.")
conversation.add_message("assistant", "Let's start with the basics of functions...")
conversation.add_message("user", "Can you show me an example?")Code Breakdown:
- Class Structure
- ConversationManager handles all aspects of conversation management and summarization
- Maintains both current messages and running summary
- Configurable summary interval (default: every 5 messages)
- Key Components
- add_message(): Tracks new messages and triggers summary updates
- update_summary(): Creates summaries using OpenAI API
- get_current_context(): Combines summary with recent messages
- Summary Management
- Automatically triggers after specified number of messages
- Preserves context by combining old summaries with new information
- Handles errors gracefully to prevent data loss
- Memory Optimization
- Keeps running summary of older conversations
- Maintains recent messages for immediate context
- Efficiently manages token usage by summarizing older content
Benefits of this Implementation:
- Maintains conversation coherence while managing context window
- Automatically handles summary generation at regular intervals
- Provides easy access to both current context and historical summary
- Scales well for long-running conversations
7.4.5 Strategy 4: Use “Rolling Summaries” for Episodic Memory
As the session progresses, dynamically summarize each section of the conversation and keep an evolving summary that gets updated every few turns. This powerful approach works by continuously monitoring and analyzing the ongoing conversation in discrete segments. The system automatically identifies natural breaks in the discussion, key decision points, and topic transitions, creating a living document that reflects the conversation's evolution.
Here's how it works in practice:
- Every few messages (typically 3-5 turns), the system analyzes the recent conversation
- It extracts essential information, decisions, and conclusions
- These are condensed into a concise but informative summary
- The summary is then merged with previous summaries, maintaining chronological flow
For example, after several messages about Python functions, the summary might begin with "Discussed function definitions, parameters, and return values" while keeping specific code examples readily available in the short-term context. As the conversation progresses to error handling, the summary would expand to include "Explored try/except blocks and their advantages over conditional statements."
You can maintain this "episodic memory" alongside your short-term buffer, creating a two-tier memory system that mirrors human cognition. The short-term buffer contains recent messages with full detail, while the episodic memory holds summarized versions of earlier conversations. This dual-memory approach serves multiple purposes:
- Maintains conversation coherence by keeping both detailed recent context and broader historical context
- Prevents context overflow by condensing older information into compact summaries
- Enables quick reference to previous topics without loading full conversation history
- Creates natural conversation flow by allowing the AI to reference both recent and historical context
This system works similarly to human memory, where we maintain vivid recent memories while older memories become more condensed and summarized over time. This natural approach to memory management helps create more engaging and contextually aware conversations while efficiently managing computational resources.
Example:
# Comprehensive conversation manager with OpenAI integration
import openai
from typing import List, Dict
import time
class ConversationManager:
def __init__(self, api_key: str):
self.api_key = api_key
openai.api_key = api_key
self.session_summary = ""
self.messages: List[Dict[str, str]] = []
self.last_summary_time = time.time()
self.summary_interval = 300 # 5 minutes
def add_message(self, role: str, content: str) -> None:
"""Add a new message and update summary if needed."""
self.messages.append({"role": role, "content": content})
# Check if it's time to update summary
if time.time() - self.last_summary_time > self.summary_interval:
self.update_summary()
def update_summary(self) -> None:
"""Update conversation summary using OpenAI."""
try:
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[
{"role": "system", "content": "Create a brief summary of this conversation."},
*self.messages[-5:] # Last 5 messages for context
],
temperature=0.7,
max_tokens=150
)
new_summary = response.choices[0].message.content
self.session_summary = f"{self.session_summary}\n{new_summary}" if self.session_summary else new_summary
self.last_summary_time = time.time()
except Exception as e:
print(f"Error updating summary: {str(e)}")
def get_context(self) -> List[Dict[str, str]]:
"""Get current conversation context with summary."""
return [
{"role": "system", "content": f"Previous context: {self.session_summary}"},
*self.messages[-5:] # Keep last 5 messages
]
async def get_response(self, user_message: str) -> str:
"""Get AI response using current context."""
self.add_message("user", user_message)
try:
response = openai.ChatCompletion.create(
model="gpt-4",
messages=self.get_context(),
temperature=0.7,
max_tokens=500
)
ai_response = response.choices[0].message.content
self.add_message("assistant", ai_response)
return ai_response
except Exception as e:
print(f"Error getting response: {str(e)}")
return "I apologize, but I encountered an error processing your request."
# Example usage
api_key = "your-openai-api-key"
conversation = ConversationManager(api_key)
# Simulate a conversation
conversation.add_message("user", "I want to learn about Python error handling.")
conversation.add_message("assistant", "Let's start with try/except blocks.")
conversation.add_message("user", "What's the difference between try/except and if/else?")
# Get response with context
response = await conversation.get_response("Can you show me an example?")Code Breakdown:
- Key Components:
- ConversationManager class handles all conversation state and OpenAI interactions
- Automatic summary generation every 5 minutes
- Maintains both recent messages and historical summary
- Type hints and error handling for robustness
- Main Methods:
- add_message(): Tracks conversation history
- update_summary(): Uses GPT-4 to create conversation summaries
- get_context(): Combines summary with recent messages
- get_response(): Handles API interaction for responses
- Features:
- Time-based summary updates instead of message count
- Proper error handling and logging
- Efficient context management with rolling window
- Async support for better performance
Then update session_summary every 5 turns using the summarization strategy from earlier.
7.4.6 Strategy 5: Modular Prompts Instead of Long Threads
For many applications, maintaining extensive message history isn't always necessary or efficient. In fact, keeping long conversation histories can lead to increased API costs, slower response times, and potentially inconsistent outputs. Instead, a more streamlined approach is to generate reusable templates with comprehensive instructions embedded right from the start. This strategy reduces token usage and improves response consistency by front-loading essential context.
Templates can include specific roles, capabilities, and constraints that would otherwise need to be repeatedly communicated. These templates act as a foundation for the AI's behavior and understanding, eliminating the need to carry context through multiple exchanges. When properly designed, they can provide the AI with clear guidelines about its role, expertise level, communication style, and specific domain knowledge.
Example: Building a AI expert coding assistant
# Template for an AI expert coding assistant
system_message = {
"role": "system",
"content": """You are a Python expert with the following capabilities:
- Generate clean, efficient, and well-commented code
- Provide detailed explanations of code functionality
- Follow best practices and PEP 8 standards
- Assume common data science libraries (pandas, numpy) are installed
- Optimize code for readability and performance
When responding:
1. Always include docstrings and comments
2. Explain complex logic
3. Handle edge cases and errors
4. Provide example usage where appropriate"""
}
# Example implementation using OpenAI API
import openai
from typing import Dict, Any
class PythonExpertAssistant:
def __init__(self, api_key: str):
"""Initialize the Python expert assistant with API key."""
self.api_key = api_key
openai.api_key = api_key
self.system_message = system_message
async def get_code_solution(self, prompt: str) -> Dict[str, Any]:
"""
Generate a code solution based on user prompt.
Args:
prompt (str): User's coding question or request
Returns:
Dict containing response and metadata
"""
try:
response = await openai.ChatCompletion.create(
model="gpt-4",
messages=[
self.system_message,
{"role": "user", "content": prompt}
],
temperature=0.7,
max_tokens=1000,
presence_penalty=0.6
)
return {
"code": response.choices[0].message.content,
"tokens_used": response.usage.total_tokens,
"status": "success"
}
except Exception as e:
return {
"error": str(e),
"status": "error"
}
# Example usage
assistant = PythonExpertAssistant("your-api-key")
response = await assistant.get_code_solution(
"Create a function to calculate Fibonacci sequence"
)Code Breakdown:
- System Message Structure
- Defines clear role and capabilities
- Sets expectations for code quality and style
- Establishes consistent response format
- Class Implementation
- Type hints for better code maintainability
- Async support for improved performance
- Proper error handling and response formatting
- API Integration
- Configurable temperature for response creativity
- Token management
- Presence penalty to encourage diverse responses
This eliminates the need to carry this context across every message.
7.4.7 Recap: Practical Tips
The context window, rather than being viewed as a limitation, should be seen as a creative opportunity that pushes us to develop more sophisticated solutions. With thoughtful architectural decisions, we can create systems that effectively manage long-term conversations while staying within token constraints. Here's how each key component contributes:
Summarization allows us to condense lengthy conversation histories into compact, meaningful representations. This preserves essential information while significantly reducing token usage. For example, a 1000-token conversation might be distilled into a 100-token summary capturing the key points.
Retrieval systems enable intelligent access to historical conversation data. By using vector embeddings or semantic search, we can pull relevant past context exactly when needed, rather than carrying the entire conversation history. This creates a more natural flow where previous topics can be recalled contextually.
Trimming strategies help maintain optimal performance by selectively removing less relevant parts of the conversation while keeping crucial context. This might involve removing older messages after summarization or pruning redundant information to stay within token limits.
Dynamic memory injection allows us to strategically insert relevant context when needed. This could include user preferences, previous interactions, or domain-specific knowledge, making conversations more personalized and contextually aware without constant repetition.
When these techniques are combined effectively, the result is an AI system that can maintain extended, natural conversations while managing computational resources efficiently. This creates applications that feel remarkably human-like in their ability to maintain context and refer back to previous interactions, even as conversations extend over long periods or multiple sessions.
7.4 Context Limit Workarounds
As powerful as OpenAI models like GPT-4o are, they still operate within a context window—a hard limit on how many tokens the model can "remember" in a single interaction. Think of this context window like a conversation buffer: it's the maximum amount of back-and-forth dialogue the AI can consider at once. For GPT-4o, this can be up to 128K tokens, which is massive—but not infinite. To put this in perspective, 128K tokens is roughly equivalent to a 100-page book, allowing for extensive conversations but still requiring careful management.
When your app reaches that limit, the model begins to "forget" earlier parts of the conversation, much like how a person might forget the beginning of a very long conversation. This "forgetting" happens automatically unless you explicitly manage the context through techniques like summarization, selective trimming, or clever engineering solutions.
The model will always prioritize the most recent content, dropping older messages from the beginning of the conversation when new ones are added. This behavior makes it crucial to implement proper context management strategies. In this section, we'll explore effective workarounds that help you keep long, meaningful interactions flowing — even beyond the model's token budget. These strategies ensure your AI maintains coherent, contextually aware conversations while efficiently managing its memory limitations.
7.4.1 The Challenge of the Context Window
A token is the fundamental building block in how language models process and understand text. Think of tokens as the individual pieces of a puzzle that make up the whole text. These tokens can vary significantly in size and complexity:
- Single Characters: The smallest tokens might be just one character, such as:
- Individual letters ("a", "b", "c")
- Punctuation marks (".", ",", "!")
- Special characters ("@", "#", "$")
- Word Fragments: Many tokens are actually parts of words:
- Common prefixes ("pre-", "un-", "re-")
- Common suffixes ("-ing", "-ed", "-tion")
- Word stems and roots that form larger words
- Complete Words: Some tokens represent entire words, particularly:
- Common English words ("the", "and", "but")
- Simple nouns ("cat", "house", "tree")
- Basic verbs ("run", "jump", "sleep")
For example:
- "ChatGPT is amazing." → roughly 5 tokens, where "Chat" and "GPT" are often processed as separate tokens, while common words like "is" are typically single tokens. This example shows how even a simple sentence can be broken down into multiple distinct tokens.
- "Once upon a time in a distant kingdom…" → might be 10–12 tokens, as common phrases like "upon a" are often broken into individual tokens, and punctuation marks like "…" can be counted as separate tokens. This demonstrates how longer phrases get divided into their constituent parts.
Understanding token counting is absolutely crucial for developers and users because it directly impacts how AI models process and respond to text. Your conversation includes several key components:
- System prompts: The instructions that define how the AI should behave
User queries: The questions and inputs you provide
Assistant replies: The responses generated by the AI model
As these components accumulate, your conversation's token count grows rapidly, much like filling up a container with water. Each new message, whether it's a question, response, or instruction, adds more tokens to this total.
When your conversation approaches the model's token limit, an important process occurs: the system begins to drop older messages automatically from the beginning of the conversation. This is similar to how a full container might overflow - the oldest content gets pushed out to make room for new information. This automatic truncation process can have significant consequences:
- Loss of Context: Important earlier details might be forgotten
- Disconnected Responses: The AI might not reference previous important information
- Confusion: Both the model and user might lose track of the conversation's thread
- Broken Continuity: The natural flow of dialogue can become disrupted
This limitation makes it essential to manage your conversation's token usage carefully and strategically to maintain coherent, contextual interactions.
7.4.2 Strategy 1: Summarize Past Dialogues
One of the most reliable workarounds is to summarize older messages and keep only the key information. This powerful technique involves carefully analyzing previous conversation turns and condensing them into concise summaries that capture essential points, decisions, and context. The process works by identifying the most important elements of each conversation segment and creating a condensed version that retains the crucial information while eliminating redundant or less relevant details.
For example, several messages discussing project requirements could be compressed into a single summary stating "User needs a Python-based data processing tool with CSV export capability." This compression might represent multiple messages that included technical discussions, feature requests, and implementation details, all distilled into one clear, actionable statement.
This approach preserves context while dramatically reducing token usage, often compressing dozens of messages into a single, information-rich summary that maintains conversational coherence while freeing up valuable context window space for new interactions. The summarization process can be implemented either automatically using AI-powered tools or manually through careful human review. The key is to maintain the essential meaning and context while significantly reducing the token count, allowing for longer, more meaningful conversations without hitting context limits. This is particularly valuable in scenarios where historical context is crucial, such as complex technical discussions, ongoing project management, or detailed customer support interactions.
Example: Auto-Summarization with OpenAI
def summarize_messages(messages, max_summary_length=120, temperature=0.3):
"""
Summarize a list of conversation messages using OpenAI's API.
Args:
messages (list): List of message dictionaries with 'role' and 'content' keys
max_summary_length (int): Maximum tokens for the summary (default: 120)
temperature (float): Creativity of the response (0.0-1.0, default: 0.3)
Returns:
dict: A system message containing the conversation summary
"""
# Format messages into a readable string
formatted_messages = []
for msg in messages:
# Skip system messages in the summary
if msg["role"] == "system":
continue
# Format each message with role and content
formatted_messages.append(f'{msg["role"].capitalize()}: {msg["content"]}')
# Create the summarization prompt
prompt = [
{
"role": "system",
"content": """You summarize conversations clearly and concisely.
Focus on key points, decisions, and important context.
Use bullet points if multiple topics are discussed."""
},
{
"role": "user",
"content": "Please summarize the following dialogue:\n\n" +
"\n".join(formatted_messages)
}
]
try:
# Call OpenAI API for summarization
response = openai.ChatCompletion.create(
model="gpt-4o",
messages=prompt,
max_tokens=max_summary_length,
temperature=temperature
)
summary = response["choices"][0]["message"]["content"]
# Return formatted system message with summary
return {
"role": "system",
"content": f"Summary of earlier conversation: {summary}"
}
except Exception as e:
# Handle potential API errors
print(f"Error during summarization: {str(e)}")
return {
"role": "system",
"content": "Error: Could not generate conversation summary."
}Code Breakdown:
- Function Definition and Documentation
- Added comprehensive docstring explaining purpose and parameters
- Added configurable parameters for summary length and temperature
- Message Formatting
- Filters out system messages to focus on user-assistant dialogue
- Capitalizes roles for better readability
- Creates a clean, formatted conversation string
- Enhanced Prompt Engineering
- Expanded system instructions for better summaries
- Suggests bullet point format for multi-topic discussions
- Error Handling
- Added try-except block to handle API failures gracefully
- Returns informative error message if summarization fails
- Best Practices
- Uses type hints and clear variable names
- Follows PEP 8 style guidelines
- Implements modular, maintainable code structure
You can call this every time your conversation reaches a certain length (e.g., 80% of the context limit), then replace earlier messages with this summary before continuing.
7.4.3 Strategy 2: Trim Irrelevant Messages
Rather than including the entire conversation history in every interaction, it's crucial to be selective about what context you maintain. This strategic approach helps optimize token usage and maintain relevant context while ensuring the AI can provide meaningful responses. By carefully selecting which information to keep, you can significantly improve the efficiency of your conversations while maintaining their quality. Here's a detailed breakdown of what you should prioritize keeping:
- The system prompt: This contains the fundamental instructions and personality settings that guide the AI's behavior. Without it, the AI might lose its intended role or purpose. The system prompt typically includes critical information like:
- Behavior guidelines and tone of voice
- Specific capabilities or limitations
- Domain-specific knowledge requirements
- The last few user–assistant exchanges: Recent interactions often contain the most relevant context for the current conversation. Usually, the last 3-5 exchanges are sufficient to maintain coherence. This is important because:
- Recent context is most relevant to current questions
- It maintains the natural flow of conversation
- It helps prevent repetition or contradictions
- Any core instructions or facts: Keep any critical information that was established earlier in the conversation, such as user preferences, specific requirements, or important context that influences the entire interaction. This includes:
- User-specified preferences or constraints
- Important decisions or agreements made during the conversation
- Key technical details or specifications that affect the entire discussion
Code Snippet: Trimming Logic
def trim_messages(messages, max_messages=6, model="gpt-4"):
"""
Trim conversation history while preserving system prompts and recent messages.
Args:
messages (list): List of message dictionaries with 'role' and 'content'
max_messages (int): Maximum number of non-system messages to keep
model (str): OpenAI model to use for potential follow-up
Returns:
list: Trimmed message history
"""
try:
# Separate system prompts and conversation
system_prompt = [m for m in messages if m["role"] == "system"]
conversation = [m for m in messages if m["role"] != "system"]
# Calculate tokens (approximate)
def estimate_tokens(text):
return len(text.split()) * 1.3 # Rough estimate
# Get recent messages while staying under limit
trimmed_conversation = conversation[-max_messages:]
# Add a system note about trimming if needed
if len(conversation) > max_messages:
system_prompt.append({
"role": "system",
"content": f"Note: {len(conversation) - max_messages} earlier messages were trimmed for context management."
})
# Combine and validate against OpenAI's limits
final_messages = system_prompt + trimmed_conversation
# Optional: Verify token count with OpenAI
total_tokens = sum(estimate_tokens(m["content"]) for m in final_messages)
if total_tokens > 8000: # Conservative limit for GPT-4
raise ValueError(f"Combined messages exceed token limit: {total_tokens}")
return final_messages
except Exception as e:
print(f"Error trimming messages: {str(e)}")
# Return last few messages as fallback
return system_prompt + conversation[-3:]
# Example usage:
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hi there!"},
# ... many messages later ...
]
trimmed = trim_messages(messages)
# Use with OpenAI API
response = openai.ChatCompletion.create(
model="gpt-4",
messages=trimmed,
temperature=0.7
)Let's break down this code that manages conversation history and token limits:
Main Function Purpose:
The trim_messages function efficiently manages conversation history by preserving system prompts while limiting regular messages.
Key Components:
- Parameters
- messages: List of message dictionaries
- max_messages: Maximum non-system messages to keep (default 6)
- model: Specifies the OpenAI model
- Message Separation
- Separates system prompts from regular conversation
- Preserves all system messages while trimming regular messages
- Token Management
- Implements a simple token estimation (1.3 tokens per word)
- Enforces an 8000 token limit for GPT-4
- Raises an error if the limit is exceeded
- History Tracking
- Keeps track of trimmed messages
- Adds a system note about how many messages were removed
- Maintains the most recent messages within the specified limit
Error Handling:
If an error occurs, the function falls back to returning the system prompts plus the last three messages of conversation.
Usage Example:
The example shows how to use this with OpenAI's API, maintaining a clean conversation history while preventing token overflow.
This implementation ensures that the most recent and relevant information stays in context while minimizing token overload.
7.4.4 Strategy 3: Offload to External Memory (Hybrid Retrieval)
If you're simulating long-term memory, you can store all interactions externally and retrieve only the relevant ones at runtime. This powerful approach uses an external database or storage system to maintain a comprehensive history of all conversations, messages, and information. Instead of burdening the immediate context with excessive data, this method allows for intelligent and selective retrieval of historical information when needed. For example, in a customer service context, the system could instantly access previous interactions with the same customer about similar issues, providing more personalized and informed responses.
Using embeddings, you can transform conversations into mathematical representations that capture their meaning and context. This sophisticated technique enables semantic search capabilities that go far beyond simple keyword matching.
Here's a detailed breakdown of how this system works:
- Each message is transformed into a high-dimensional vector using embedding models
- These vectors capture the semantic meaning of the text by converting words and phrases into numerical representations
- Similar concepts end up closer together in the vector space, enabling intuitive relationship mapping
- The embedding process considers context, synonyms, and related concepts, not just exact matches
- When new queries come in, the system can:
- Convert the new query to a vector using the same embedding model
- Find the most similar stored vectors using efficient similarity search algorithms
- Retrieve only those relevant pieces of context, prioritizing the most semantically related information
- Dynamically adjust the amount of context based on relevance scores
This sophisticated approach allows for efficient and relevant context retrieval without overwhelming the token limits. The system can maintain a virtually unlimited memory while only pulling in the most pertinent information for each interaction. This is particularly valuable in applications requiring deep historical context, such as long-term customer relationships, educational platforms, or complex project management systems.
Essential Tools for Implementation:
openai.Embedding- OpenAI's embedding API that converts text into numerical vectors, capturing semantic meaning and relationships between different pieces of text. This is fundamental for creating searchable vector representations of your conversation history.FAISS- Facebook AI's powerful similarity search library, optimized for searching through millions of high-dimensional vectors quickly. OrPinecone- A managed vector database service that handles vector storage and similarity search with automatic scaling and real-time updates.- Vector search frameworks:
chromadb- An open-source embedding database that makes it easy to store and query your vector embeddings with additional metadataWeaviate- A vector search engine that combines vector storage with GraphQL-based queries and automatic classification capabilities
As we discussed in Chapter 6, section 6.4 about RAG (Retrieval-Augmented Generation), the key principle remains straightforward: store more, inject less. This means maintaining a comprehensive external knowledge base while selectively retrieving only the most relevant information for each interaction, rather than trying to stuff everything into the immediate context window.
class ConversationManager:
def __init__(self, openai_api_key):
self.api_key = openai_api_key
self.summary = ""
self.messages = []
self.summary_interval = 5 # Summarize every 5 messages
self.message_count = 0
def add_message(self, role, content):
"""Add a new message to the conversation."""
self.messages.append({"role": role, "content": content})
self.message_count += 1
# Check if it's time to create a summary
if self.message_count % self.summary_interval == 0:
self.update_summary()
def update_summary(self):
"""Create a summary of recent conversation."""
try:
# Create prompt for summarization
summary_prompt = {
"role": "system",
"content": "Please create a brief summary of the following conversation. "
"Focus on key points and decisions made."
}
# Get last few messages to summarize
recent_messages = self.messages[-self.summary_interval:]
# Request summary from OpenAI
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[
summary_prompt,
*recent_messages,
{"role": "user", "content": "Please summarize our discussion."}
],
temperature=0.7
)
# Update the running summary
new_summary = response.choices[0].message.content
if self.summary:
self.summary = f"{self.summary}\n\nUpdate: {new_summary}"
else:
self.summary = new_summary
# Trim old messages but keep the summary
self.messages = [
{"role": "system", "content": f"Previous context: {self.summary}"},
*recent_messages
]
except Exception as e:
print(f"Error updating summary: {str(e)}")
def get_current_context(self):
"""Get current conversation context including summary."""
if self.summary:
return [
{"role": "system", "content": f"Previous context: {self.summary}"},
*self.messages
]
return self.messages
# Example usage
conversation = ConversationManager("your-api-key")
# Add some messages
conversation.add_message("user", "Can you help me learn Python?")
conversation.add_message("assistant", "Of course! What specific topics interest you?")
conversation.add_message("user", "I'd like to learn about functions.")
conversation.add_message("assistant", "Let's start with the basics of functions...")
conversation.add_message("user", "Can you show me an example?")Code Breakdown:
- Class Structure
- ConversationManager handles all aspects of conversation management and summarization
- Maintains both current messages and running summary
- Configurable summary interval (default: every 5 messages)
- Key Components
- add_message(): Tracks new messages and triggers summary updates
- update_summary(): Creates summaries using OpenAI API
- get_current_context(): Combines summary with recent messages
- Summary Management
- Automatically triggers after specified number of messages
- Preserves context by combining old summaries with new information
- Handles errors gracefully to prevent data loss
- Memory Optimization
- Keeps running summary of older conversations
- Maintains recent messages for immediate context
- Efficiently manages token usage by summarizing older content
Benefits of this Implementation:
- Maintains conversation coherence while managing context window
- Automatically handles summary generation at regular intervals
- Provides easy access to both current context and historical summary
- Scales well for long-running conversations
7.4.5 Strategy 4: Use “Rolling Summaries” for Episodic Memory
As the session progresses, dynamically summarize each section of the conversation and keep an evolving summary that gets updated every few turns. This powerful approach works by continuously monitoring and analyzing the ongoing conversation in discrete segments. The system automatically identifies natural breaks in the discussion, key decision points, and topic transitions, creating a living document that reflects the conversation's evolution.
Here's how it works in practice:
- Every few messages (typically 3-5 turns), the system analyzes the recent conversation
- It extracts essential information, decisions, and conclusions
- These are condensed into a concise but informative summary
- The summary is then merged with previous summaries, maintaining chronological flow
For example, after several messages about Python functions, the summary might begin with "Discussed function definitions, parameters, and return values" while keeping specific code examples readily available in the short-term context. As the conversation progresses to error handling, the summary would expand to include "Explored try/except blocks and their advantages over conditional statements."
You can maintain this "episodic memory" alongside your short-term buffer, creating a two-tier memory system that mirrors human cognition. The short-term buffer contains recent messages with full detail, while the episodic memory holds summarized versions of earlier conversations. This dual-memory approach serves multiple purposes:
- Maintains conversation coherence by keeping both detailed recent context and broader historical context
- Prevents context overflow by condensing older information into compact summaries
- Enables quick reference to previous topics without loading full conversation history
- Creates natural conversation flow by allowing the AI to reference both recent and historical context
This system works similarly to human memory, where we maintain vivid recent memories while older memories become more condensed and summarized over time. This natural approach to memory management helps create more engaging and contextually aware conversations while efficiently managing computational resources.
Example:
# Comprehensive conversation manager with OpenAI integration
import openai
from typing import List, Dict
import time
class ConversationManager:
def __init__(self, api_key: str):
self.api_key = api_key
openai.api_key = api_key
self.session_summary = ""
self.messages: List[Dict[str, str]] = []
self.last_summary_time = time.time()
self.summary_interval = 300 # 5 minutes
def add_message(self, role: str, content: str) -> None:
"""Add a new message and update summary if needed."""
self.messages.append({"role": role, "content": content})
# Check if it's time to update summary
if time.time() - self.last_summary_time > self.summary_interval:
self.update_summary()
def update_summary(self) -> None:
"""Update conversation summary using OpenAI."""
try:
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[
{"role": "system", "content": "Create a brief summary of this conversation."},
*self.messages[-5:] # Last 5 messages for context
],
temperature=0.7,
max_tokens=150
)
new_summary = response.choices[0].message.content
self.session_summary = f"{self.session_summary}\n{new_summary}" if self.session_summary else new_summary
self.last_summary_time = time.time()
except Exception as e:
print(f"Error updating summary: {str(e)}")
def get_context(self) -> List[Dict[str, str]]:
"""Get current conversation context with summary."""
return [
{"role": "system", "content": f"Previous context: {self.session_summary}"},
*self.messages[-5:] # Keep last 5 messages
]
async def get_response(self, user_message: str) -> str:
"""Get AI response using current context."""
self.add_message("user", user_message)
try:
response = openai.ChatCompletion.create(
model="gpt-4",
messages=self.get_context(),
temperature=0.7,
max_tokens=500
)
ai_response = response.choices[0].message.content
self.add_message("assistant", ai_response)
return ai_response
except Exception as e:
print(f"Error getting response: {str(e)}")
return "I apologize, but I encountered an error processing your request."
# Example usage
api_key = "your-openai-api-key"
conversation = ConversationManager(api_key)
# Simulate a conversation
conversation.add_message("user", "I want to learn about Python error handling.")
conversation.add_message("assistant", "Let's start with try/except blocks.")
conversation.add_message("user", "What's the difference between try/except and if/else?")
# Get response with context
response = await conversation.get_response("Can you show me an example?")Code Breakdown:
- Key Components:
- ConversationManager class handles all conversation state and OpenAI interactions
- Automatic summary generation every 5 minutes
- Maintains both recent messages and historical summary
- Type hints and error handling for robustness
- Main Methods:
- add_message(): Tracks conversation history
- update_summary(): Uses GPT-4 to create conversation summaries
- get_context(): Combines summary with recent messages
- get_response(): Handles API interaction for responses
- Features:
- Time-based summary updates instead of message count
- Proper error handling and logging
- Efficient context management with rolling window
- Async support for better performance
Then update session_summary every 5 turns using the summarization strategy from earlier.
7.4.6 Strategy 5: Modular Prompts Instead of Long Threads
For many applications, maintaining extensive message history isn't always necessary or efficient. In fact, keeping long conversation histories can lead to increased API costs, slower response times, and potentially inconsistent outputs. Instead, a more streamlined approach is to generate reusable templates with comprehensive instructions embedded right from the start. This strategy reduces token usage and improves response consistency by front-loading essential context.
Templates can include specific roles, capabilities, and constraints that would otherwise need to be repeatedly communicated. These templates act as a foundation for the AI's behavior and understanding, eliminating the need to carry context through multiple exchanges. When properly designed, they can provide the AI with clear guidelines about its role, expertise level, communication style, and specific domain knowledge.
Example: Building a AI expert coding assistant
# Template for an AI expert coding assistant
system_message = {
"role": "system",
"content": """You are a Python expert with the following capabilities:
- Generate clean, efficient, and well-commented code
- Provide detailed explanations of code functionality
- Follow best practices and PEP 8 standards
- Assume common data science libraries (pandas, numpy) are installed
- Optimize code for readability and performance
When responding:
1. Always include docstrings and comments
2. Explain complex logic
3. Handle edge cases and errors
4. Provide example usage where appropriate"""
}
# Example implementation using OpenAI API
import openai
from typing import Dict, Any
class PythonExpertAssistant:
def __init__(self, api_key: str):
"""Initialize the Python expert assistant with API key."""
self.api_key = api_key
openai.api_key = api_key
self.system_message = system_message
async def get_code_solution(self, prompt: str) -> Dict[str, Any]:
"""
Generate a code solution based on user prompt.
Args:
prompt (str): User's coding question or request
Returns:
Dict containing response and metadata
"""
try:
response = await openai.ChatCompletion.create(
model="gpt-4",
messages=[
self.system_message,
{"role": "user", "content": prompt}
],
temperature=0.7,
max_tokens=1000,
presence_penalty=0.6
)
return {
"code": response.choices[0].message.content,
"tokens_used": response.usage.total_tokens,
"status": "success"
}
except Exception as e:
return {
"error": str(e),
"status": "error"
}
# Example usage
assistant = PythonExpertAssistant("your-api-key")
response = await assistant.get_code_solution(
"Create a function to calculate Fibonacci sequence"
)Code Breakdown:
- System Message Structure
- Defines clear role and capabilities
- Sets expectations for code quality and style
- Establishes consistent response format
- Class Implementation
- Type hints for better code maintainability
- Async support for improved performance
- Proper error handling and response formatting
- API Integration
- Configurable temperature for response creativity
- Token management
- Presence penalty to encourage diverse responses
This eliminates the need to carry this context across every message.
7.4.7 Recap: Practical Tips
The context window, rather than being viewed as a limitation, should be seen as a creative opportunity that pushes us to develop more sophisticated solutions. With thoughtful architectural decisions, we can create systems that effectively manage long-term conversations while staying within token constraints. Here's how each key component contributes:
Summarization allows us to condense lengthy conversation histories into compact, meaningful representations. This preserves essential information while significantly reducing token usage. For example, a 1000-token conversation might be distilled into a 100-token summary capturing the key points.
Retrieval systems enable intelligent access to historical conversation data. By using vector embeddings or semantic search, we can pull relevant past context exactly when needed, rather than carrying the entire conversation history. This creates a more natural flow where previous topics can be recalled contextually.
Trimming strategies help maintain optimal performance by selectively removing less relevant parts of the conversation while keeping crucial context. This might involve removing older messages after summarization or pruning redundant information to stay within token limits.
Dynamic memory injection allows us to strategically insert relevant context when needed. This could include user preferences, previous interactions, or domain-specific knowledge, making conversations more personalized and contextually aware without constant repetition.
When these techniques are combined effectively, the result is an AI system that can maintain extended, natural conversations while managing computational resources efficiently. This creates applications that feel remarkably human-like in their ability to maintain context and refer back to previous interactions, even as conversations extend over long periods or multiple sessions.
7.4 Context Limit Workarounds
As powerful as OpenAI models like GPT-4o are, they still operate within a context window—a hard limit on how many tokens the model can "remember" in a single interaction. Think of this context window like a conversation buffer: it's the maximum amount of back-and-forth dialogue the AI can consider at once. For GPT-4o, this can be up to 128K tokens, which is massive—but not infinite. To put this in perspective, 128K tokens is roughly equivalent to a 100-page book, allowing for extensive conversations but still requiring careful management.
When your app reaches that limit, the model begins to "forget" earlier parts of the conversation, much like how a person might forget the beginning of a very long conversation. This "forgetting" happens automatically unless you explicitly manage the context through techniques like summarization, selective trimming, or clever engineering solutions.
The model will always prioritize the most recent content, dropping older messages from the beginning of the conversation when new ones are added. This behavior makes it crucial to implement proper context management strategies. In this section, we'll explore effective workarounds that help you keep long, meaningful interactions flowing — even beyond the model's token budget. These strategies ensure your AI maintains coherent, contextually aware conversations while efficiently managing its memory limitations.
7.4.1 The Challenge of the Context Window
A token is the fundamental building block in how language models process and understand text. Think of tokens as the individual pieces of a puzzle that make up the whole text. These tokens can vary significantly in size and complexity:
- Single Characters: The smallest tokens might be just one character, such as:
- Individual letters ("a", "b", "c")
- Punctuation marks (".", ",", "!")
- Special characters ("@", "#", "$")
- Word Fragments: Many tokens are actually parts of words:
- Common prefixes ("pre-", "un-", "re-")
- Common suffixes ("-ing", "-ed", "-tion")
- Word stems and roots that form larger words
- Complete Words: Some tokens represent entire words, particularly:
- Common English words ("the", "and", "but")
- Simple nouns ("cat", "house", "tree")
- Basic verbs ("run", "jump", "sleep")
For example:
- "ChatGPT is amazing." → roughly 5 tokens, where "Chat" and "GPT" are often processed as separate tokens, while common words like "is" are typically single tokens. This example shows how even a simple sentence can be broken down into multiple distinct tokens.
- "Once upon a time in a distant kingdom…" → might be 10–12 tokens, as common phrases like "upon a" are often broken into individual tokens, and punctuation marks like "…" can be counted as separate tokens. This demonstrates how longer phrases get divided into their constituent parts.
Understanding token counting is absolutely crucial for developers and users because it directly impacts how AI models process and respond to text. Your conversation includes several key components:
- System prompts: The instructions that define how the AI should behave
User queries: The questions and inputs you provide
Assistant replies: The responses generated by the AI model
As these components accumulate, your conversation's token count grows rapidly, much like filling up a container with water. Each new message, whether it's a question, response, or instruction, adds more tokens to this total.
When your conversation approaches the model's token limit, an important process occurs: the system begins to drop older messages automatically from the beginning of the conversation. This is similar to how a full container might overflow - the oldest content gets pushed out to make room for new information. This automatic truncation process can have significant consequences:
- Loss of Context: Important earlier details might be forgotten
- Disconnected Responses: The AI might not reference previous important information
- Confusion: Both the model and user might lose track of the conversation's thread
- Broken Continuity: The natural flow of dialogue can become disrupted
This limitation makes it essential to manage your conversation's token usage carefully and strategically to maintain coherent, contextual interactions.
7.4.2 Strategy 1: Summarize Past Dialogues
One of the most reliable workarounds is to summarize older messages and keep only the key information. This powerful technique involves carefully analyzing previous conversation turns and condensing them into concise summaries that capture essential points, decisions, and context. The process works by identifying the most important elements of each conversation segment and creating a condensed version that retains the crucial information while eliminating redundant or less relevant details.
For example, several messages discussing project requirements could be compressed into a single summary stating "User needs a Python-based data processing tool with CSV export capability." This compression might represent multiple messages that included technical discussions, feature requests, and implementation details, all distilled into one clear, actionable statement.
This approach preserves context while dramatically reducing token usage, often compressing dozens of messages into a single, information-rich summary that maintains conversational coherence while freeing up valuable context window space for new interactions. The summarization process can be implemented either automatically using AI-powered tools or manually through careful human review. The key is to maintain the essential meaning and context while significantly reducing the token count, allowing for longer, more meaningful conversations without hitting context limits. This is particularly valuable in scenarios where historical context is crucial, such as complex technical discussions, ongoing project management, or detailed customer support interactions.
Example: Auto-Summarization with OpenAI
def summarize_messages(messages, max_summary_length=120, temperature=0.3):
"""
Summarize a list of conversation messages using OpenAI's API.
Args:
messages (list): List of message dictionaries with 'role' and 'content' keys
max_summary_length (int): Maximum tokens for the summary (default: 120)
temperature (float): Creativity of the response (0.0-1.0, default: 0.3)
Returns:
dict: A system message containing the conversation summary
"""
# Format messages into a readable string
formatted_messages = []
for msg in messages:
# Skip system messages in the summary
if msg["role"] == "system":
continue
# Format each message with role and content
formatted_messages.append(f'{msg["role"].capitalize()}: {msg["content"]}')
# Create the summarization prompt
prompt = [
{
"role": "system",
"content": """You summarize conversations clearly and concisely.
Focus on key points, decisions, and important context.
Use bullet points if multiple topics are discussed."""
},
{
"role": "user",
"content": "Please summarize the following dialogue:\n\n" +
"\n".join(formatted_messages)
}
]
try:
# Call OpenAI API for summarization
response = openai.ChatCompletion.create(
model="gpt-4o",
messages=prompt,
max_tokens=max_summary_length,
temperature=temperature
)
summary = response["choices"][0]["message"]["content"]
# Return formatted system message with summary
return {
"role": "system",
"content": f"Summary of earlier conversation: {summary}"
}
except Exception as e:
# Handle potential API errors
print(f"Error during summarization: {str(e)}")
return {
"role": "system",
"content": "Error: Could not generate conversation summary."
}Code Breakdown:
- Function Definition and Documentation
- Added comprehensive docstring explaining purpose and parameters
- Added configurable parameters for summary length and temperature
- Message Formatting
- Filters out system messages to focus on user-assistant dialogue
- Capitalizes roles for better readability
- Creates a clean, formatted conversation string
- Enhanced Prompt Engineering
- Expanded system instructions for better summaries
- Suggests bullet point format for multi-topic discussions
- Error Handling
- Added try-except block to handle API failures gracefully
- Returns informative error message if summarization fails
- Best Practices
- Uses type hints and clear variable names
- Follows PEP 8 style guidelines
- Implements modular, maintainable code structure
You can call this every time your conversation reaches a certain length (e.g., 80% of the context limit), then replace earlier messages with this summary before continuing.
7.4.3 Strategy 2: Trim Irrelevant Messages
Rather than including the entire conversation history in every interaction, it's crucial to be selective about what context you maintain. This strategic approach helps optimize token usage and maintain relevant context while ensuring the AI can provide meaningful responses. By carefully selecting which information to keep, you can significantly improve the efficiency of your conversations while maintaining their quality. Here's a detailed breakdown of what you should prioritize keeping:
- The system prompt: This contains the fundamental instructions and personality settings that guide the AI's behavior. Without it, the AI might lose its intended role or purpose. The system prompt typically includes critical information like:
- Behavior guidelines and tone of voice
- Specific capabilities or limitations
- Domain-specific knowledge requirements
- The last few user–assistant exchanges: Recent interactions often contain the most relevant context for the current conversation. Usually, the last 3-5 exchanges are sufficient to maintain coherence. This is important because:
- Recent context is most relevant to current questions
- It maintains the natural flow of conversation
- It helps prevent repetition or contradictions
- Any core instructions or facts: Keep any critical information that was established earlier in the conversation, such as user preferences, specific requirements, or important context that influences the entire interaction. This includes:
- User-specified preferences or constraints
- Important decisions or agreements made during the conversation
- Key technical details or specifications that affect the entire discussion
Code Snippet: Trimming Logic
def trim_messages(messages, max_messages=6, model="gpt-4"):
"""
Trim conversation history while preserving system prompts and recent messages.
Args:
messages (list): List of message dictionaries with 'role' and 'content'
max_messages (int): Maximum number of non-system messages to keep
model (str): OpenAI model to use for potential follow-up
Returns:
list: Trimmed message history
"""
try:
# Separate system prompts and conversation
system_prompt = [m for m in messages if m["role"] == "system"]
conversation = [m for m in messages if m["role"] != "system"]
# Calculate tokens (approximate)
def estimate_tokens(text):
return len(text.split()) * 1.3 # Rough estimate
# Get recent messages while staying under limit
trimmed_conversation = conversation[-max_messages:]
# Add a system note about trimming if needed
if len(conversation) > max_messages:
system_prompt.append({
"role": "system",
"content": f"Note: {len(conversation) - max_messages} earlier messages were trimmed for context management."
})
# Combine and validate against OpenAI's limits
final_messages = system_prompt + trimmed_conversation
# Optional: Verify token count with OpenAI
total_tokens = sum(estimate_tokens(m["content"]) for m in final_messages)
if total_tokens > 8000: # Conservative limit for GPT-4
raise ValueError(f"Combined messages exceed token limit: {total_tokens}")
return final_messages
except Exception as e:
print(f"Error trimming messages: {str(e)}")
# Return last few messages as fallback
return system_prompt + conversation[-3:]
# Example usage:
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hi there!"},
# ... many messages later ...
]
trimmed = trim_messages(messages)
# Use with OpenAI API
response = openai.ChatCompletion.create(
model="gpt-4",
messages=trimmed,
temperature=0.7
)Let's break down this code that manages conversation history and token limits:
Main Function Purpose:
The trim_messages function efficiently manages conversation history by preserving system prompts while limiting regular messages.
Key Components:
- Parameters
- messages: List of message dictionaries
- max_messages: Maximum non-system messages to keep (default 6)
- model: Specifies the OpenAI model
- Message Separation
- Separates system prompts from regular conversation
- Preserves all system messages while trimming regular messages
- Token Management
- Implements a simple token estimation (1.3 tokens per word)
- Enforces an 8000 token limit for GPT-4
- Raises an error if the limit is exceeded
- History Tracking
- Keeps track of trimmed messages
- Adds a system note about how many messages were removed
- Maintains the most recent messages within the specified limit
Error Handling:
If an error occurs, the function falls back to returning the system prompts plus the last three messages of conversation.
Usage Example:
The example shows how to use this with OpenAI's API, maintaining a clean conversation history while preventing token overflow.
This implementation ensures that the most recent and relevant information stays in context while minimizing token overload.
7.4.4 Strategy 3: Offload to External Memory (Hybrid Retrieval)
If you're simulating long-term memory, you can store all interactions externally and retrieve only the relevant ones at runtime. This powerful approach uses an external database or storage system to maintain a comprehensive history of all conversations, messages, and information. Instead of burdening the immediate context with excessive data, this method allows for intelligent and selective retrieval of historical information when needed. For example, in a customer service context, the system could instantly access previous interactions with the same customer about similar issues, providing more personalized and informed responses.
Using embeddings, you can transform conversations into mathematical representations that capture their meaning and context. This sophisticated technique enables semantic search capabilities that go far beyond simple keyword matching.
Here's a detailed breakdown of how this system works:
- Each message is transformed into a high-dimensional vector using embedding models
- These vectors capture the semantic meaning of the text by converting words and phrases into numerical representations
- Similar concepts end up closer together in the vector space, enabling intuitive relationship mapping
- The embedding process considers context, synonyms, and related concepts, not just exact matches
- When new queries come in, the system can:
- Convert the new query to a vector using the same embedding model
- Find the most similar stored vectors using efficient similarity search algorithms
- Retrieve only those relevant pieces of context, prioritizing the most semantically related information
- Dynamically adjust the amount of context based on relevance scores
This sophisticated approach allows for efficient and relevant context retrieval without overwhelming the token limits. The system can maintain a virtually unlimited memory while only pulling in the most pertinent information for each interaction. This is particularly valuable in applications requiring deep historical context, such as long-term customer relationships, educational platforms, or complex project management systems.
Essential Tools for Implementation:
openai.Embedding- OpenAI's embedding API that converts text into numerical vectors, capturing semantic meaning and relationships between different pieces of text. This is fundamental for creating searchable vector representations of your conversation history.FAISS- Facebook AI's powerful similarity search library, optimized for searching through millions of high-dimensional vectors quickly. OrPinecone- A managed vector database service that handles vector storage and similarity search with automatic scaling and real-time updates.- Vector search frameworks:
chromadb- An open-source embedding database that makes it easy to store and query your vector embeddings with additional metadataWeaviate- A vector search engine that combines vector storage with GraphQL-based queries and automatic classification capabilities
As we discussed in Chapter 6, section 6.4 about RAG (Retrieval-Augmented Generation), the key principle remains straightforward: store more, inject less. This means maintaining a comprehensive external knowledge base while selectively retrieving only the most relevant information for each interaction, rather than trying to stuff everything into the immediate context window.
class ConversationManager:
def __init__(self, openai_api_key):
self.api_key = openai_api_key
self.summary = ""
self.messages = []
self.summary_interval = 5 # Summarize every 5 messages
self.message_count = 0
def add_message(self, role, content):
"""Add a new message to the conversation."""
self.messages.append({"role": role, "content": content})
self.message_count += 1
# Check if it's time to create a summary
if self.message_count % self.summary_interval == 0:
self.update_summary()
def update_summary(self):
"""Create a summary of recent conversation."""
try:
# Create prompt for summarization
summary_prompt = {
"role": "system",
"content": "Please create a brief summary of the following conversation. "
"Focus on key points and decisions made."
}
# Get last few messages to summarize
recent_messages = self.messages[-self.summary_interval:]
# Request summary from OpenAI
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[
summary_prompt,
*recent_messages,
{"role": "user", "content": "Please summarize our discussion."}
],
temperature=0.7
)
# Update the running summary
new_summary = response.choices[0].message.content
if self.summary:
self.summary = f"{self.summary}\n\nUpdate: {new_summary}"
else:
self.summary = new_summary
# Trim old messages but keep the summary
self.messages = [
{"role": "system", "content": f"Previous context: {self.summary}"},
*recent_messages
]
except Exception as e:
print(f"Error updating summary: {str(e)}")
def get_current_context(self):
"""Get current conversation context including summary."""
if self.summary:
return [
{"role": "system", "content": f"Previous context: {self.summary}"},
*self.messages
]
return self.messages
# Example usage
conversation = ConversationManager("your-api-key")
# Add some messages
conversation.add_message("user", "Can you help me learn Python?")
conversation.add_message("assistant", "Of course! What specific topics interest you?")
conversation.add_message("user", "I'd like to learn about functions.")
conversation.add_message("assistant", "Let's start with the basics of functions...")
conversation.add_message("user", "Can you show me an example?")Code Breakdown:
- Class Structure
- ConversationManager handles all aspects of conversation management and summarization
- Maintains both current messages and running summary
- Configurable summary interval (default: every 5 messages)
- Key Components
- add_message(): Tracks new messages and triggers summary updates
- update_summary(): Creates summaries using OpenAI API
- get_current_context(): Combines summary with recent messages
- Summary Management
- Automatically triggers after specified number of messages
- Preserves context by combining old summaries with new information
- Handles errors gracefully to prevent data loss
- Memory Optimization
- Keeps running summary of older conversations
- Maintains recent messages for immediate context
- Efficiently manages token usage by summarizing older content
Benefits of this Implementation:
- Maintains conversation coherence while managing context window
- Automatically handles summary generation at regular intervals
- Provides easy access to both current context and historical summary
- Scales well for long-running conversations
7.4.5 Strategy 4: Use “Rolling Summaries” for Episodic Memory
As the session progresses, dynamically summarize each section of the conversation and keep an evolving summary that gets updated every few turns. This powerful approach works by continuously monitoring and analyzing the ongoing conversation in discrete segments. The system automatically identifies natural breaks in the discussion, key decision points, and topic transitions, creating a living document that reflects the conversation's evolution.
Here's how it works in practice:
- Every few messages (typically 3-5 turns), the system analyzes the recent conversation
- It extracts essential information, decisions, and conclusions
- These are condensed into a concise but informative summary
- The summary is then merged with previous summaries, maintaining chronological flow
For example, after several messages about Python functions, the summary might begin with "Discussed function definitions, parameters, and return values" while keeping specific code examples readily available in the short-term context. As the conversation progresses to error handling, the summary would expand to include "Explored try/except blocks and their advantages over conditional statements."
You can maintain this "episodic memory" alongside your short-term buffer, creating a two-tier memory system that mirrors human cognition. The short-term buffer contains recent messages with full detail, while the episodic memory holds summarized versions of earlier conversations. This dual-memory approach serves multiple purposes:
- Maintains conversation coherence by keeping both detailed recent context and broader historical context
- Prevents context overflow by condensing older information into compact summaries
- Enables quick reference to previous topics without loading full conversation history
- Creates natural conversation flow by allowing the AI to reference both recent and historical context
This system works similarly to human memory, where we maintain vivid recent memories while older memories become more condensed and summarized over time. This natural approach to memory management helps create more engaging and contextually aware conversations while efficiently managing computational resources.
Example:
# Comprehensive conversation manager with OpenAI integration
import openai
from typing import List, Dict
import time
class ConversationManager:
def __init__(self, api_key: str):
self.api_key = api_key
openai.api_key = api_key
self.session_summary = ""
self.messages: List[Dict[str, str]] = []
self.last_summary_time = time.time()
self.summary_interval = 300 # 5 minutes
def add_message(self, role: str, content: str) -> None:
"""Add a new message and update summary if needed."""
self.messages.append({"role": role, "content": content})
# Check if it's time to update summary
if time.time() - self.last_summary_time > self.summary_interval:
self.update_summary()
def update_summary(self) -> None:
"""Update conversation summary using OpenAI."""
try:
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[
{"role": "system", "content": "Create a brief summary of this conversation."},
*self.messages[-5:] # Last 5 messages for context
],
temperature=0.7,
max_tokens=150
)
new_summary = response.choices[0].message.content
self.session_summary = f"{self.session_summary}\n{new_summary}" if self.session_summary else new_summary
self.last_summary_time = time.time()
except Exception as e:
print(f"Error updating summary: {str(e)}")
def get_context(self) -> List[Dict[str, str]]:
"""Get current conversation context with summary."""
return [
{"role": "system", "content": f"Previous context: {self.session_summary}"},
*self.messages[-5:] # Keep last 5 messages
]
async def get_response(self, user_message: str) -> str:
"""Get AI response using current context."""
self.add_message("user", user_message)
try:
response = openai.ChatCompletion.create(
model="gpt-4",
messages=self.get_context(),
temperature=0.7,
max_tokens=500
)
ai_response = response.choices[0].message.content
self.add_message("assistant", ai_response)
return ai_response
except Exception as e:
print(f"Error getting response: {str(e)}")
return "I apologize, but I encountered an error processing your request."
# Example usage
api_key = "your-openai-api-key"
conversation = ConversationManager(api_key)
# Simulate a conversation
conversation.add_message("user", "I want to learn about Python error handling.")
conversation.add_message("assistant", "Let's start with try/except blocks.")
conversation.add_message("user", "What's the difference between try/except and if/else?")
# Get response with context
response = await conversation.get_response("Can you show me an example?")Code Breakdown:
- Key Components:
- ConversationManager class handles all conversation state and OpenAI interactions
- Automatic summary generation every 5 minutes
- Maintains both recent messages and historical summary
- Type hints and error handling for robustness
- Main Methods:
- add_message(): Tracks conversation history
- update_summary(): Uses GPT-4 to create conversation summaries
- get_context(): Combines summary with recent messages
- get_response(): Handles API interaction for responses
- Features:
- Time-based summary updates instead of message count
- Proper error handling and logging
- Efficient context management with rolling window
- Async support for better performance
Then update session_summary every 5 turns using the summarization strategy from earlier.
7.4.6 Strategy 5: Modular Prompts Instead of Long Threads
For many applications, maintaining extensive message history isn't always necessary or efficient. In fact, keeping long conversation histories can lead to increased API costs, slower response times, and potentially inconsistent outputs. Instead, a more streamlined approach is to generate reusable templates with comprehensive instructions embedded right from the start. This strategy reduces token usage and improves response consistency by front-loading essential context.
Templates can include specific roles, capabilities, and constraints that would otherwise need to be repeatedly communicated. These templates act as a foundation for the AI's behavior and understanding, eliminating the need to carry context through multiple exchanges. When properly designed, they can provide the AI with clear guidelines about its role, expertise level, communication style, and specific domain knowledge.
Example: Building a AI expert coding assistant
# Template for an AI expert coding assistant
system_message = {
"role": "system",
"content": """You are a Python expert with the following capabilities:
- Generate clean, efficient, and well-commented code
- Provide detailed explanations of code functionality
- Follow best practices and PEP 8 standards
- Assume common data science libraries (pandas, numpy) are installed
- Optimize code for readability and performance
When responding:
1. Always include docstrings and comments
2. Explain complex logic
3. Handle edge cases and errors
4. Provide example usage where appropriate"""
}
# Example implementation using OpenAI API
import openai
from typing import Dict, Any
class PythonExpertAssistant:
def __init__(self, api_key: str):
"""Initialize the Python expert assistant with API key."""
self.api_key = api_key
openai.api_key = api_key
self.system_message = system_message
async def get_code_solution(self, prompt: str) -> Dict[str, Any]:
"""
Generate a code solution based on user prompt.
Args:
prompt (str): User's coding question or request
Returns:
Dict containing response and metadata
"""
try:
response = await openai.ChatCompletion.create(
model="gpt-4",
messages=[
self.system_message,
{"role": "user", "content": prompt}
],
temperature=0.7,
max_tokens=1000,
presence_penalty=0.6
)
return {
"code": response.choices[0].message.content,
"tokens_used": response.usage.total_tokens,
"status": "success"
}
except Exception as e:
return {
"error": str(e),
"status": "error"
}
# Example usage
assistant = PythonExpertAssistant("your-api-key")
response = await assistant.get_code_solution(
"Create a function to calculate Fibonacci sequence"
)Code Breakdown:
- System Message Structure
- Defines clear role and capabilities
- Sets expectations for code quality and style
- Establishes consistent response format
- Class Implementation
- Type hints for better code maintainability
- Async support for improved performance
- Proper error handling and response formatting
- API Integration
- Configurable temperature for response creativity
- Token management
- Presence penalty to encourage diverse responses
This eliminates the need to carry this context across every message.
7.4.7 Recap: Practical Tips
The context window, rather than being viewed as a limitation, should be seen as a creative opportunity that pushes us to develop more sophisticated solutions. With thoughtful architectural decisions, we can create systems that effectively manage long-term conversations while staying within token constraints. Here's how each key component contributes:
Summarization allows us to condense lengthy conversation histories into compact, meaningful representations. This preserves essential information while significantly reducing token usage. For example, a 1000-token conversation might be distilled into a 100-token summary capturing the key points.
Retrieval systems enable intelligent access to historical conversation data. By using vector embeddings or semantic search, we can pull relevant past context exactly when needed, rather than carrying the entire conversation history. This creates a more natural flow where previous topics can be recalled contextually.
Trimming strategies help maintain optimal performance by selectively removing less relevant parts of the conversation while keeping crucial context. This might involve removing older messages after summarization or pruning redundant information to stay within token limits.
Dynamic memory injection allows us to strategically insert relevant context when needed. This could include user preferences, previous interactions, or domain-specific knowledge, making conversations more personalized and contextually aware without constant repetition.
When these techniques are combined effectively, the result is an AI system that can maintain extended, natural conversations while managing computational resources efficiently. This creates applications that feel remarkably human-like in their ability to maintain context and refer back to previous interactions, even as conversations extend over long periods or multiple sessions.
7.4 Context Limit Workarounds
As powerful as OpenAI models like GPT-4o are, they still operate within a context window—a hard limit on how many tokens the model can "remember" in a single interaction. Think of this context window like a conversation buffer: it's the maximum amount of back-and-forth dialogue the AI can consider at once. For GPT-4o, this can be up to 128K tokens, which is massive—but not infinite. To put this in perspective, 128K tokens is roughly equivalent to a 100-page book, allowing for extensive conversations but still requiring careful management.
When your app reaches that limit, the model begins to "forget" earlier parts of the conversation, much like how a person might forget the beginning of a very long conversation. This "forgetting" happens automatically unless you explicitly manage the context through techniques like summarization, selective trimming, or clever engineering solutions.
The model will always prioritize the most recent content, dropping older messages from the beginning of the conversation when new ones are added. This behavior makes it crucial to implement proper context management strategies. In this section, we'll explore effective workarounds that help you keep long, meaningful interactions flowing — even beyond the model's token budget. These strategies ensure your AI maintains coherent, contextually aware conversations while efficiently managing its memory limitations.
7.4.1 The Challenge of the Context Window
A token is the fundamental building block in how language models process and understand text. Think of tokens as the individual pieces of a puzzle that make up the whole text. These tokens can vary significantly in size and complexity:
- Single Characters: The smallest tokens might be just one character, such as:
- Individual letters ("a", "b", "c")
- Punctuation marks (".", ",", "!")
- Special characters ("@", "#", "$")
- Word Fragments: Many tokens are actually parts of words:
- Common prefixes ("pre-", "un-", "re-")
- Common suffixes ("-ing", "-ed", "-tion")
- Word stems and roots that form larger words
- Complete Words: Some tokens represent entire words, particularly:
- Common English words ("the", "and", "but")
- Simple nouns ("cat", "house", "tree")
- Basic verbs ("run", "jump", "sleep")
For example:
- "ChatGPT is amazing." → roughly 5 tokens, where "Chat" and "GPT" are often processed as separate tokens, while common words like "is" are typically single tokens. This example shows how even a simple sentence can be broken down into multiple distinct tokens.
- "Once upon a time in a distant kingdom…" → might be 10–12 tokens, as common phrases like "upon a" are often broken into individual tokens, and punctuation marks like "…" can be counted as separate tokens. This demonstrates how longer phrases get divided into their constituent parts.
Understanding token counting is absolutely crucial for developers and users because it directly impacts how AI models process and respond to text. Your conversation includes several key components:
- System prompts: The instructions that define how the AI should behave
User queries: The questions and inputs you provide
Assistant replies: The responses generated by the AI model
As these components accumulate, your conversation's token count grows rapidly, much like filling up a container with water. Each new message, whether it's a question, response, or instruction, adds more tokens to this total.
When your conversation approaches the model's token limit, an important process occurs: the system begins to drop older messages automatically from the beginning of the conversation. This is similar to how a full container might overflow - the oldest content gets pushed out to make room for new information. This automatic truncation process can have significant consequences:
- Loss of Context: Important earlier details might be forgotten
- Disconnected Responses: The AI might not reference previous important information
- Confusion: Both the model and user might lose track of the conversation's thread
- Broken Continuity: The natural flow of dialogue can become disrupted
This limitation makes it essential to manage your conversation's token usage carefully and strategically to maintain coherent, contextual interactions.
7.4.2 Strategy 1: Summarize Past Dialogues
One of the most reliable workarounds is to summarize older messages and keep only the key information. This powerful technique involves carefully analyzing previous conversation turns and condensing them into concise summaries that capture essential points, decisions, and context. The process works by identifying the most important elements of each conversation segment and creating a condensed version that retains the crucial information while eliminating redundant or less relevant details.
For example, several messages discussing project requirements could be compressed into a single summary stating "User needs a Python-based data processing tool with CSV export capability." This compression might represent multiple messages that included technical discussions, feature requests, and implementation details, all distilled into one clear, actionable statement.
This approach preserves context while dramatically reducing token usage, often compressing dozens of messages into a single, information-rich summary that maintains conversational coherence while freeing up valuable context window space for new interactions. The summarization process can be implemented either automatically using AI-powered tools or manually through careful human review. The key is to maintain the essential meaning and context while significantly reducing the token count, allowing for longer, more meaningful conversations without hitting context limits. This is particularly valuable in scenarios where historical context is crucial, such as complex technical discussions, ongoing project management, or detailed customer support interactions.
Example: Auto-Summarization with OpenAI
def summarize_messages(messages, max_summary_length=120, temperature=0.3):
"""
Summarize a list of conversation messages using OpenAI's API.
Args:
messages (list): List of message dictionaries with 'role' and 'content' keys
max_summary_length (int): Maximum tokens for the summary (default: 120)
temperature (float): Creativity of the response (0.0-1.0, default: 0.3)
Returns:
dict: A system message containing the conversation summary
"""
# Format messages into a readable string
formatted_messages = []
for msg in messages:
# Skip system messages in the summary
if msg["role"] == "system":
continue
# Format each message with role and content
formatted_messages.append(f'{msg["role"].capitalize()}: {msg["content"]}')
# Create the summarization prompt
prompt = [
{
"role": "system",
"content": """You summarize conversations clearly and concisely.
Focus on key points, decisions, and important context.
Use bullet points if multiple topics are discussed."""
},
{
"role": "user",
"content": "Please summarize the following dialogue:\n\n" +
"\n".join(formatted_messages)
}
]
try:
# Call OpenAI API for summarization
response = openai.ChatCompletion.create(
model="gpt-4o",
messages=prompt,
max_tokens=max_summary_length,
temperature=temperature
)
summary = response["choices"][0]["message"]["content"]
# Return formatted system message with summary
return {
"role": "system",
"content": f"Summary of earlier conversation: {summary}"
}
except Exception as e:
# Handle potential API errors
print(f"Error during summarization: {str(e)}")
return {
"role": "system",
"content": "Error: Could not generate conversation summary."
}Code Breakdown:
- Function Definition and Documentation
- Added comprehensive docstring explaining purpose and parameters
- Added configurable parameters for summary length and temperature
- Message Formatting
- Filters out system messages to focus on user-assistant dialogue
- Capitalizes roles for better readability
- Creates a clean, formatted conversation string
- Enhanced Prompt Engineering
- Expanded system instructions for better summaries
- Suggests bullet point format for multi-topic discussions
- Error Handling
- Added try-except block to handle API failures gracefully
- Returns informative error message if summarization fails
- Best Practices
- Uses type hints and clear variable names
- Follows PEP 8 style guidelines
- Implements modular, maintainable code structure
You can call this every time your conversation reaches a certain length (e.g., 80% of the context limit), then replace earlier messages with this summary before continuing.
7.4.3 Strategy 2: Trim Irrelevant Messages
Rather than including the entire conversation history in every interaction, it's crucial to be selective about what context you maintain. This strategic approach helps optimize token usage and maintain relevant context while ensuring the AI can provide meaningful responses. By carefully selecting which information to keep, you can significantly improve the efficiency of your conversations while maintaining their quality. Here's a detailed breakdown of what you should prioritize keeping:
- The system prompt: This contains the fundamental instructions and personality settings that guide the AI's behavior. Without it, the AI might lose its intended role or purpose. The system prompt typically includes critical information like:
- Behavior guidelines and tone of voice
- Specific capabilities or limitations
- Domain-specific knowledge requirements
- The last few user–assistant exchanges: Recent interactions often contain the most relevant context for the current conversation. Usually, the last 3-5 exchanges are sufficient to maintain coherence. This is important because:
- Recent context is most relevant to current questions
- It maintains the natural flow of conversation
- It helps prevent repetition or contradictions
- Any core instructions or facts: Keep any critical information that was established earlier in the conversation, such as user preferences, specific requirements, or important context that influences the entire interaction. This includes:
- User-specified preferences or constraints
- Important decisions or agreements made during the conversation
- Key technical details or specifications that affect the entire discussion
Code Snippet: Trimming Logic
def trim_messages(messages, max_messages=6, model="gpt-4"):
"""
Trim conversation history while preserving system prompts and recent messages.
Args:
messages (list): List of message dictionaries with 'role' and 'content'
max_messages (int): Maximum number of non-system messages to keep
model (str): OpenAI model to use for potential follow-up
Returns:
list: Trimmed message history
"""
try:
# Separate system prompts and conversation
system_prompt = [m for m in messages if m["role"] == "system"]
conversation = [m for m in messages if m["role"] != "system"]
# Calculate tokens (approximate)
def estimate_tokens(text):
return len(text.split()) * 1.3 # Rough estimate
# Get recent messages while staying under limit
trimmed_conversation = conversation[-max_messages:]
# Add a system note about trimming if needed
if len(conversation) > max_messages:
system_prompt.append({
"role": "system",
"content": f"Note: {len(conversation) - max_messages} earlier messages were trimmed for context management."
})
# Combine and validate against OpenAI's limits
final_messages = system_prompt + trimmed_conversation
# Optional: Verify token count with OpenAI
total_tokens = sum(estimate_tokens(m["content"]) for m in final_messages)
if total_tokens > 8000: # Conservative limit for GPT-4
raise ValueError(f"Combined messages exceed token limit: {total_tokens}")
return final_messages
except Exception as e:
print(f"Error trimming messages: {str(e)}")
# Return last few messages as fallback
return system_prompt + conversation[-3:]
# Example usage:
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hi there!"},
# ... many messages later ...
]
trimmed = trim_messages(messages)
# Use with OpenAI API
response = openai.ChatCompletion.create(
model="gpt-4",
messages=trimmed,
temperature=0.7
)Let's break down this code that manages conversation history and token limits:
Main Function Purpose:
The trim_messages function efficiently manages conversation history by preserving system prompts while limiting regular messages.
Key Components:
- Parameters
- messages: List of message dictionaries
- max_messages: Maximum non-system messages to keep (default 6)
- model: Specifies the OpenAI model
- Message Separation
- Separates system prompts from regular conversation
- Preserves all system messages while trimming regular messages
- Token Management
- Implements a simple token estimation (1.3 tokens per word)
- Enforces an 8000 token limit for GPT-4
- Raises an error if the limit is exceeded
- History Tracking
- Keeps track of trimmed messages
- Adds a system note about how many messages were removed
- Maintains the most recent messages within the specified limit
Error Handling:
If an error occurs, the function falls back to returning the system prompts plus the last three messages of conversation.
Usage Example:
The example shows how to use this with OpenAI's API, maintaining a clean conversation history while preventing token overflow.
This implementation ensures that the most recent and relevant information stays in context while minimizing token overload.
7.4.4 Strategy 3: Offload to External Memory (Hybrid Retrieval)
If you're simulating long-term memory, you can store all interactions externally and retrieve only the relevant ones at runtime. This powerful approach uses an external database or storage system to maintain a comprehensive history of all conversations, messages, and information. Instead of burdening the immediate context with excessive data, this method allows for intelligent and selective retrieval of historical information when needed. For example, in a customer service context, the system could instantly access previous interactions with the same customer about similar issues, providing more personalized and informed responses.
Using embeddings, you can transform conversations into mathematical representations that capture their meaning and context. This sophisticated technique enables semantic search capabilities that go far beyond simple keyword matching.
Here's a detailed breakdown of how this system works:
- Each message is transformed into a high-dimensional vector using embedding models
- These vectors capture the semantic meaning of the text by converting words and phrases into numerical representations
- Similar concepts end up closer together in the vector space, enabling intuitive relationship mapping
- The embedding process considers context, synonyms, and related concepts, not just exact matches
- When new queries come in, the system can:
- Convert the new query to a vector using the same embedding model
- Find the most similar stored vectors using efficient similarity search algorithms
- Retrieve only those relevant pieces of context, prioritizing the most semantically related information
- Dynamically adjust the amount of context based on relevance scores
This sophisticated approach allows for efficient and relevant context retrieval without overwhelming the token limits. The system can maintain a virtually unlimited memory while only pulling in the most pertinent information for each interaction. This is particularly valuable in applications requiring deep historical context, such as long-term customer relationships, educational platforms, or complex project management systems.
Essential Tools for Implementation:
openai.Embedding- OpenAI's embedding API that converts text into numerical vectors, capturing semantic meaning and relationships between different pieces of text. This is fundamental for creating searchable vector representations of your conversation history.FAISS- Facebook AI's powerful similarity search library, optimized for searching through millions of high-dimensional vectors quickly. OrPinecone- A managed vector database service that handles vector storage and similarity search with automatic scaling and real-time updates.- Vector search frameworks:
chromadb- An open-source embedding database that makes it easy to store and query your vector embeddings with additional metadataWeaviate- A vector search engine that combines vector storage with GraphQL-based queries and automatic classification capabilities
As we discussed in Chapter 6, section 6.4 about RAG (Retrieval-Augmented Generation), the key principle remains straightforward: store more, inject less. This means maintaining a comprehensive external knowledge base while selectively retrieving only the most relevant information for each interaction, rather than trying to stuff everything into the immediate context window.
class ConversationManager:
def __init__(self, openai_api_key):
self.api_key = openai_api_key
self.summary = ""
self.messages = []
self.summary_interval = 5 # Summarize every 5 messages
self.message_count = 0
def add_message(self, role, content):
"""Add a new message to the conversation."""
self.messages.append({"role": role, "content": content})
self.message_count += 1
# Check if it's time to create a summary
if self.message_count % self.summary_interval == 0:
self.update_summary()
def update_summary(self):
"""Create a summary of recent conversation."""
try:
# Create prompt for summarization
summary_prompt = {
"role": "system",
"content": "Please create a brief summary of the following conversation. "
"Focus on key points and decisions made."
}
# Get last few messages to summarize
recent_messages = self.messages[-self.summary_interval:]
# Request summary from OpenAI
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[
summary_prompt,
*recent_messages,
{"role": "user", "content": "Please summarize our discussion."}
],
temperature=0.7
)
# Update the running summary
new_summary = response.choices[0].message.content
if self.summary:
self.summary = f"{self.summary}\n\nUpdate: {new_summary}"
else:
self.summary = new_summary
# Trim old messages but keep the summary
self.messages = [
{"role": "system", "content": f"Previous context: {self.summary}"},
*recent_messages
]
except Exception as e:
print(f"Error updating summary: {str(e)}")
def get_current_context(self):
"""Get current conversation context including summary."""
if self.summary:
return [
{"role": "system", "content": f"Previous context: {self.summary}"},
*self.messages
]
return self.messages
# Example usage
conversation = ConversationManager("your-api-key")
# Add some messages
conversation.add_message("user", "Can you help me learn Python?")
conversation.add_message("assistant", "Of course! What specific topics interest you?")
conversation.add_message("user", "I'd like to learn about functions.")
conversation.add_message("assistant", "Let's start with the basics of functions...")
conversation.add_message("user", "Can you show me an example?")Code Breakdown:
- Class Structure
- ConversationManager handles all aspects of conversation management and summarization
- Maintains both current messages and running summary
- Configurable summary interval (default: every 5 messages)
- Key Components
- add_message(): Tracks new messages and triggers summary updates
- update_summary(): Creates summaries using OpenAI API
- get_current_context(): Combines summary with recent messages
- Summary Management
- Automatically triggers after specified number of messages
- Preserves context by combining old summaries with new information
- Handles errors gracefully to prevent data loss
- Memory Optimization
- Keeps running summary of older conversations
- Maintains recent messages for immediate context
- Efficiently manages token usage by summarizing older content
Benefits of this Implementation:
- Maintains conversation coherence while managing context window
- Automatically handles summary generation at regular intervals
- Provides easy access to both current context and historical summary
- Scales well for long-running conversations
7.4.5 Strategy 4: Use “Rolling Summaries” for Episodic Memory
As the session progresses, dynamically summarize each section of the conversation and keep an evolving summary that gets updated every few turns. This powerful approach works by continuously monitoring and analyzing the ongoing conversation in discrete segments. The system automatically identifies natural breaks in the discussion, key decision points, and topic transitions, creating a living document that reflects the conversation's evolution.
Here's how it works in practice:
- Every few messages (typically 3-5 turns), the system analyzes the recent conversation
- It extracts essential information, decisions, and conclusions
- These are condensed into a concise but informative summary
- The summary is then merged with previous summaries, maintaining chronological flow
For example, after several messages about Python functions, the summary might begin with "Discussed function definitions, parameters, and return values" while keeping specific code examples readily available in the short-term context. As the conversation progresses to error handling, the summary would expand to include "Explored try/except blocks and their advantages over conditional statements."
You can maintain this "episodic memory" alongside your short-term buffer, creating a two-tier memory system that mirrors human cognition. The short-term buffer contains recent messages with full detail, while the episodic memory holds summarized versions of earlier conversations. This dual-memory approach serves multiple purposes:
- Maintains conversation coherence by keeping both detailed recent context and broader historical context
- Prevents context overflow by condensing older information into compact summaries
- Enables quick reference to previous topics without loading full conversation history
- Creates natural conversation flow by allowing the AI to reference both recent and historical context
This system works similarly to human memory, where we maintain vivid recent memories while older memories become more condensed and summarized over time. This natural approach to memory management helps create more engaging and contextually aware conversations while efficiently managing computational resources.
Example:
# Comprehensive conversation manager with OpenAI integration
import openai
from typing import List, Dict
import time
class ConversationManager:
def __init__(self, api_key: str):
self.api_key = api_key
openai.api_key = api_key
self.session_summary = ""
self.messages: List[Dict[str, str]] = []
self.last_summary_time = time.time()
self.summary_interval = 300 # 5 minutes
def add_message(self, role: str, content: str) -> None:
"""Add a new message and update summary if needed."""
self.messages.append({"role": role, "content": content})
# Check if it's time to update summary
if time.time() - self.last_summary_time > self.summary_interval:
self.update_summary()
def update_summary(self) -> None:
"""Update conversation summary using OpenAI."""
try:
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[
{"role": "system", "content": "Create a brief summary of this conversation."},
*self.messages[-5:] # Last 5 messages for context
],
temperature=0.7,
max_tokens=150
)
new_summary = response.choices[0].message.content
self.session_summary = f"{self.session_summary}\n{new_summary}" if self.session_summary else new_summary
self.last_summary_time = time.time()
except Exception as e:
print(f"Error updating summary: {str(e)}")
def get_context(self) -> List[Dict[str, str]]:
"""Get current conversation context with summary."""
return [
{"role": "system", "content": f"Previous context: {self.session_summary}"},
*self.messages[-5:] # Keep last 5 messages
]
async def get_response(self, user_message: str) -> str:
"""Get AI response using current context."""
self.add_message("user", user_message)
try:
response = openai.ChatCompletion.create(
model="gpt-4",
messages=self.get_context(),
temperature=0.7,
max_tokens=500
)
ai_response = response.choices[0].message.content
self.add_message("assistant", ai_response)
return ai_response
except Exception as e:
print(f"Error getting response: {str(e)}")
return "I apologize, but I encountered an error processing your request."
# Example usage
api_key = "your-openai-api-key"
conversation = ConversationManager(api_key)
# Simulate a conversation
conversation.add_message("user", "I want to learn about Python error handling.")
conversation.add_message("assistant", "Let's start with try/except blocks.")
conversation.add_message("user", "What's the difference between try/except and if/else?")
# Get response with context
response = await conversation.get_response("Can you show me an example?")Code Breakdown:
- Key Components:
- ConversationManager class handles all conversation state and OpenAI interactions
- Automatic summary generation every 5 minutes
- Maintains both recent messages and historical summary
- Type hints and error handling for robustness
- Main Methods:
- add_message(): Tracks conversation history
- update_summary(): Uses GPT-4 to create conversation summaries
- get_context(): Combines summary with recent messages
- get_response(): Handles API interaction for responses
- Features:
- Time-based summary updates instead of message count
- Proper error handling and logging
- Efficient context management with rolling window
- Async support for better performance
Then update session_summary every 5 turns using the summarization strategy from earlier.
7.4.6 Strategy 5: Modular Prompts Instead of Long Threads
For many applications, maintaining extensive message history isn't always necessary or efficient. In fact, keeping long conversation histories can lead to increased API costs, slower response times, and potentially inconsistent outputs. Instead, a more streamlined approach is to generate reusable templates with comprehensive instructions embedded right from the start. This strategy reduces token usage and improves response consistency by front-loading essential context.
Templates can include specific roles, capabilities, and constraints that would otherwise need to be repeatedly communicated. These templates act as a foundation for the AI's behavior and understanding, eliminating the need to carry context through multiple exchanges. When properly designed, they can provide the AI with clear guidelines about its role, expertise level, communication style, and specific domain knowledge.
Example: Building a AI expert coding assistant
# Template for an AI expert coding assistant
system_message = {
"role": "system",
"content": """You are a Python expert with the following capabilities:
- Generate clean, efficient, and well-commented code
- Provide detailed explanations of code functionality
- Follow best practices and PEP 8 standards
- Assume common data science libraries (pandas, numpy) are installed
- Optimize code for readability and performance
When responding:
1. Always include docstrings and comments
2. Explain complex logic
3. Handle edge cases and errors
4. Provide example usage where appropriate"""
}
# Example implementation using OpenAI API
import openai
from typing import Dict, Any
class PythonExpertAssistant:
def __init__(self, api_key: str):
"""Initialize the Python expert assistant with API key."""
self.api_key = api_key
openai.api_key = api_key
self.system_message = system_message
async def get_code_solution(self, prompt: str) -> Dict[str, Any]:
"""
Generate a code solution based on user prompt.
Args:
prompt (str): User's coding question or request
Returns:
Dict containing response and metadata
"""
try:
response = await openai.ChatCompletion.create(
model="gpt-4",
messages=[
self.system_message,
{"role": "user", "content": prompt}
],
temperature=0.7,
max_tokens=1000,
presence_penalty=0.6
)
return {
"code": response.choices[0].message.content,
"tokens_used": response.usage.total_tokens,
"status": "success"
}
except Exception as e:
return {
"error": str(e),
"status": "error"
}
# Example usage
assistant = PythonExpertAssistant("your-api-key")
response = await assistant.get_code_solution(
"Create a function to calculate Fibonacci sequence"
)Code Breakdown:
- System Message Structure
- Defines clear role and capabilities
- Sets expectations for code quality and style
- Establishes consistent response format
- Class Implementation
- Type hints for better code maintainability
- Async support for improved performance
- Proper error handling and response formatting
- API Integration
- Configurable temperature for response creativity
- Token management
- Presence penalty to encourage diverse responses
This eliminates the need to carry this context across every message.
7.4.7 Recap: Practical Tips
The context window, rather than being viewed as a limitation, should be seen as a creative opportunity that pushes us to develop more sophisticated solutions. With thoughtful architectural decisions, we can create systems that effectively manage long-term conversations while staying within token constraints. Here's how each key component contributes:
Summarization allows us to condense lengthy conversation histories into compact, meaningful representations. This preserves essential information while significantly reducing token usage. For example, a 1000-token conversation might be distilled into a 100-token summary capturing the key points.
Retrieval systems enable intelligent access to historical conversation data. By using vector embeddings or semantic search, we can pull relevant past context exactly when needed, rather than carrying the entire conversation history. This creates a more natural flow where previous topics can be recalled contextually.
Trimming strategies help maintain optimal performance by selectively removing less relevant parts of the conversation while keeping crucial context. This might involve removing older messages after summarization or pruning redundant information to stay within token limits.
Dynamic memory injection allows us to strategically insert relevant context when needed. This could include user preferences, previous interactions, or domain-specific knowledge, making conversations more personalized and contextually aware without constant repetition.
When these techniques are combined effectively, the result is an AI system that can maintain extended, natural conversations while managing computational resources efficiently. This creates applications that feel remarkably human-like in their ability to maintain context and refer back to previous interactions, even as conversations extend over long periods or multiple sessions.