How agents remember
and act on the world

The Memory Stack

Agent memory has five distinct layers, each with different latency, capacity, and persistence characteristics. Most production agents use two or three layers in combination. In-context memory is the simplest: everything in the current prompt window. Fast, zero infrastructure, but bounded by context length and lost at conversation end. External memory (vector databases, key-value stores) persists across sessions and scales to millions of records, but requires retrieval — a separate step that introduces latency and retrieval errors.

Episodic memory stores specific past interactions — "last Tuesday I helped this user plan a trip to Tokyo." Semantic memory stores general world knowledge, facts, and concepts the agent has learned. Procedural memory encodes how to do things — workflows, templates, and skill programs the agent can load and execute.

In-Context Memory Management

Most agents start with in-context memory alone. The challenge is managing a growing context window across many turns. Three strategies keep it under control. Sliding window: keep only the last N messages, dropping the oldest. Simple but loses important early context. Summarisation: periodically compress older turns into a running summary appended to the system prompt. Preserves key facts while reducing token count. Selective retention: use an LLM or rule-based filter to decide which messages are worth keeping verbatim — tool results containing data, user-stated preferences, key decisions — and discard the rest.

Summarisation is the most robust strategy for long agentic tasks. Trigger it when the context reaches 70% of the model's limit, not 100% — leaving room for the summarisation call itself and the next few turns.

External Memory with Vector Databases

Vector databases (Pinecone, Weaviate, Chroma, pgvector) store text as dense embeddings and retrieve the most semantically similar records to a query. The agent calls a retrieval tool, gets back k relevant chunks, and injects them into context before reasoning. This is the foundation of every RAG system (covered in depth in 03.2).

The key design decisions for external memory: embedding model (must match at write and read time — never mix models), chunk size (too small loses context, too large dilutes relevance), metadata filtering (always store source, date, and any domain-specific attributes alongside the embedding), and freshness strategy (stale memories can be more harmful than no memory).

Procedural Memory — Skill Programs

Procedural memory is underused in production agents. The idea: store reusable "programs" — structured sequences of steps, prompt templates, or even code — that the agent can retrieve and execute. When the agent learns that a particular approach works well for a class of tasks, it stores that approach as a skill. On future similar tasks, it retrieves and adapts the skill rather than re-deriving the solution from scratch.

This dramatically improves consistency and efficiency for repetitive tasks. Implement it as a simple key-value store keyed by task type, with the agent using a classifier to match incoming tasks to stored skills.

def manage_context(messages, model_limit: int, threshold: float = 0.70):
    token_count = count_tokens(messages)
    
    if token_count < model_limit * threshold:
        return messages   # Still within budget

    # Split: always keep system prompt + recent 6 turns
    system   = messages[0]
    to_compress = messages[1:-6]
    recent   = messages[-6:]

    summary = llm_summarise(
        to_compress,
        prompt="Summarise the key decisions, facts, and "
               "tool results from this conversation history."
    )

    summary_msg = {
        "role": "system",
        "content": f"[Context summary]\n{summary}"
    }
    return [system, summary_msg] + recent

Naive RAG and Its Failures

Naive RAG has four steps: chunk your documents, embed each chunk, store in a vector DB, then at query time embed the question and retrieve the top-k most similar chunks. Simple to implement, but it fails in predictable ways. Chunking mismatch: the answer spans two chunks that are retrieved separately and lose coherence. Embedding gap: the question and the relevant passage are semantically similar but not close in embedding space because they use different vocabulary. Retrieval noise: top-k includes irrelevant chunks that distract or mislead the LLM. Context overflow: too many chunks inflate the prompt and bury the relevant signal.

Chunking Strategies

Chunk size is the most impactful RAG tuning parameter. Fixed-size character splits are the naive baseline — fast but context-unaware. Sentence-aware splitting respects sentence boundaries and dramatically improves coherence. Semantic chunking groups sentences by embedding similarity, keeping related ideas together even if they cross paragraph breaks. Document-structure chunking uses markdown headers, HTML tags, or PDF section boundaries to create chunks that match the document's own logical divisions.

For most use cases: target 256–512 tokens per chunk, add 20–50 token overlap between adjacent chunks, and always store the chunk's source, section title, and position metadata alongside the embedding. The metadata is essential for filtering and for generating citations.

Advanced RAG — Re-ranking and Hybrid Search

Advanced RAG improves retrieval quality with two key additions. Re-ranking: retrieve a larger candidate set (top-20 or top-50) using vector similarity, then run a cross-encoder re-ranker (e.g. Cohere Rerank, BGE-Reranker) that scores each candidate against the full query. Cross-encoders are slower than bi-encoders but significantly more accurate. The top-3 to top-5 after re-ranking are far more relevant than the top-3 from vector search alone.

Hybrid search combines dense vector retrieval with sparse keyword retrieval (BM25). Dense retrieval excels at semantic similarity; sparse retrieval excels at exact keyword matches. Combining them with Reciprocal Rank Fusion (RRF) consistently outperforms either alone, especially for technical queries with specific terms, product codes, or proper nouns that embeddings handle poorly.

Agentic RAG — Self-Correcting Retrieval

Agentic RAG gives the LLM agency over the retrieval process itself. Rather than a single fixed retrieval step, the agent can: reformulate the query if the first retrieval returns poor results, issue multiple sub-queries to answer complex multi-part questions, verify retrieved facts against a secondary source, and decide when retrieval is not needed (the answer is already in context or can be reasoned directly).

This self-correction loop is the key differentiator from naive and advanced RAG. Implement it by giving the agent a retrieve(query) tool and a verify(claim, source) tool, and instructing it in the system prompt to assess retrieval quality before using the results. Agents that evaluate their own retrieval produce significantly fewer hallucinations on knowledge-intensive tasks.

def hybrid_search(query: str, k: int = 5):
    # Dense retrieval — semantic similarity
    q_emb   = embed(query)
    dense   = vector_db.search(q_emb, top_k=30)

    # Sparse retrieval — BM25 keyword match
    sparse  = bm25_index.search(query, top_k=30)

    # Reciprocal Rank Fusion
    scores  = {}
    for rank, doc in enumerate(dense):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (rank + 60)
    for rank, doc in enumerate(sparse):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (rank + 60)

    # Re-rank top-30 with cross-encoder, return top-k
    candidates = top_docs_by_score(scores, n=30)
    return reranker.rerank(query, candidates)[:k]

RAG Evaluation

RAG systems must be evaluated on two separate dimensions: retrieval quality (are the right chunks retrieved? — measured by recall@k and MRR) and generation quality (does the LLM faithfully use the retrieved context? — measured by faithfulness and answer relevance using frameworks like RAGAS). Never evaluate only the final answer; a correct answer generated from wrong chunks is a lucky failure waiting to recur.

Tool Schema Design

The tool schema is the specification the LLM reads to understand what a tool does, what arguments it takes, and what it returns. Good schemas have four properties: precise names — search_web(query) not do_search(q); rich descriptions — explain when to use the tool, not just what it does; typed parameters with constraints and examples; and clear return descriptions — what shape of data comes back and what each field means.

The description field is the most important and most neglected. The LLM reads descriptions to decide which tool to call. Ambiguous or minimal descriptions cause the model to call the wrong tool or to call no tool when it should. Write descriptions as if explaining to a careful junior engineer who has never seen your codebase.

{
  "name": "search_knowledge_base",
  "description": "Search the internal knowledge base for relevant
    documents. Use this when the user asks a question that
    requires product documentation, policies, or historical
    records. Do NOT use for real-time data or external web
    content — use search_web for those queries.",
  "parameters": {
    "type": "object",
    "properties": {
      "query": {
        "type": "string",
        "description": "Natural language search query.
          Be specific. Example: 'refund policy for
          digital purchases' not 'refund'"
      },
      "top_k": {
        "type": "integer",
        "description": "Number of results (1–10). Default 5.",
        "default": 5, "minimum": 1, "maximum": 10
      }
    },
    "required": ["query"]
  }
}

Parallel Tool Calls

Modern LLM APIs (Anthropic, OpenAI) support parallel tool calls — the model can request multiple tool invocations in a single turn, and the agent runs them concurrently before returning results. This can cut multi-tool latency by 60–80% when tools are independent.

Parallel calls introduce a coordination challenge: the agent must wait for all concurrent calls to complete before reasoning over their results. Use asyncio.gather or equivalent. For tools with interdependencies — where the output of tool A is the input to tool B — sequential execution is still required. Represent these as separate turns, not parallel calls.

Tool Selection Strategy

When an agent has many tools, the selection problem becomes significant — a long tool list bloats the prompt, and the model may pick an incorrect tool. Three strategies address this. Tool filtering: at each turn, use a cheap classifier to narrow the available tool set to the most relevant subset (5–10 tools) before calling the main LLM. Tool routing: a small router model decides which tool to call, then a specialised prompt handles that tool's arguments. Dynamic tool loading: tools are loaded from a registry on demand, keeping the active tool set small.

MCP — Model Context Protocol

MCP is an open standard for connecting LLM agents to external tools and data sources through a standardised server interface. Rather than hand-coding every API integration, an MCP-compatible agent can discover and use any MCP server's tools automatically. The agent connects to an MCP server, queries its available tools, and calls them using the standard MCP protocol — the integration code is written once, on the server side.

MCP is particularly valuable in multi-agent systems where different agents need access to different tool sets — each agent connects to the MCP servers relevant to its role, without the orchestrator needing to know the details of each integration. The ecosystem of MCP servers is growing rapidly, covering databases, APIs, file systems, browsers, and development tools.

import asyncio

async def execute_parallel_tools(tool_calls: list):
    # Cap concurrency to avoid rate limits
    semaphore = asyncio.Semaphore(3)

    async def run_one(call):
        async with semaphore:
            try:
                result = await execute_tool(call.name, call.args)
                return {"id": call.id, "result": result, "ok": True}
            except Exception as e:
                return {"id": call.id, "error": str(e), "ok": False}

    return await asyncio.gather(*[run_one(c) for c in tool_calls])

Error Handling for Tools

Tool errors should always be returned to the agent as structured observations, never raised as exceptions that terminate the loop. The structured error should include: the error type, a human-readable message, and — critically — a suggested recovery action. An error message that says "404: resource not found — try searching by name instead of ID" gives the agent a path forward. One that says "ERROR" does not.

Two Approaches: DOM vs. Screenshot

Browser agents work in one of two modes. DOM-based agents interact with the page's HTML structure — clicking elements by selector, reading text from DOM nodes, filling input fields programmatically. This is reliable and fast but breaks when pages use heavy JavaScript rendering, shadow DOM, or dynamically generated element IDs. Tools: Playwright, Puppeteer, Selenium.

Vision-based (screenshot grounding) agents take a screenshot of the screen, send it to a vision-capable LLM, and receive back click coordinates or action descriptions. This works on any UI — web, desktop, or mobile — but is slower, more expensive, and prone to errors when UI elements are small or ambiguous. Tools: Claude Computer Use API, OpenAI CUA, Anthropic's agent framework.

In practice, the best browser agents combine both: DOM-based interaction for structured pages and form inputs, screenshot grounding as a fallback for dynamic or unusual UIs.

Web Scraping Patterns

For data extraction tasks (as opposed to UI automation), structured scraping is more reliable than general browser agents. Give the agent a fetch_page(url) tool that returns the page as clean markdown (after stripping navigation, ads, and boilerplate via a library like Trafilatura or Readability). The LLM then extracts structured data from the markdown — far more reliable than extracting from raw HTML or screenshots.

For dynamic pages (SPAs, infinite scroll, login-required), use Playwright with page.wait_for_selector() to ensure content is loaded before extraction. Implement rate limiting and respect robots.txt — both for legal compliance and to avoid being blocked.

Screenshot Grounding Techniques

Vision-based agents ground their actions on screenshots by identifying the pixel coordinates of UI elements they want to interact with. Reliability improves significantly with three techniques: Set-of-Marks (SoM) prompting — overlay numbered labels on each interactive element before sending to the LLM, so the model identifies an element by its label number rather than estimating raw coordinates. Zoom and crop — crop to the relevant region of the screen before grounding, reducing the signal-to-noise ratio. Accessibility tree augmentation — combine the screenshot with the page's accessibility tree text, giving the LLM both visual and structural information.

Reliability Engineering

Browser agents fail more often than API-based agents. Pages change, CAPTCHAs appear, network requests time out, and UI elements shift between deployments. Build resilience in from the start: explicit waits over fixed sleeps; retry with backoff for transient failures; state verification after every action (take a screenshot, confirm the expected change occurred); and fallback selectors when a primary CSS selector fails.

Track action success rate per page. A browser agent with 90% per-action reliability on a 10-step task has only a 35% end-to-end success rate. Each reliability improvement at the action level has compounding effect on task-level success.

from playwright.async_api import async_playwright
import trafilatura

async def fetch_page_markdown(url: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page    = await browser.new_page()

        await page.goto(url, wait_until="networkidle")
        await page.wait_for_timeout(1000)   # JS settle

        html     = await page.content()
        markdown = trafilatura.extract(
            html, output_format="markdown",
            include_tables=True, no_fallback=False
        )
        await browser.close()
        return markdown or ""