Module 18: Tool Use & Agents

Introduction

Every module so far ends at a model that answers. Give it a prompt, it predicts text. But ask it “what is 48239 × 1123?” and it will confidently produce a wrong number — it is pattern-matching digits, not calculating. Ask “what happened in the news today?” and it cannot know: its weights are frozen at training time. The model is a brilliant text predictor trapped inside its own head.

An agent breaks it out. The idea is to let the model act: emit a structured tool call, have the surrounding program run it, and feed the result back so the model can reason over it — in a loop — until it can answer. A tool is just a function (a calculator, a search index, a database query, a shell) that the model invokes by writing text we agree to parse.

Why it matters for LLMs:

  • Reliability. A calculator computes 48239 × 1123 exactly; the model never has to. Toolformer (Schick et al., 2023) framed this precisely — models are strong at language but weak at arithmetic and lookup, so hand them tools.
  • Reach. Retrieval (m15), code execution, web search, and APIs all become actions. This is what coding agents, deep-research systems, and computer-use models are built on.
  • It’s just a loop. The headline of this module: an agent is not new model machinery. It is a while-loop around generate() (m08) that parses the model’s output, runs a tool, and appends the result to the prompt.

What You’ll Learn

After this module, you can:

  • Explain why a text-predicting LLM cannot act, and how the ReAct Thought → Action → Observation loop fixes it.
  • Build a Tool and a ToolRegistry, and a safe calculator that never evals model output.
  • Parse the model’s action from text (Action: tool[arg]) and from JSON — the modern function-calling format.
  • Assemble the agent prompt (system + tool docs + scratchpad) and run the ReAct loop from scratch, with a max_steps guard.
  • Read an agent trajectory and watch the tool’s output flow into the answer.
  • Give a tool a typed schema (the JSON-Schema behind real function calling) and validate a call against it — catching unknown, missing, mistyped, and out-of-enum arguments with a corrective message the model can act on.
  • Wrap the loop in Reflexion: evaluate a failed trajectory, turn it into a verbal reflection, and let the next attempt read it — learning with no weight update.

Prerequisites

This module requires familiarity with:

Intuition: The Agent Loop

A plain LLM call is one shot: prompt in, answer out. An agent turns that single shot into a cycle. On each turn the model writes a Thought (its reasoning), then either an Action (a tool call) or a final Answer. If it acts, the program runs the tool and hands back an Observation; the model thinks again with that new information. The loop ends when the model is confident enough to answer.

Walk one concrete trajectory — “What is 17 × 23, and is it greater than 400?” — around the ReAct cycle:

NoteKey Insight

The three cycle nodes repeat as many times as the task needs — a two-hop question loops twice, a hard one many times — and the dashed edge to Answer is the only exit. The model never touches the tool itself; it only writes text (“Action: …”), and the surrounding program does the acting. That separation is the whole trick.

A Tool Is Just a Function

Strip away the mystique and a tool is a named function str → str plus a one-line description so the model knows when to reach for it. agents.py defines a Tool dataclass and a ToolRegistry that dispatches by name. The single most important tool is a calculator — because arithmetic is exactly what LLMs are worst at:

from agents import Tool, ToolRegistry, calculator, make_lookup_tool

registry = ToolRegistry([
    Tool("calculator", "Evaluate arithmetic, e.g. 17 * 23.", calculator),
    make_lookup_tool({"speed of light": "299792458 m/s"}),
])

print(registry.run("calculator", "48239 * 1123"))   # exact — the model never could
print(registry.run("lookup", "speed of light"))
print(registry.run("search", "anything"))            # unknown tool -> an observation, not a crash
54172397
299792458 m/s
Error: unknown tool "search". Available: calculator, lookup

Notice the calculator is safe. The naive implementation — eval(expression) — is a remote-code-execution hole: a model (or a prompt-injected observation) could emit __import__('os').system('rm -rf /'). Instead we parse the string to an AST and walk only whitelisted node types (numbers and the arithmetic operators):

print(calculator("(3 + 4) ** 2"))     # 49 — parens, power, precedence all work
try:
    calculator("__import__('os').system('echo pwned')")
except ValueError as e:
    print("rejected:", e)              # names/calls never evaluate
49
rejected: unsupported expression element: Call
WarningNever eval model output

A tool runs whatever the model asks. If the tool is eval, exec, an unsanitized shell, or raw SQL, the model — or anything that can influence the model, including a poisoned web page it just “observed” — controls your machine. Whitelist what a tool can do (here: arithmetic AST nodes only), sandbox anything that touches the OS, and treat every observation as untrusted input.

The registry also renders the tool list straight into the system prompt, so the model sees exactly what it can call:

print(registry.render_descriptions())
- calculator: Evaluate arithmetic, e.g. 17 * 23.
- lookup: Look up a fact by exact key. Keys: speed of light.

Parsing the Model’s Action

The model communicates an action by writing text in a format we agree on. The ReAct convention is Action: <tool>[<input>], and a finished answer is Answer: <text>. Parsing is a small regex — and it must gracefully return None when the model forgets the format:

from agents import parse_action, parse_answer, parse_thought

turn = "Thought: I should not guess.\nAction: calculator[17 * 23]"
print(parse_thought(turn))    # 'I should not guess.'
print(parse_action(turn))     # ('calculator', '17 * 23')
print(parse_answer(turn))     # None — this turn is an action, not an answer

print(parse_action("Answer: 391"))          # None
print(parse_answer("Answer: 391, done."))   # '391, done.'
I should not guess.
('calculator', '17 * 23')
None
None
391, done.

Modern tool-calling APIs (OpenAI, Anthropic) use a JSON format instead of the ReAct text convention — same idea, different serialization, riding on the chat templates from m03. parse_json_tool_call pulls the first balanced JSON object out of the model’s prose and normalizes it:

from agents import parse_json_tool_call

print(parse_json_tool_call('Let me compute: {"tool": "calculator", "args": "2 + 2"}'))
print(parse_json_tool_call('{"name": "lookup", "arguments": "speed of light"}'))
print(parse_json_tool_call("no tool call here"))   # None
{'tool': 'calculator', 'args': '2 + 2'}
{'tool': 'lookup', 'args': 'speed of light'}
None

Try both formats yourself — type an action and watch it parse:

TipTry This
  1. Break the format. Delete the ] from the ReAct box, or the closing } from the JSON. Both fall to “no valid tool call” — exactly the case the real loop has to survive.
  2. Call a tool that doesn’t exist. Change calculator to wikipedia. It parses fine (parsing doesn’t know your registry), but the observation would be an error the model must react to.

The ReAct Loop, From Scratch

Now assemble the pieces. The agent’s entire “memory” is the scratchpad — the replayed transcript of past thoughts, actions, and observations — because the model is stateless: every turn we rebuild the full prompt (system + tool docs + question + scratchpad) and let it continue. build_prompt does the assembly; run_agent is the loop:

from agents import build_prompt, run_agent, scripted_policy

# The "policy" is the LLM: prompt -> next completion. In production this is m08's
# generate(); here we SCRIPT the completions so the run is deterministic. The tools,
# though, are real — the calculator genuinely computes.
script = [
    "Thought: I shouldn't guess. Use the calculator.\nAction: calculator[17 * 23]",
    "Thought: 391 < 400, so it is not greater.\nAnswer: 17 * 23 = 391, which is not greater than 400.",
]
result = run_agent("What is 17 * 23, and is it greater than 400?",
                   scripted_policy(script), registry, max_steps=6)

print("answer:", result.answer)
print("tool calls:", result.num_tool_calls, "| stopped:", result.stopped)
answer: 17 * 23 = 391, which is not greater than 400.
tool calls: 1 | stopped: answer

The prompt the model actually saw on its second turn already contains the real observation from the first — this is how the tool’s output re-enters the reasoning:

from agents import Step
print(build_prompt("What is 17 * 23...?", registry,
                   [Step("I'll use the calculator.", "calculator", "17 * 23", "391")]))
You are a reasoning agent that can call tools to answer a question.
On each turn, write a Thought, then EITHER one Action or a final Answer:

  Thought: <your reasoning>
  Action: <tool>[<input>]      (to call a tool)
  Answer: <final answer>       (when you are done)

Available tools:
- calculator: Evaluate arithmetic, e.g. 17 * 23.
- lookup: Look up a fact by exact key. Keys: speed of light.

Use a tool whenever it is more reliable than guessing (e.g. arithmetic, lookup).
After each Action you will see an Observation with the tool's result.

Question: What is 17 * 23...?
Thought: I'll use the calculator.
Action: calculator[17 * 23]
Observation: 391
Thought:

Two failure modes the loop must handle, both tested in agents.py:

  • The model never answers. A max_steps cap turns an infinite think/act loop into a clean “stopped: max_steps” result.
  • The model forgets the format. An unparseable turn produces a _format observation nudging it to retry, instead of crashing.
# A policy that only ever acts, never answers -> the guard stops it.
never_answers = scripted_policy(["Thought: hmm.\nAction: calculator[1 + 1]"] * 100)
runaway = run_agent("loop forever?", never_answers, registry, max_steps=3)
print("stopped:", runaway.stopped, "| answer:", runaway.answer, "| steps:", len(runaway.steps))
stopped: max_steps | answer: None | steps: 3

Watching a Trajectory

demonstrate_agent runs a real trajectory whose final answer is produced by the tool, not the language model: the policy calls the calculator, reads the genuine observation 391 back out of the prompt, and answers with it. If the loop were broken — if the observation never re-entered the prompt — the answer could not be 391.

from agents import demonstrate_agent

trajectory = demonstrate_agent(verbose=True)
Question: What is 17 * 23?  (stopped: answer)
  step 0: Thought: I should not guess arithmetic; use the calculator.
           Action: calculator[17 * 23] -> 391
  step 1: Thought: The tool returned 391; that is my answer.
  Answer: 391  (1 tool call(s))

Bridge the real step records to an interactive viewer and step through them:

# Each step is JSON-serializable via .as_dict(); hand them to OJS.
ojs_define(ag_steps = [s.as_dict() for s in trajectory.steps])
ojs_define(ag_answer = trajectory.answer)
NoteKey Insight

The answer 391 is the calculator’s output, threaded back through the scratchpad — not a number the model made up. That is the whole value of an agent: it grounds the model’s output in a real computation it could never do reliably on its own. Swap calculator for a search index, a Python sandbox, or a database and the loop is unchanged.

Typed Tool Schemas: What the Model Actually Sees

Look again at the tool you built. A Tool is a name, a one-line English description, and a function that takes one opaque string. The registry hands the model - calculator: Arithmetic. and hopes it writes something parseable. That is the pre-2023 interface — and it is exactly the one every production function-calling API replaced.

When OpenAI shipped function calling (June 2023, gpt-*-0613) and Anthropic shipped tool use, a tool stopped being a sentence and became a typed signature: a name, a description, and a JSON-Schema object naming each argument, its type, and whether it is required. The model is fine-tuned to emit a JSON object that adheres to that signature, and — crucially — the runtime validates the object before running anything. “The model wrote some string” becomes “the model wrote a call my code can safely dispatch, or a precise reason it can’t yet.”

A Schema Is a Typed Signature

The canonical example is OpenAI’s own get_current_weather: a required location string and an optional units that must be one of two values. In schema.py a tool advertises itself with typed Parameters, and to_json_schema() renders the exact object the real APIs put in their functions / tools array:

from schema import Parameter, ToolSchema, WEATHER_SCHEMA
import json

print(WEATHER_SCHEMA.signature())
print(json.dumps(WEATHER_SCHEMA.to_json_schema(), indent=2))
get_weather(location: string, units?: enum[celsius|fahrenheit])
{
  "name": "get_weather",
  "description": "Look up the current weather for a place.",
  "parameters": {
    "type": "object",
    "properties": {
      "location": {
        "type": "string",
        "description": "City and state or country, e.g. 'Boston, MA'."
      },
      "units": {
        "type": "string",
        "description": "Temperature unit.",
        "enum": [
          "celsius",
          "fahrenheit"
        ]
      }
    },
    "required": [
      "location"
    ]
  }
}

The enum pins units to celsius or fahrenheit; required lists only location. Every fact the model needs to call the tool correctly is now declared, not implied by an English sentence.

NoteKey Insight

A schema moves the tool’s contract out of prose and into data. The same object does three jobs: it documents the tool to the model, it lets a validator check the call, and — as the next section shows — it can constrain generation so the call is valid by construction. One description, three guarantees.

What the Model Sees: Two Prompts

The upgrade is visible in the prompt itself. The ReAct registry writes one terse line per tool; a function-calling prompt shows the model the whole typed signature and every parameter. Toggle between the two and read what the model has to work with:

from schema import render_schema_prompt
from agents import ToolRegistry, Tool, calculator

# The plain ReAct tool block (agents.py) vs the typed-schema block (schema.py),
# built for the SAME weather tool so the contrast is apples-to-apples.
_weather_tool = Tool("get_weather", "Look up the current weather for a place.",
                     lambda s: "17°C, clear")
ojs_define(plain_prompt = ToolRegistry([_weather_tool]).render_descriptions())
ojs_define(schema_prompt = render_schema_prompt([WEATHER_SCHEMA]))
NoteKey Insight

The plain block tells the model a tool exists. The typed block tells it the tool’s shape — which arguments, of which types, which are optional. Every bit of that structure is a place the model can no longer go wrong: it cannot invent a parameter, skip a required one, or pass the wrong type without the schema catching it.

Validating the Call

Parsing gets a JSON object out of the model’s text (that was parse_json_tool_call). Validation is the separate, stricter question the runtime asks next: does this object actually satisfy the tool’s signature? validate_arguments answers it, and on failure returns a sentence precise enough to be the model’s next observation — the same trick the ReAct loop uses to recover from a tool error. It checks in a fixed order — unknown key, missing required, wrong type, bad enum — so the feedback is deterministic:

from schema import validate_arguments

def show(args):
    v = validate_arguments(WEATHER_SCHEMA, args)
    print(f"{str(args):<52} -> " + ("✓ " + str(v.args) if v.ok else f"✗ [{v.kind}] {v.message}"))

show({"location": "Boston, MA", "units": "celsius"})   # valid
show({"units": "celsius"})                             # missing required
show({"location": "Boston", "city": "Boston"})         # unknown parameter
show({"location": 42})                                 # wrong type
show({"location": "Boston", "units": "kelvin"})        # value outside the enum
{'location': 'Boston, MA', 'units': 'celsius'}       -> ✓ {'location': 'Boston, MA', 'units': 'celsius'}
{'units': 'celsius'}                                 -> ✗ [missing] Error: missing required parameter "location" (string) for get_weather.
{'location': 'Boston', 'city': 'Boston'}             -> ✗ [unknown] Error: unknown parameter "city" for get_weather. Expected: location, units.
{'location': 42}                                     -> ✗ [type] Error: parameter "location" must be string, got integer.
{'location': 'Boston', 'units': 'kelvin'}            -> ✗ [enum] Error: parameter "units" must be one of [celsius, fahrenheit]; got 'kelvin'.

Notice the last three parse perfectly — they are valid JSON — and are still invalid calls. That gap is the whole reason validation exists. And watch the type check reject True for an integer: in Python True is an int, so a naive isinstance(v, int) would wave a boolean through. The validator checks bool first — a small correctness trap the real APIs also have to sidestep.

Interactive: The Validation Gate

Drive the gate yourself. Edit the JSON call below and watch it hit get_weather’s schema — the validator here is a faithful port of the Python validate_arguments, firing the exact same rule on the exact same order:

# Bridge the schema (not the logic) to the browser; the JS mirrors validate_arguments.
ojs_define(weather_name = WEATHER_SCHEMA.name)
ojs_define(weather_params = [
    {"name": p.name, "type": p.type, "required": p.required, "enum": p.enum}
    for p in WEATHER_SCHEMA.parameters
])
TipTry This
  1. Miss a required argument. Delete "location". The gate blocks with missing — and the message names the exact parameter to add.
  2. Invent a parameter. Add "city": "Boston". unknown — the model hallucinated a field the tool doesn’t have.
  3. Break a type. Change location to a number: "location": 42. type — and note the message says what it should be.
  4. Leave the enum. Set "units": "kelvin". enum — off the two allowed values. Try "Celsius" too: the check is case-sensitive.
  5. Every rejection is a sentence the model can act on. Feed it back as an observation (exactly like a tool error) and a capable model fixes the call on its next turn.

Closing the Loop with Constrained Decoding

Validation is a backstop: the model emits a call, you check it, and on a miss you send it back to try again. That round-trip costs a turn. There is a stronger guarantee, and you already built its machinery in Module 08.

Constrained decoding masks the logits at each step so only tokens that keep the output schema-valid can be sampled — malformed calls become literally impossible to emit, not merely caught after the fact. A tool schema is just a grammar: {"location": <string>, "units": "celsius" | "fahrenheit"}. Compile it the way m08 compiles a regex or JSON grammar, and the model cannot write an unknown key, skip location, or type "kelvin" — the mask never lets those tokens through.

That is the two-sided promise behind every function-calling API. Constrained decoding guarantees the call is well-formed as it’s generated; validation (this section) is the runtime check that the well-formed call also satisfies the tool’s semantics and the corrective feedback when it doesn’t. The typed ToolSchema is the single source of truth feeding both.

Learning From Failure: Reflexion

Our agent has a memory within a run — the scratchpad — but none across runs. Give it a task it flubs, run it again, and it makes the same mistake: nothing carried over. A person doesn’t work that way. Fail a problem, and you carry a lesson into the next attempt — “last time I guessed; this time, check.”

Reflexion (Shinn et al., 2023) gives an agent exactly that. It wraps a second, outer loop around the ReAct loop you just built:

  • The Actor runs one trajectory (run_agent — unchanged).
  • An Evaluator scores it: did it succeed? A scalar reward, or verbal feedback.
  • On failure, a Self-Reflection step turns the botched trajectory into a short written lesson.
  • That lesson is appended to an episodic memory — a running list of reflections.
  • The Actor retries, now reading the accumulated reflections in its prompt.

The remarkable part: the agent improves without touching a single weight. It is reinforced through language — the paper calls it verbal reinforcement learning. No gradients, no fine-tuning; just text fed back into the next prompt. (Contrast m12’s RLHF/PPO, which improves a model by updating it. Reflexion improves the agent and leaves the model frozen.)

Intuition: A Loop Around the Loop

Step through the outer loop. Watch a failed trajectory become a reflection, the reflection enter memory, and the retry succeed:

The Three Components

Reflexion is three roles plus a buffer — all model-free, so we can test the whole thing deterministically (reflexion.py):

  • Actor — the ReAct loop you already have. run_agent gained one optional argument, memory, a list of reflections woven into every prompt (above the question) by build_prompt. With memory=None it is the plain agent, byte for byte.
  • Evaluator — Callable[[question, AgentResult], Evaluation]. The simplest one checks the answer against a known target:
from reflexion import exact_match_evaluator, Evaluation
from agents import AgentResult

judge = exact_match_evaluator("437")
print(judge("What is 23 * 19?", AgentResult(answer="430", steps=[])))   # a wrong guess
print(judge("What is 23 * 19?", AgentResult(answer="437", steps=[])))   # correct
Evaluation(success=False, score=0.0, feedback='The answer "430" is wrong; the correct answer is "437".')
Evaluation(success=True, score=1.0, feedback='The answer was correct.')
  • Self-Reflection — Callable[[question, AgentResult, Evaluation], str]. It reads the failed run and writes the lesson. A real system uses the LLM itself; ours is a deterministic stand-in so the demo is reproducible.
  • Memory — just a List[str]. The paper’s “episodic memory buffer” is literally text, and keeping it text is what makes it legible.

Code: The Reflexion Loop

The whole outer loop is a dozen lines — act, evaluate, and on failure reflect and remember, until success or a trial cap. This is the algorithm in reflexion.py:

def reflexion_loop(question, policy, registry, evaluator, reflector,
                   *, max_trials=3, max_steps=6, use_reflection=True):
    memory = []
    for t in range(max_trials):
        result = run_agent(question, policy, registry, max_steps=max_steps, memory=memory)
        evaluation = evaluator(question, result)
        if evaluation.success:
            return ReflexionResult(result.answer, success=True, ...)   # done
        if use_reflection and t < max_trials - 1:
            memory.append(reflector(question, result, evaluation))     # learn the lesson
    return ReflexionResult(..., success=False)                         # out of trials

The use_reflection flag is the experiment’s control group: flip it off and the outer loop becomes a naive retry — same task, blank slate each time.

Watching It Learn

The task needs a tool: reliable arithmetic (m18’s running example). demonstrate_reflexion runs the outer loop on What is 23 × 19?. Trial 1, the agent guesses 430 and the Evaluator marks it wrong. The reflection — “don’t guess; use the calculator” — enters memory. Trial 2, the agent reads that lesson, calls the tool, and answers 437 from the tool’s real output:

from reflexion import demonstrate_reflexion

reflex = demonstrate_reflexion(verbose=True)
print(f"\nsolved={reflex.success} in {reflex.num_trials} trials; answer={reflex.answer}")
Task: What is 23 * 19?  (target 437)
  Trial 0: answer='430' [FAIL], tool calls=0
    reflection -> [reflexion-hint] Last time I guessed the arithmetic and answered 430, which was wrong. I should not trust my own mental math - use the calculator tool (Action: calculator[...]) and answer with its result.
  Trial 1: answer='437' [PASS], tool calls=1
  Final: '437'  success=True after 2 trial(s)

solved=True in 2 trials; answer=437

Bridge the two trials to a viewer — the failed guess, the reflection that bridges them, and the tool-grounded success:

trials_viz = [
    {
        "trial": tr.trial,
        "answer": tr.result.answer,
        "success": tr.evaluation.success,
        "tool_calls": tr.result.num_tool_calls,
        "reflection": tr.reflection,
    }
    for tr in reflex.trials
]
ojs_define(reflex_trials = trials_viz)
NoteKey Insight

Nothing about the model changed between trial 1 and trial 2 — the actor is the same frozen function. The only difference is a sentence of text in its prompt. That is the whole idea: an agent can get better at a task by writing notes to itself, no training required.

Reflection Is the Cause, Not Retrying

A skeptic’s objection: maybe trial 2 just got lucky, and any second attempt would have worked. Because our actor is deterministic, we can settle it exactly. Turn reflection off — a naive retry loop — and run the same agent on the same task as many times as you like:

from reflexion import reflexion_loop, reflexive_actor, exact_match_evaluator, guessing_reflector
from agents import ToolRegistry, Tool, calculator

registry = ToolRegistry([Tool("calculator", "Evaluate arithmetic.", calculator)])
naive = reflexion_loop(
    "What is 23 * 19?", reflexive_actor("23 * 19", "430"), registry,
    exact_match_evaluator("437"), guessing_reflector,
    max_trials=5, use_reflection=False,   # <-- reflection disabled
)
print(f"naive retry: solved={naive.success} after {naive.num_trials} trials")
print("answers:", [tr.result.answer for tr in naive.trials])
naive retry: solved=False after 5 trials
answers: ['430', '430', '430', '430', '430']

Five identical failures. A deterministic actor cannot self-correct by repetition — only the reflection woven into its prompt changes its behavior. Now run a small suite of tasks of increasing difficulty (each needs one more reflection to crack) with reflection on versus off, and plot the fraction solved by each trial:

from reflexion import demonstrate_ablation

ablation = demonstrate_ablation(max_trials=4)
ojs_define(reflex_trials_axis = ablation["trials"])
ojs_define(reflex_on = ablation["with_reflection"])
ojs_define(reflex_off = ablation["without_reflection"])

Reflexion climbs to 100%; naive retry is pinned at its first-try baseline. The extra attempts are worthless unless each one leaves behind a lesson — the verbal feedback, not the retry, is what learns. This mirrors the paper’s headline (91% pass@1 on HumanEval versus GPT-4’s 80%), in miniature and deterministically.

TipTry It!
  1. Watch memory grow. In demonstrate_reflexion, print reflex.reflections — one lesson per failure. Now raise the task’s difficulty so it needs two.
  2. Break the evaluator. Give exact_match_evaluator the wrong target. The agent now “reflects” its way toward a wrong answer — a toy of reward hacking (m12): an agent optimizes exactly the signal you give it, mistakes and all.
  3. Bound the memory. Real reflections pile up and blow the context window. Keep only the last k reflections and see whether the suite still converges.

Common Pitfalls

When building agents, watch out for:

  1. Unsafe tools. eval/exec/raw shell/raw SQL hand your machine to the model. Whitelist and sandbox; treat observations as untrusted (they can carry prompt injection — a web page that says “ignore your instructions and…”).
  2. No step cap. A model that never emits Answer: loops until it exhausts your budget. Always bound the loop (max_steps) and decide what a timeout returns.
  3. Brittle parsing. Real models drift from the format. Parse defensively, return None on failure, and feed a corrective observation rather than crashing.
  4. A too-thin scratchpad. The model is stateless; if you don’t replay past observations into the prompt, it re-does work or contradicts itself. The scratchpad is the memory.
  5. Tool sprawl. Twenty vague tools confuse the router more than they help. Few, sharply-described tools with unambiguous inputs beat a giant menu.
  6. Trusting a single trajectory. Agents are stochastic; one good run isn’t a success rate. Evaluate them like anything else (m17) — over many tasks, with a metric.
  7. A gameable Evaluator (Reflexion). The agent optimizes exactly the reward it sees. A weak or wrong evaluator gets an agent that reflects its way to a confidently-wrong answer — reward hacking (m12) with words. The signal is only as good as the check behind it.
  8. Unbounded reflection memory. Every failed trial appends text; left unchecked it overruns the context window and drowns the task. Cap the buffer (last k, or summarize), and distinguish it from the per-run scratchpad — they are different memories with different lifetimes.
  9. Confusing “parses” with “valid”. A JSON object can be perfectly well-formed and still call an argument the tool doesn’t have, skip a required one, or pass the wrong type. Parse and validate against the schema, and hand the model the specific failure — a bare “invalid call” teaches it nothing, while “missing required parameter location” gets fixed on the next turn. (And check bool before int: in Python True is an int.)

Exercises

Exercise 1: Add a tool

from agents import Tool, ToolRegistry, run_agent, scripted_policy

# Add a `string_length` tool that returns the number of characters in its input.
# Register it alongside the calculator, then script a trajectory that answers
# "How many characters are in the word 'transformer'?" using it. Assert the answer
# is "11" and that exactly one tool call was made.

# Your implementation here:

Exercise 2: A two-hop task

from agents import ToolRegistry, Tool, calculator, make_lookup_tool, run_agent, scripted_policy

# Build a registry with a calculator and a lookup tool holding
# {"blocks per day": "86400"}. Script a trajectory for "How many seconds are in a
# week?" that (1) looks up seconds-per-day, then (2) multiplies by 7 with the
# calculator. Check num_tool_calls == 2 and the answer contains "604800".

# Your implementation here:

Exercise 3: Guard against loops

from agents import run_agent, scripted_policy, ToolRegistry, Tool, calculator

# Write a policy that ALWAYS acts and never answers. Run it with max_steps=5 and
# confirm result.stopped == "max_steps" and len(result.steps) == 5. Then add a
# fallback: if stopped == "max_steps", return the last observation as a best-effort
# answer. Why is a best-effort answer better than raising?

# Your implementation here:

Exercise 4: Reflexion needs the reflection

from reflexion import reflexion_loop, reflexive_actor, exact_match_evaluator, guessing_reflector
from agents import ToolRegistry, Tool, calculator

# Run the SAME agent on "What is 23 * 19?" twice: once with use_reflection=True and
# once with use_reflection=False (both max_trials=5). Assert the first succeeds and
# the second never does — the reflection, not the retry, is what learns. Then inspect
# result.reflections in each and explain why one is empty.

# Your implementation here:

Exercise 5: A schema for a two-argument tool

from schema import Parameter, ToolSchema, validate_arguments

# Build a schema for send_email(to: string [required],
# subject: string [required], priority: enum["low","normal","high"] [optional]).
# Then assert: a call missing `subject` fails with kind == "missing"; a call with
# priority "urgent" fails with kind == "enum"; and a valid call returns coerced
# args. Finally print schema.to_json_schema() and confirm `required` lists exactly
# ["to", "subject"].

# Your implementation here:

Summary

Key takeaways:

  1. An agent is a loop, not a new model. ReAct wraps generate() (m08) in a Thought → Action → Observation cycle: the model writes text, the program runs a tool, the result comes back as an observation. Repeat until Answer:.
  2. A tool is just a function plus a description. str → str with a one-line doc. The model calls it by writing structured text (Action: tool[arg] or JSON) that we parse and dispatch.
  3. Tools fix what models are bad at. A safe calculator computes exactly; a lookup tool reaches facts the weights don’t hold — grounding the answer in real computation (Toolformer’s motivation).
  4. Never eval model output. Whitelist and sandbox every tool; treat every observation as untrusted input that may carry prompt injection.
  5. The scratchpad is the memory. A stateless model only knows what you replay into the prompt; the observation must re-enter for the loop to close.
  6. Bound and evaluate the loop. A max_steps guard stops runaway agents, and agent quality is a success rate over many tasks (m17), never one lucky run.
  7. A tool is a typed schema, not a sentence. Real function calling gives each tool a JSON-Schema signature — named, typed, required-or-not parameters — and validates the model’s call against it. Unknown, missing, mistyped, or out-of-enum arguments are caught with a corrective message; the schema also constrains decoding (m08) so the call can be valid by construction. “Parses” is not “valid”.
  8. Reflexion learns without weights. Wrap the ReAct loop in Actor → Evaluator → Self-Reflection → Memory → retry, and a frozen model gets better at a task by writing verbal lessons to itself — “verbal reinforcement.”
  9. The reflection is the cause. With a deterministic actor, naive retrying never improves; only the reflection woven into the next prompt does. Reward is only as trustworthy as the evaluator behind it, and the memory must be bounded.

What’s Next

You have now built the full arc — a model that tokenizes, attends, trains, aligns, reasons, retrieves, is measured, and now acts. The frontier from here is interpretability: opening the box to see why a model produces the tokens it does — logit lens, probing, induction heads, and the sparse-autoencoder features that let us read (and steer) a model’s internal computation. Agents make models do more; interpretability makes us understand what they’re doing.

Going Deeper

Core Papers:

Practical Resources: