05 · AI & LLM Systems
Agents, tools & MCP — autonomy is engineered, not prompted
This is your home turf: you run an agentic system in production — MCP servers, an adversarial evaluator stage, a human approval gate. This page gives you the shared vocabulary (Anthropic's pattern names, MCP's architecture terms) so you can describe what you already built in the language interviewers recognize.
1What makes an agent
The cleanest definition is Anthropic's, worth quoting: workflows are systems where LLMs and tools are orchestrated through predefined code paths; agents are systems where the LLM dynamically directs its own process and tool usage, deciding what to do next based on what it observes.
while not done:
perceive # context: task + tool results + environment state so far
reason # model decides: what's the next action? am I done?
act # execute a tool call; append the result to context
Every agent — coding agents, computer-use, Sentinel's analysis stage — is this loop with different tools and stop conditions. Three properties fall out of it:
In normal software, branching is code you wrote; in an agent, branching is a model decision. That's the power (handles situations you never enumerated) and the risk (behavior is probabilistic — you can't unit-test every path; you eval distributions of behavior).
The loop's memory is the transcript: task, prior tool calls, results. Manage that context well and the agent stays sharp; let it silt up and quality decays (§9).
An agent can only do what its tools allow — the tool list is your security boundary and blast-radius control, far more reliable than prompt instructions (§8).
2The loop, live — an issue arrives
Walk one concrete run of a Sentinel-style pipeline. Flip the reviewer toggle to watch the evaluator–optimizer loop physically reject a draft and send it back with a critique.
3Tool / function calling — the mechanics
How the model "does things":
| Step | What happens |
|---|---|
| 1 · define | You send the request with tool definitions — name, description, JSON Schema for parameters. |
| 2 · emit | The model, instead of (or before) answering, emits a structured tool-use block: {name: "get_logs", input: {service: "payments", since: "1h"}}. It's still just generated tokens — trained/constrained to match schemas. |
| 3 · execute | Your code executes the call. The model never runs anything — this is the load-bearing fact. Validation, authorization, sandboxing, rate limits, and the decision to comply all live in your runtime. |
| 4 · loop | You append the tool result to the conversation and call the model again. Loop until it produces a final answer. |
Engineering details that separate practitioners from readers
- Descriptions are prompts. The model chooses tools by reading names/descriptions; vague descriptions → wrong tool choice. Great tools have crisp one-purpose descriptions, examples for tricky parameters, and unambiguous names (
search_logsvsquerybeatsrunvsexec). - Tool design > prompt design for agent quality. Fewer, higher-level tools beat many primitive ones — a
get_incident_context(service, window)beats four chained low-level calls the model must orchestrate; every extra decision is an error opportunity. Return results in a model-friendly shape: concise, structured, relevant — a tool that dumps 50KB of JSON poisons the context (§9). - Errors are information: return actionable messages ("date must be ISO-8601; got '2days'") — agents self-correct well when told how they failed. But cap retries: a tool failing identically 5 times should abort the loop, not consume the token budget.
- Validate every input as untrusted: the model can emit any arguments — schema-validate (Pydantic), then authorize (is this agent allowed to touch that resource?). Hallucinated parameters are routine, not exceptional.
- Parallel tool calls (model emits several at once) cut latency for independent reads — but require idempotency discipline for writes.
4MCP — Model Context Protocol
The problem it solves: the M×N integration explosion — M agent applications × N tools/data sources, each pairing hand-integrated. MCP standardizes the interface so any MCP-speaking client can use any MCP server. The popular framing is "USB-C for AI tools"; the more precise one is LSP for tools — as the Language Server Protocol did for editors×languages, MCP does for agents×integrations.
What a server exposes — the three primitives
| Primitive | Who decides | Meaning | Examples |
|---|---|---|---|
| Tools | the model | Model-controlled actions ("call this"); the LLM decides when to invoke, host mediates. | search_logs, create_pr |
| Resources | the application | App-controlled data ("read this") — addressable by URI, loaded into context by the host/app rather than "called" by the model. | file contents, log streams, schemas |
| Prompts | the user | User-controlled templates ("use this workflow") — reusable parameterized prompts a user can invoke. | /summarize-incident |
The distinction is who decides — naming that trichotomy signals real MCP knowledge. (Servers can also request sampling — asking the host's LLM to complete something — keeping the server model-agnostic.)
Plumbing: JSON-RPC 2.0 messages; transports are stdio (server as local subprocess — simple, secure-by-locality) or streamable HTTP (remote servers; OAuth for auth). Sessions negotiate capabilities at init, and servers can notify clients when tool/resource lists change.
5The five Anthropic workflow patterns
From "Building Effective Agents" — the industry's shared vocabulary for LLM system design. Use these names in interviews.
When: the task decomposes into clean, fixed sequential steps — draft → critique → revise; extract → transform → format — each call consuming the previous output, optionally with programmatic "gates" between. Each step is simpler and independently testable; you trade latency for accuracy.
When: inputs fall into distinct classes needing different handling — a classifier (LLM or heuristic) sends simple questions to a small model, refunds to the refund flow, hard reasoning to the frontier model. Each path's prompt stays sharp, plus the cost/latency win.
When: you need speed or confidence — sectioning (independent subtasks: review five files at once) or voting (same task N times, aggregate — majority verdict on "is this vulnerable?").
When: subtasks can't be known in advance — a central LLM dynamically decomposes the task, dispatches workers, synthesizes results. The pattern for complex multi-file/multi-source work — and the closest named pattern to what Sentinel's pipeline does.
When: evaluation is easier than generation (usually true — recognizing a flaw is easier than avoiding all flaws): a generator produces, an evaluator critiques against criteria, loop until accept or budget exhausted. Sentinel's adversarial reviewer is a named instance of this.
6Multi-agent patterns — and their honest costs
Split into multiple agents only when one context window can't hold the whole job, or roles genuinely conflict:
- Supervisor / sub-agents: a lead agent decomposes and delegates; workers run in their own context windows and return summaries. The real win is context isolation — a worker can burn 100k tokens exploring and return the 500 that matter — plus parallelism. This is orchestrator–workers where workers are full agent loops.
- Adversarial pairing: generator vs critic as separate agents. The separation matters: a model reviewing its own in-context output inherits its own framing and misses its own blind spots; a fresh-context reviewer whose prompt's job is to find problems is a genuinely different distribution. You built exactly this — explain why the separation matters, not just that it exists.
- Role/debate ensembles: multiple perspectives argue, a judge aggregates — occasionally worth it for high-stakes judgment; often just 3× cost.
7Human-in-the-loop design
The question underneath: which actions is the system allowed to take without a human?
- Approval gates at irreversibility boundaries. Classify actions: reversible + low blast radius → autonomous (reading logs, drafting, opening a PR — revertable); irreversible or high blast radius → gated (merging to prod, spending money, sending external messages, deleting data). Sentinel's Telegram gate sits exactly at the merge boundary — the last reversible moment. Framing gate placement as "find the irreversibility boundary" is the senior articulation.
- Make approval real, not rubber-stamp theater. An approver shown a wall of JSON approves blind. The approval surface must present: what will change, the evidence/reasoning, the risk level, an easy diff. Design the approval UX as carefully as the agent — otherwise you've automated the generation of unread checkboxes. 100+ PRs/month through a gate only works because review cost per item is kept low; that's a design achievement worth describing.
- Escalation as a tool: give the agent an
ask_human(question, context)tool so uncertainty has a legitimate exit — reduces forced guessing, and what gets escalated becomes a signal for where the agent needs improvement. - Graduated autonomy: start everything gated; expand autonomy per action-class as eval + track-record justify it; keep audit logs of every action + approval. Autonomy is earned by data, not granted by optimism.
- Attention economics (the deep point): human review capacity is the scarce resource. If the agent generates more decisions than humans can genuinely review, you've moved the bottleneck, not removed it. Triage (auto-approve trivial classes, gate risky ones), batch, and measure review quality, not just throughput.
8Guardrails, sandboxing & the lethal trifecta
Defense-in-depth layers, inside-out:
| Layer | Strength | What it does |
|---|---|---|
| 1 · Prompt-level | weakest | System-prompt rules, instruction hierarchy. Necessary, never sufficient — treat as UX, not security. |
| 2 · Tool-boundary | strongest, deterministic | The agent physically can't do what no tool allows. Least-privilege tools — read-only credentials for analysis agents; scoped tokens (this repo, not the org); allowlists (SELECT-only SQL; fetchable domains). Code, not prompts, is where "can't" lives. |
| 3 · Execution sandboxing | strong | Anything running generated code gets a disposable container/VM — no prod credentials, egress-filtered network, resource limits, killed after use. Agent-generated code is untrusted code by definition. |
| 4 · Output filtering | medium | Schema-check structured outputs; scan outbound content (secrets/PII redaction — an agent with log access can see secrets; don't let them exit via a PR description); policy checks on proposed actions before execution. |
| 5 · Rate/budget limits | medium | Cap tokens, tool calls, wall-clock, and spend per task — turns a runaway loop from an incident into a truncated run. |
| 6 · Audit + human gates | backstop | §7 — approval at irreversibility boundaries, audit logs of every action. |
The lethal trifecta (Simon Willison's framing — cite it): an agent combining (a) access to private data, (b) exposure to untrusted input, and (c) an exfiltration channel (network egress, message-sending) is exploitable — remove or gate at least one leg. Sentinel reads untrusted content (issues, logs) and has repo access — which is precisely why write-actions terminate at review + human approval. Present your own architecture as the mitigation.
9Agent failure modes — the field guide
| Failure | What it looks like / cause | Fixes |
|---|---|---|
| Infinite / unproductive loops | Retrying a failing tool identically; oscillating between two states; "let me try again" forever. Cause: no progress signal, error results re-fed verbatim. | Hard caps (iterations, tokens, cost, wall-clock); loop detection (same call + same args twice → intervene); escalate-to-human as the loop-breaker. |
| Context pollution / poisoning | Transcript fills with huge tool dumps, stale errors, dead-end explorations; quality visibly decays late in a long run. One poisoned observation (a misleading error) can anchor all later reasoning — early wrong beliefs compound. | Summarize/truncate tool outputs at the source (the 20 relevant log lines, not 5,000); compaction (summarize history, drop superseded results); sub-agents to quarantine exploratory mess; fresh context per phase; evaluator checkpoints. |
| Cascading errors | Step 3's small mistake (wrong file identified) silently invalidates steps 4–9 — the agent builds confidently on a false premise. | Validate at stage boundaries (file exists? patch applies? tests pass?) — deterministic checks between probabilistic steps; checkpoints so recovery ≠ restart; evaluator passes where errors compound most. |
| Wrong / hallucinated tool calls | Calling a tool that doesn't exist, or plausible-but-wrong arguments. | Tighter descriptions, fewer overlapping tools, schema validation with corrective error messages. |
| Goal drift / scope creep | Asked to fix a bug, the agent refactors the module. | Explicit scope constraints in task framing; diff-size budgets; reviewer checks "does this change match the ticket?" — your adversarial reviewer can literally enforce this. |
| Premature success claims | "Done!" without verification. | Define done as a checkable condition (tests pass, endpoint responds) and make the agent run the check — never trust self-reported completion. |
| Sycophantic self-evaluation | The generator grading its own work grades generously. | Evaluators get fresh context and an adversarial charter. |
10Rapid-fire interview Q&A
Q: Workflow vs agent — how do you choose?
Least autonomy that solves the problem. Enumerable steps → workflow: predictable, testable, cheap. Path-depends-on-findings → agentic loop, wrapped in budgets and checks. Production systems are usually workflows with agentic steps — mine is: fixed pipeline, agentic log-analysis stage.
Q: Explain MCP like I've never seen it.
A standard protocol between LLM apps and integrations — LSP for tools. Hosts (the LLM app) run clients that connect 1:1 to servers; servers expose tools (model-invoked actions), resources (app-loaded data), and prompts (user-invoked templates) over JSON-RPC via stdio or HTTP. Solves M×N: write one GitHub server, every MCP host can use it. I run MCP servers in production for log analysis — the win is decoupling: swap/upgrade/permission integrations without touching agent code, and one mediation point for auth and audit.
Q: How do you stop an agent from doing something destructive?
Layered: least-privilege tools so destructive actions are impossible by construction; sandboxed execution for generated code; deterministic validation between stages; budgets to bound runaways; and human approval gates at irreversibility boundaries — in my system, nothing merges without a human tap, by design not policy.
Q: Your agent works in demos, fails in prod — where do you look?
Traces first (full transcript: every tool call, args, result). Usual suspects in order: context pollution in long runs (huge tool dumps), tool-selection confusion (overlapping/vague descriptions), cascading early errors (add stage-boundary validation), and input-distribution drift from the demo set (fix the evals, not just the bug).
Q: When would you go multi-agent?
Two honest reasons: context isolation — workers burn tokens exploring and return distilled summaries — and genuine role separation, like generator vs adversarial critic, where fresh context removes self-review bias. Not for anthropomorphic org charts; every added agent is distributed-systems overhead with lossy natural-language interfaces.
Self-test
Name the five Anthropic patterns and one use-case each.
Prompt chaining (draft → critique → revise), routing (classifier sends inputs to specialized paths/models), parallelization (sectioning for latency, voting for confidence), orchestrator–workers (dynamic decomposition of multi-file work), evaluator–optimizer (generator + critic loop until accept — Sentinel's adversarial reviewer).
What are MCP's three roles and three primitives — and what's the "who decides" trichotomy?
Roles: host (LLM app — owns conversation, permissions, merged tool list), client (protocol connector, 1:1 per server), server (exposes a system's capabilities, host-agnostic). Primitives: tools (the model decides), resources (the application decides), prompts (the user decides). Transports: stdio and streamable HTTP, JSON-RPC 2.0.
State the lethal trifecta. Which leg does Sentinel cut?
Private-data access + untrusted input + an exfiltration channel = exploitable; remove or gate one leg. Sentinel reads untrusted issues/logs and has repo access, so all write-actions terminate at adversarial review + human approval — the consequential-action leg is gated.
Why does the adversarial reviewer get a fresh context instead of the generator reviewing itself?
A model reviewing its own in-context output inherits its own framing and blind spots, and self-evaluation is sycophantic. A fresh-context reviewer with an adversarial charter samples a genuinely different distribution — that's why evaluator–optimizer works: recognizing a flaw is easier than avoiding all flaws.
Where do you place a human approval gate, and what makes approval "real"?
At the irreversibility boundary — the last reversible moment (Sentinel: merge). Reversible/low-blast-radius actions run autonomously; irreversible ones are gated. Real approval means the surface shows what changes, the evidence, the risk, an easy diff — and review capacity is treated as the scarce resource (triage, batch, measure review quality).
An agent keeps calling the same failing tool. Name the failure mode and three fixes.
Unproductive loop: no progress signal, error re-fed verbatim. Fixes: hard caps (iterations/tokens/cost/wall-clock), loop detection (identical call + args twice → intervene), actionable error messages so it can self-correct, and escalate-to-human as the loop-breaker.