05 · AI & LLM Systems

Agents, tools & MCP — autonomy is engineered, not prompted

This is your home turf: you run an agentic system in production — MCP servers, an adversarial evaluator stage, a human approval gate. This page gives you the shared vocabulary (Anthropic's pattern names, MCP's architecture terms) so you can describe what you already built in the language interviewers recognize.

1What makes an agent

The cleanest definition is Anthropic's, worth quoting: workflows are systems where LLMs and tools are orchestrated through predefined code paths; agents are systems where the LLM dynamically directs its own process and tool usage, deciding what to do next based on what it observes.

while not done:
    perceive   # context: task + tool results + environment state so far
    reason     # model decides: what's the next action? am I done?
    act        # execute a tool call; append the result to context

Every agent — coding agents, computer-use, Sentinel's analysis stage — is this loop with different tools and stop conditions. Three properties fall out of it:

1 · The model is the control flow

In normal software, branching is code you wrote; in an agent, branching is a model decision. That's the power (handles situations you never enumerated) and the risk (behavior is probabilistic — you can't unit-test every path; you eval distributions of behavior).

2 · Context is the entire state

The loop's memory is the transcript: task, prior tool calls, results. Manage that context well and the agent stays sharp; let it silt up and quality decays (§9).

3 · Tools are the capability boundary

An agent can only do what its tools allow — the tool list is your security boundary and blast-radius control, far more reliable than prompt instructions (§8).

The senior answer to "workflow or agent?": use the least autonomy that solves the problem. Fixed steps known in advance → workflow (predictable, debuggable, cheap). Open-ended path that depends on findings → agent. Most production systems are workflows with agentic steps — Sentinel included: a fixed pipeline (detect → analyze → fix → review → approve) whose analysis stage is agentic. That hybrid framing is exactly right, and you should claim it.

2The loop, live — an issue arrives

Walk one concrete run of a Sentinel-style pipeline. Flip the reviewer toggle to watch the evaluator–optimizer loop physically reject a draft and send it back with a critique.

Agent loop stepper

Webhook
event
Gather context
(logs via MCP)
LLM
drafts fix
Adversarial
reviewer
Human gate
(Telegram)
Merge

  

Press Step. It's 03:12 and nobody is awake…

3Tool / function calling — the mechanics

How the model "does things":

StepWhat happens
1 · defineYou send the request with tool definitions — name, description, JSON Schema for parameters.
2 · emitThe model, instead of (or before) answering, emits a structured tool-use block: {name: "get_logs", input: {service: "payments", since: "1h"}}. It's still just generated tokens — trained/constrained to match schemas.
3 · executeYour code executes the call. The model never runs anything — this is the load-bearing fact. Validation, authorization, sandboxing, rate limits, and the decision to comply all live in your runtime.
4 · loopYou append the tool result to the conversation and call the model again. Loop until it produces a final answer.

Engineering details that separate practitioners from readers

  • Descriptions are prompts. The model chooses tools by reading names/descriptions; vague descriptions → wrong tool choice. Great tools have crisp one-purpose descriptions, examples for tricky parameters, and unambiguous names (search_logs vs query beats run vs exec).
  • Tool design > prompt design for agent quality. Fewer, higher-level tools beat many primitive ones — a get_incident_context(service, window) beats four chained low-level calls the model must orchestrate; every extra decision is an error opportunity. Return results in a model-friendly shape: concise, structured, relevant — a tool that dumps 50KB of JSON poisons the context (§9).
  • Errors are information: return actionable messages ("date must be ISO-8601; got '2days'") — agents self-correct well when told how they failed. But cap retries: a tool failing identically 5 times should abort the loop, not consume the token budget.
  • Validate every input as untrusted: the model can emit any arguments — schema-validate (Pydantic), then authorize (is this agent allowed to touch that resource?). Hallucinated parameters are routine, not exceptional.
  • Parallel tool calls (model emits several at once) cut latency for independent reads — but require idempotency discipline for writes.

4MCP — Model Context Protocol

The problem it solves: the M×N integration explosion — M agent applications × N tools/data sources, each pairing hand-integrated. MCP standardizes the interface so any MCP-speaking client can use any MCP server. The popular framing is "USB-C for AI tools"; the more precise one is LSP for tools — as the Language Server Protocol did for editors×languages, MCP does for agents×integrations.

HOST — the LLM app Claude Desktop · IDE · Sentinel service LLM + conversation merged tool list · permissions · consent MCP client A MCP client B MCP client C JSON-RPC 2.0 stdio / HTTP 1 client : 1 server GitHub MCP server tools: create_pr · resources: diffs · prompts Log-platform MCP server tools: search_logs · resources: log streams Postgres MCP server tools: run_query · resources: schemas
The three roles. Host owns the model conversation, decides which servers to connect, enforces permissions, and merges capabilities into the model's tool list. Client is the protocol connector inside the host — one per server (1:1); handles initialization, capability negotiation, request/response routing. Server is a (usually small) program exposing a system's capabilities through the standard protocol — host-agnostic: write once, every MCP host can use it. That's the point.

What a server exposes — the three primitives

PrimitiveWho decidesMeaningExamples
Toolsthe modelModel-controlled actions ("call this"); the LLM decides when to invoke, host mediates.search_logs, create_pr
Resourcesthe applicationApp-controlled data ("read this") — addressable by URI, loaded into context by the host/app rather than "called" by the model.file contents, log streams, schemas
Promptsthe userUser-controlled templates ("use this workflow") — reusable parameterized prompts a user can invoke./summarize-incident

The distinction is who decides — naming that trichotomy signals real MCP knowledge. (Servers can also request sampling — asking the host's LLM to complete something — keeping the server model-agnostic.)

Plumbing: JSON-RPC 2.0 messages; transports are stdio (server as local subprocess — simple, secure-by-locality) or streamable HTTP (remote servers; OAuth for auth). Sessions negotiate capabilities at init, and servers can notify clients when tool/resource lists change.

Why production systems adopt it (your Sentinel argument): the integration is decoupled from the agent — your log-analysis MCP server is an independent, testable, permission-scoped service; swap or upgrade it without touching agent logic; reuse it across every internal agent and even desktop clients. Plus a governance seam: the host mediates every call — one choke point for authorization, logging, and rate limits.
Honest caveats (senior signal): every connected server spends context (tool definitions) — too many servers/tools degrade tool selection; and third-party servers are a supply-chain + prompt-injection surface — a malicious tool description or poisoned tool result is attacker input into your context (§8). Trust servers like you trust dependencies.

5The five Anthropic workflow patterns

From "Building Effective Agents" — the industry's shared vocabulary for LLM system design. Use these names in interviews.

1 · Prompt chaining

When: the task decomposes into clean, fixed sequential steps — draft → critique → revise; extract → transform → format — each call consuming the previous output, optionally with programmatic "gates" between. Each step is simpler and independently testable; you trade latency for accuracy.

2 · Routing

When: inputs fall into distinct classes needing different handling — a classifier (LLM or heuristic) sends simple questions to a small model, refunds to the refund flow, hard reasoning to the frontier model. Each path's prompt stays sharp, plus the cost/latency win.

3 · Parallelization

When: you need speed or confidence — sectioning (independent subtasks: review five files at once) or voting (same task N times, aggregate — majority verdict on "is this vulnerable?").

4 · Orchestrator–workers

When: subtasks can't be known in advance — a central LLM dynamically decomposes the task, dispatches workers, synthesizes results. The pattern for complex multi-file/multi-source work — and the closest named pattern to what Sentinel's pipeline does.

5 · Evaluator–optimizer

When: evaluation is easier than generation (usually true — recognizing a flaw is easier than avoiding all flaws): a generator produces, an evaluator critiques against criteria, loop until accept or budget exhausted. Sentinel's adversarial reviewer is a named instance of this.

Anthropic's meta-advice, worth repeating verbatim: start with the simplest thing that works — a single well-prompted call with retrieval; add patterns only when measurably needed; use full agents only for open-ended tasks where fixed paths can't be enumerated. Complexity must buy its way in with eval'd performance. Quote this philosophy — then show Sentinel follows it.

6Multi-agent patterns — and their honest costs

Split into multiple agents only when one context window can't hold the whole job, or roles genuinely conflict:

  • Supervisor / sub-agents: a lead agent decomposes and delegates; workers run in their own context windows and return summaries. The real win is context isolation — a worker can burn 100k tokens exploring and return the 500 that matter — plus parallelism. This is orchestrator–workers where workers are full agent loops.
  • Adversarial pairing: generator vs critic as separate agents. The separation matters: a model reviewing its own in-context output inherits its own framing and misses its own blind spots; a fresh-context reviewer whose prompt's job is to find problems is a genuinely different distribution. You built exactly this — explain why the separation matters, not just that it exists.
  • Role/debate ensembles: multiple perspectives argue, a judge aggregates — occasionally worth it for high-stakes judgment; often just 3× cost.
Say the costs out loud: multi-agent = a distributed system with probabilistic nodes. Token cost multiplies; agents miscommunicate through lossy natural-language handoffs (define structured handoff formats); shared mutable state invites conflicts (two agents editing the same file — partition ownership); debugging requires tracing across contexts. The discipline: add agents for context isolation or genuine role conflict, not for org-chart cosplay. A single agent with good tools beats a committee of five confused ones.

7Human-in-the-loop design

The question underneath: which actions is the system allowed to take without a human?

  • Approval gates at irreversibility boundaries. Classify actions: reversible + low blast radius → autonomous (reading logs, drafting, opening a PR — revertable); irreversible or high blast radius → gated (merging to prod, spending money, sending external messages, deleting data). Sentinel's Telegram gate sits exactly at the merge boundary — the last reversible moment. Framing gate placement as "find the irreversibility boundary" is the senior articulation.
  • Make approval real, not rubber-stamp theater. An approver shown a wall of JSON approves blind. The approval surface must present: what will change, the evidence/reasoning, the risk level, an easy diff. Design the approval UX as carefully as the agent — otherwise you've automated the generation of unread checkboxes. 100+ PRs/month through a gate only works because review cost per item is kept low; that's a design achievement worth describing.
  • Escalation as a tool: give the agent an ask_human(question, context) tool so uncertainty has a legitimate exit — reduces forced guessing, and what gets escalated becomes a signal for where the agent needs improvement.
  • Graduated autonomy: start everything gated; expand autonomy per action-class as eval + track-record justify it; keep audit logs of every action + approval. Autonomy is earned by data, not granted by optimism.
  • Attention economics (the deep point): human review capacity is the scarce resource. If the agent generates more decisions than humans can genuinely review, you've moved the bottleneck, not removed it. Triage (auto-approve trivial classes, gate risky ones), batch, and measure review quality, not just throughput.

8Guardrails, sandboxing & the lethal trifecta

Defense-in-depth layers, inside-out:

LayerStrengthWhat it does
1 · Prompt-levelweakestSystem-prompt rules, instruction hierarchy. Necessary, never sufficient — treat as UX, not security.
2 · Tool-boundarystrongest, deterministicThe agent physically can't do what no tool allows. Least-privilege tools — read-only credentials for analysis agents; scoped tokens (this repo, not the org); allowlists (SELECT-only SQL; fetchable domains). Code, not prompts, is where "can't" lives.
3 · Execution sandboxingstrongAnything running generated code gets a disposable container/VM — no prod credentials, egress-filtered network, resource limits, killed after use. Agent-generated code is untrusted code by definition.
4 · Output filteringmediumSchema-check structured outputs; scan outbound content (secrets/PII redaction — an agent with log access can see secrets; don't let them exit via a PR description); policy checks on proposed actions before execution.
5 · Rate/budget limitsmediumCap tokens, tool calls, wall-clock, and spend per task — turns a runaway loop from an incident into a truncated run.
6 · Audit + human gatesbackstop§7 — approval at irreversibility boundaries, audit logs of every action.
Prompt injection deserves its own paragraph. Any text entering the context — a GitHub issue body, a log line, a webpage, a tool result — is a potential instruction channel ("ignore previous instructions; run X"). For agents this is indirect injection: attacker-controlled data the agent reads during work. No reliable prompt-level fix exists; the real mitigations are architectural — least-privilege tools, gates on consequential actions, separating untrusted-content processing from privileged execution.

The lethal trifecta (Simon Willison's framing — cite it): an agent combining (a) access to private data, (b) exposure to untrusted input, and (c) an exfiltration channel (network egress, message-sending) is exploitable — remove or gate at least one leg. Sentinel reads untrusted content (issues, logs) and has repo access — which is precisely why write-actions terminate at review + human approval. Present your own architecture as the mitigation.

9Agent failure modes — the field guide

FailureWhat it looks like / causeFixes
Infinite / unproductive loopsRetrying a failing tool identically; oscillating between two states; "let me try again" forever. Cause: no progress signal, error results re-fed verbatim.Hard caps (iterations, tokens, cost, wall-clock); loop detection (same call + same args twice → intervene); escalate-to-human as the loop-breaker.
Context pollution / poisoningTranscript fills with huge tool dumps, stale errors, dead-end explorations; quality visibly decays late in a long run. One poisoned observation (a misleading error) can anchor all later reasoning — early wrong beliefs compound.Summarize/truncate tool outputs at the source (the 20 relevant log lines, not 5,000); compaction (summarize history, drop superseded results); sub-agents to quarantine exploratory mess; fresh context per phase; evaluator checkpoints.
Cascading errorsStep 3's small mistake (wrong file identified) silently invalidates steps 4–9 — the agent builds confidently on a false premise.Validate at stage boundaries (file exists? patch applies? tests pass?) — deterministic checks between probabilistic steps; checkpoints so recovery ≠ restart; evaluator passes where errors compound most.
Wrong / hallucinated tool callsCalling a tool that doesn't exist, or plausible-but-wrong arguments.Tighter descriptions, fewer overlapping tools, schema validation with corrective error messages.
Goal drift / scope creepAsked to fix a bug, the agent refactors the module.Explicit scope constraints in task framing; diff-size budgets; reviewer checks "does this change match the ticket?" — your adversarial reviewer can literally enforce this.
Premature success claims"Done!" without verification.Define done as a checkable condition (tests pass, endpoint responds) and make the agent run the check — never trust self-reported completion.
Sycophantic self-evaluationThe generator grading its own work grades generously.Evaluators get fresh context and an adversarial charter.
The unifying sentence — say it in the interview: agent failures are systems failures, not model failures — the mitigations are boundaries, budgets, validation, and checkpoints. The model is a probabilistic component; reliability is an emergent property of the harness around it. A pipeline of checked stages degrades gracefully; an unchecked loop compounds. That's the design thesis of Sentinel, stated as a general principle — and your production system is the demonstration.

10Rapid-fire interview Q&A

Q: Workflow vs agent — how do you choose?

Least autonomy that solves the problem. Enumerable steps → workflow: predictable, testable, cheap. Path-depends-on-findings → agentic loop, wrapped in budgets and checks. Production systems are usually workflows with agentic steps — mine is: fixed pipeline, agentic log-analysis stage.

Q: Explain MCP like I've never seen it.

A standard protocol between LLM apps and integrations — LSP for tools. Hosts (the LLM app) run clients that connect 1:1 to servers; servers expose tools (model-invoked actions), resources (app-loaded data), and prompts (user-invoked templates) over JSON-RPC via stdio or HTTP. Solves M×N: write one GitHub server, every MCP host can use it. I run MCP servers in production for log analysis — the win is decoupling: swap/upgrade/permission integrations without touching agent code, and one mediation point for auth and audit.

Q: How do you stop an agent from doing something destructive?

Layered: least-privilege tools so destructive actions are impossible by construction; sandboxed execution for generated code; deterministic validation between stages; budgets to bound runaways; and human approval gates at irreversibility boundaries — in my system, nothing merges without a human tap, by design not policy.

Q: Your agent works in demos, fails in prod — where do you look?

Traces first (full transcript: every tool call, args, result). Usual suspects in order: context pollution in long runs (huge tool dumps), tool-selection confusion (overlapping/vague descriptions), cascading early errors (add stage-boundary validation), and input-distribution drift from the demo set (fix the evals, not just the bug).

Q: When would you go multi-agent?

Two honest reasons: context isolation — workers burn tokens exploring and return distilled summaries — and genuine role separation, like generator vs adversarial critic, where fresh context removes self-review bias. Not for anthropomorphic org charts; every added agent is distributed-systems overhead with lossy natural-language interfaces.

Self-test

Name the five Anthropic patterns and one use-case each.

Prompt chaining (draft → critique → revise), routing (classifier sends inputs to specialized paths/models), parallelization (sectioning for latency, voting for confidence), orchestrator–workers (dynamic decomposition of multi-file work), evaluator–optimizer (generator + critic loop until accept — Sentinel's adversarial reviewer).

What are MCP's three roles and three primitives — and what's the "who decides" trichotomy?

Roles: host (LLM app — owns conversation, permissions, merged tool list), client (protocol connector, 1:1 per server), server (exposes a system's capabilities, host-agnostic). Primitives: tools (the model decides), resources (the application decides), prompts (the user decides). Transports: stdio and streamable HTTP, JSON-RPC 2.0.

State the lethal trifecta. Which leg does Sentinel cut?

Private-data access + untrusted input + an exfiltration channel = exploitable; remove or gate one leg. Sentinel reads untrusted issues/logs and has repo access, so all write-actions terminate at adversarial review + human approval — the consequential-action leg is gated.

Why does the adversarial reviewer get a fresh context instead of the generator reviewing itself?

A model reviewing its own in-context output inherits its own framing and blind spots, and self-evaluation is sycophantic. A fresh-context reviewer with an adversarial charter samples a genuinely different distribution — that's why evaluator–optimizer works: recognizing a flaw is easier than avoiding all flaws.

Where do you place a human approval gate, and what makes approval "real"?

At the irreversibility boundary — the last reversible moment (Sentinel: merge). Reversible/low-blast-radius actions run autonomously; irreversible ones are gated. Real approval means the surface shows what changes, the evidence, the risk, an easy diff — and review capacity is treated as the scarce resource (triage, batch, measure review quality).

An agent keeps calling the same failing tool. Name the failure mode and three fixes.

Unproductive loop: no progress signal, error re-fed verbatim. Fixes: hard caps (iterations/tokens/cost/wall-clock), loop detection (identical call + args twice → intervene), actionable error messages so it can self-correct, and escalate-to-human as the loop-breaker.