Agentic AI in production: architecture, guardrails, and evaluation
Most teams reaching for agents don't need one. A field guide to agent architecture, guardrails and evals — and the production maths that decide if it ships.
AMDIM · Engineering
June 12, 2026
We get the same request a lot lately: "we want to build an agent." Our first question tends to annoy people: do you actually need one?
Most of the time, the honest answer is no. A workflow would be faster, cheaper, and far easier to debug — and you'd ship it this quarter instead of next year. The production data agrees, in fact. Gartner, after polling more than 3,400 organisations, expects over 40% of agentic-AI projects to be cancelled by the end of 2027 — done in by escalating cost, fuzzy value, and weak risk controls.
That's not an argument against agents. It's an argument for building them on purpose. An autonomous agent is a trade — you hand the model control over its own steps, and in return you give up determinism, predictable cost, and easy debugging. Sometimes that trade is worth it. Often it isn't. And the engineering decisions that decide whether your agent survives contact with production all get made early, in the architecture.
So this is a field guide to those decisions: what actually counts as "agentic," when to reach for an agent instead of a workflow, the patterns that hold up, how to guardrail against prompt injection, how to evaluate something that won't sit still — and the production maths nobody mentions in the demo.
The gist: climb the ladder of autonomy only as far as the task forces you to. Treat prompt injection as a property to design around, not a bug to patch. Evaluate for reliability across repeated runs, not one lucky pass. And budget for ~15× the tokens of a chat plus errors that compound across steps.
What actually counts as "agentic"?
The cleanest definition still comes from Anthropic's Building Effective Agents. Workflows orchestrate LLMs and tools "through predefined code paths." Agents are "systems where LLMs dynamically direct their own processes and tool usage." The difference is simply who decides the next step — your code, or the model at runtime.
It helps to stop thinking of "agent" as a yes/no and start thinking of a ladder. Each rung adds capability and takes away a little of your control.
exhibit
Four things separate a real agent from a RAG pipeline: it plans (breaks a goal into steps), it uses tools (acts on the world through APIs), it remembers (carries state across steps), and it loops (observes the result, adjusts, keeps going). RAG is reactive — it answers a question. An agent is goal-directed — it chases an outcome. Chip Huyen sums it up nicely in AI Engineering: an agent is "a system with APIs, state, plans, monitoring, and failure recovery." Note how little of that is "the model."
When do you actually need an agent?
Here's the decision that prevents most of those cancelled projects. Reach for an agent only when the task is genuinely open-ended — when you can't predict how many steps it'll take and can't hardcode the path. For everything else, a workflow wins on cost, speed, and the ability to actually debug it at 2am.
And "everything else" is a bigger category than people expect. Before you reach for autonomy, Anthropic catalogues five workflow patterns that quietly solve most "AI feature" problems with boring, predictable code:
- Prompt chaining — fixed sequential steps, each an LLM call, for tasks that cleanly decompose.
- Routing — classify the input, send it to a specialised prompt or model.
- Parallelisation — run subtasks at once, or vote across runs for confidence.
- Orchestrator–workers — a lead model splits the work and delegates, when the subtasks aren't known up front.
- Evaluator–optimiser — one model generates, another critiques, in a loop with clear criteria.
The most expensive architectural mistake we see in 2026 is reaching for an autonomous agent where a workflow would have shipped. Most production value still lives in deterministic workflows — they're just less exciting to talk about.
If you remember Anthropic's one-liner — "find the simplest solution possible, and only increase complexity when needed" — you'll avoid most of the graveyard.
The patterns that hold up
When the task genuinely warrants an agent, a handful of patterns have earned their place.
The loop. The default is ReAct (Yao et al., 2022): interleave a reasoning trace with a tool call, observe, repeat. It's robust, but it reasons step by step with no global plan, so it tends to wander on long tasks. Plan-and-execute writes the whole plan up front and then runs it — cheaper, fewer LLM calls, but brittle the moment reality diverges from the plan. Reflection adds a self-critique-and-retry step, which buys quality at the price of more tokens and latency. Most production systems end up as some pragmatic blend.
Orchestration. Start with a single agent. Its tool-selection accuracy starts to fall apart somewhere past ten tools — and when it does, that's your signal to move to a supervisor: a router that delegates to specialist workers and stitches their results together. It's easier to reason about and debug, though it isn't free; community measurements put the cost at roughly 1.5–2× the latency and 2–3× the tokens. A peer-to-peer swarm is a further step you take only when the data says latency is your bottleneck and your agents rarely misroute. Don't start there because a conference talk made it look cool.
Context engineering has more or less replaced "prompt engineering" as the thing that decides whether a long-running agent holds together. Three techniques from Anthropic's work are worth designing in from day one: compaction (summarise and reset as you approach the context-window limit), structured note-taking (the agent writes to an external scratchpad and reads it back — persistent memory at almost no token cost), and sub-agent isolation (subagents swallow the big, noisy tool outputs so the main agent only sees the clean result).
Guardrails start from an uncomfortable truth
Here's where production agents are won or lost, and it begins with something most teams don't want to hear: prompt injection isn't a bug you patch — it's a structural property of any system that mixes instructions and untrusted data. It has held the #1 spot in the OWASP Top 10 for LLM Applications for two editions running, and the 2025 OWASP list for agentic applications adds the agent-specific flavours: goal hijacking, tool misuse, memory poisoning, excessive permissions.
The mental model we keep coming back to is Simon Willison's lethal trifecta: an agent becomes an exfiltration risk the moment it combines all three of —
- access to private data,
- exposure to untrusted content, and
- the ability to communicate externally.
Knock out any one and the attack path closes. Meta turned this into a build rule with its "Agents Rule of Two": in a single session, an agent should have no more than two of those three — and if it genuinely needs all three, route it through a human. Willison called it "the best practical advice for building secure LLM-powered agent systems today," and we agree.
In practice that means designing for least privilege from the start, not bolting filters on afterwards:
# Least-privilege tool access: validate, scope, and gate side-effects.
ALLOWED_TABLES = {"orders", "shipments"} # explicit allow-list, never "*"
def query_db(sql: str, table: str) -> list[dict]:
if table not in ALLOWED_TABLES: # 1. allow-list, not free choice
raise PermissionError(f"table {table!r} not permitted")
return db.run(sql, read_only=True) # 2. read-only credential
def issue_refund(order_id: str, amount: float, ctx: Session) -> Result:
if amount > ctx.auto_approve_limit: # 3. human gate on irreversible action
return ctx.escalate_to_human(order_id, amount)
return payments.refund(order_id, amount) # scoped, idempotent, audited
Nothing in there is clever. Validate inputs and outputs against schemas and allow-lists. Scope every tool credential to least privilege and sandbox anything that executes code. Put a human in front of irreversible actions. And keep the component that reads untrusted content well away from the one that holds secrets or can reach the outside world. All of it is cheaper to build in than to retrofit after an incident — and incidents here tend to make the news.
How do you evaluate something that won't sit still?
You can't ship an agent on vibes, and you can't test it like deterministic software either. The single most useful idea here comes from Sierra's τ-bench, which draws a sharp line between pass@k ("at least one of k attempts succeeds") and pass^k ("all k attempts succeed"). Production cares about pass^k — and because pass^k is just pᵏ, it falls off a cliff. On τ-bench's retail tasks, GPT-4o scored around 61% on a single try but its pass^8 dropped below 25%. A model that looks great once can be wildly unreliable across eight runs. And production is the eighth run.
So build the evaluation in layers. Start with task success rate — did it hit the goal. Add trajectory evaluation — score the path it took, the tool calls and intermediate decisions, against a reference, so you can see where it went wrong rather than just that it did. Layer in LLM-as-judge for the fuzzier qualities, but calibrate the judge against human labels, because it's non-deterministic too. Then run your curated cases offline as a regression suite before deploy, and score live traffic online to catch drift after. Tools like LangSmith, agentevals and Promptfoo make this routine.
One warning while you're here: be sceptical of public coding-agent leaderboard numbers. SWE-bench has documented contamination problems, so verify any score against the live leaderboard before you quote it in a deck.
The production maths that bite
Three numbers decide whether an agent is viable, and all three are invisible in a demo.
Cost. Agents are hungry. Anthropic's multi-agent research system burned roughly 15× the tokens of a single chat. The more useful finding from their post-mortem: token usage explained about 80% of the performance variance — more than tool calls or model choice combined. So the economics only work for genuinely high-value tasks, and the right unit to budget is cost per completed task, not cost per prompt.
Reliability. Errors don't add up — they multiply. Chain enough steps and per-step reliability turns against you fast:
| Per-step reliability | 5 steps | 10 steps | 20 steps |
|---|---|---|---|
| 95% | 77% | 60% | 36% |
| 99% | 95% | 90% | 82% |
At 95% a step — which feels perfectly fine — a 20-step task finishes end-to-end about a third of the time. And it's worse than the raw maths, because a model fed its own earlier mistakes compounds them. This is the real case for short horizons, checkpoints, and human gates: each one chops the exponent.
Observability. You can't debug what you can't see. The industry is settling on OpenTelemetry's GenAI conventions — a span tree of invoke_agent → chat → execute_tool with token and model attributes — so you can trace a full trajectory in the tools you already run. Pair that with deliberate model selection: a strong planner leading cheaper workers, frontier reasoning reserved for the planning, small fast models doing the routing and extraction.
Where it goes wrong
The failure modes are predictable, which is the good news — predictable means avoidable:
- reaching for an agent where a workflow would do (the number-one cause of cost and cancellations);
- piling more than ten tools onto a single agent instead of routing or splitting;
- optimising for pass@1 and shipping something that fails on repetition;
- wiring private data, untrusted content and external comms into one agent — the accidental lethal trifecta;
- runaway cost from recursive sub-agents and oversized tool outputs, with no budgets or circuit breakers;
- logging only final outputs, so failures are invisible;
- and plain old "agent washing" — Gartner reckons only about 130 of the thousands of self-described "agentic" vendors are the real thing.
The takeaway
If you keep one idea: an agent is a trade, not an upgrade. Start at the lowest rung that solves the problem and climb only when the task forces you. Budget for the tokens and the compounding errors. Treat prompt injection as a design constraint and apply the Rule of Two. Evaluate for pass^k, instrument every trajectory, and keep a human in front of the irreversible. Do that, and you're building the agent that ships — not the one that joins Gartner's 40%.
Frequently asked questions
What is the difference between an AI agent and a workflow? A workflow orchestrates LLMs and tools through code paths you write — the control flow is fixed and predictable. An agent lets the model decide its own steps and tool use at runtime. Use a workflow whenever the task is decomposable and predictable; reserve agents for open-ended tasks where you can't predict the number of steps.
How do you stop prompt injection in AI agents? You can't fully "fix" it — treat it as structural. Apply the lethal-trifecta principle: never let one agent combine access to private data, exposure to untrusted content, and the ability to communicate externally. Meta's Rule of Two limits an agent to two of those three per session and routes the rest through human approval. Add input/output validation, least-privilege and sandboxed tools, and approval gates on irreversible actions.
How do you evaluate an AI agent? Combine outcome metrics (task success rate), trajectory evaluation (scoring the path and tool calls against a reference), and calibrated LLM-as-judge scoring — run offline as a regression suite before deploy, and online to catch drift after. Crucially, measure pass^k (reliability across repeated attempts), not just pass@1, because success probability decays exponentially across runs.
Why do so many agentic AI projects fail? Gartner expects over 40% to be cancelled by the end of 2027, mostly from escalating cost, unclear value, and inadequate risk controls — often made worse by choosing an autonomous agent where a workflow would have done, over-tooling, ignoring the compounding-error maths, and "agent washing."
Thinking about putting an agent into production — and want to know whether your architecture, guardrails and evals are ready before you commit? That's the work we do, building agentic AI that's supervised, observable and safe to run. The Agent Readiness Scorecard is a quick way to find the gaps before you commit.
/ keep reading
Strategy · 10 min read
Pilot to production: why most AI stalls — and the operating model that ships it
Only ~48% of AI projects reach production. The blocker isn't the model — it's the operating model. Here's the playbook that ships.
Read the articleStrategy · 10 min read
AI ROI: a board-ready framework for funding AI that pays back
Enterprises spent $30–40B on GenAI pilots and 95% saw no P&L return. Here's how to fund the 5% that pay back.
Read the article