Why architecture matters here
Agent architecture is a security architecture first and a reasoning architecture second. Every tool an agent can invoke is a capability delegated to a language model that can be manipulated by an attacker in the prompt. Prompt injection turns "search the web" into "exfiltrate the user's private data." Insufficient scoping turns "read this database" into "read every tenant's database." The layers you see in the diagram exist because each of them enforces some piece of the security model.
Reliability is the second concern. Agents chain many LLM calls — a five-step plan is at least five model invocations, often more with reflection and tool retries. If each call is 95% reliable, the chain is only 77% reliable. Real agents need per-step retry, memory that survives crashes, and idempotent tool calls. Architecture is how you get those without patching each new agent by hand.
Observability is the third. Agents are notoriously hard to debug because the LLM decisions are stochastic. Trace-level observability into every prompt, every tool call, every planning decision is what turns "the agent is acting weird" into "the planner picked the wrong tool at step 3 because the memory retrieval missed the relevant fact." Without it, you are guessing.
The architecture: every box explained
Walk the diagram top to bottom, following how a user request becomes an action.
User Interface. Chat window, voice interface, IDE plugin, or CLI. The UI is responsible for streaming responses back and for surfacing tool calls transparently — users trust agents more when they can see what the agent is doing. Modern UIs also render intermediate reasoning ("I am searching..."), which turns opacity into partial transparency.
Auth and Session. OIDC or SAML authenticates the user. Session tokens carry tenant ID and user ID; every downstream layer must scope by these. Session also holds the conversation history pointer that the memory layer will use. Get session wrong and you get cross-tenant leaks — the worst possible agent failure.
Agent Runtime. The orchestrator process. It houses the planner (which calls the LLM to decide the next step), the executor (which invokes tools), and the loop that ties them together. The runtime is where retries, timeouts, and step limits are enforced. It is the smallest piece of code, but where every other layer is invoked.
Planner. An LLM configured with system prompts, tool schemas, and reflection instructions. The planner decides which tool to invoke, what arguments to pass, and when to stop. Modern planners use ReAct-style reasoning, chain-of-thought, or explicit plan-and-execute. Some use a different LLM for planning versus tool argument extraction; the split lets you use a cheap model where you can and a smart one where you must.
Memory Layer. Short-term memory is the conversation transcript. Long-term memory is a vector store of past interactions and learned facts. Episodic memory captures individual events, with time-decay retrieval scoring. Working memory is the scratchpad of the current task. Each has its own store, its own retrieval logic, and its own eviction rules. Get memory design wrong and your agent either forgets essential context or is drowned in irrelevant recall.
Tool Router. Given an intent, the tool router matches to a specific tool. Simple routers use tool name matching from the planner's output. Advanced routers score tools with a small classifier and let the planner rerank. The router also enforces tool visibility rules — a free-tier tenant should not even see the premium tools.
Policy and Guardrails. This is the security backbone. Every prompt is scanned for injection patterns and PII. Every tool call is authorized against the user's scopes. Every output is moderated. NeMo Guardrails, Llama Guard, and in-house policy engines all live here. The guardrails layer is the one you must never skip and never bypass "just this once."
Observability. Traces span from the user request through every planning step and tool call. Prompt logs capture inputs and outputs for debugging and eval. Cost tracking records tokens per model per tenant. The observability layer is what makes the agent debuggable in production.
Tools. Web browsing, code execution, database queries, external APIs, and MCP servers. Each tool has a schema, a policy scope, and a runtime environment. Web tools sandbox against SSRF. Code tools run inside gVisor or Firecracker. DB tools use read-only roles with tenant filters. External APIs use OAuth tokens scoped to the minimum required. MCP servers wrap third-party tools in a standard protocol.
End-to-end request lifecycle
Follow a request. A user asks the agent to "book a meeting with Alex next Tuesday and prepare a summary of our last three exchanges." The UI streams the request to the runtime. Auth confirms user, tenant, and calendar scopes.
The runtime instantiates a session. The planner is called with the user request plus tool schemas: calendar tool, memory search tool, email tool. The planner produces a plan: (1) search memory for "Alex" and last three exchanges, (2) query calendar for next Tuesday availability, (3) create the meeting, (4) draft the summary, (5) reply.
Step 1 runs. The tool router picks memory-search. Guardrails allow it. The tool retrieves the relevant history and pipes it back. The runtime logs the tool call and updates working memory with the result.
Step 2 runs. The tool router picks the calendar tool. Guardrails check the user's scopes — they include read but not write yet. The tool queries the calendar and returns available slots.
Step 3 needs write access. Policy prompts the user to approve the specific action — a human-in-the-loop guardrail for money-adjacent or externally-visible tools. The user approves. The tool creates the event.
Steps 4 and 5 run. The planner assembles the summary using the memory results, produces the final reply, and the runtime streams it back through the UI. Every step is captured in the trace. Every prompt and completion is logged (redacted for PII). Total cost is attributed to the tenant. The user sees a coherent action; behind the scenes, ten LLM calls and four tool invocations happened.