Why architecture matters here
LLM agent architecture matters because agents are LLMs that take actions in the world. Every action can succeed, fail, or misbehave in ways that affect users. Retry logic, budget caps, permission scopes, and structured outputs together are what prevent agents from becoming expensive random-action generators.
Cost is a first-order concern. A poorly bounded agent burns tokens in loops or on unnecessary tool calls. A well-designed one uses cheap models where it can, expensive ones only when needed, and tracks cost per session.
Reliability is where architecture separates good agents from bad. Reflection loops that recognize stuck states, tool registries that enforce schemas, and structured outputs that eliminate parse errors — each is a specific mechanism that turns "sometimes works" into "reliably works."
The architecture: every piece explained
Walk the diagram top to bottom.
User Goal. The task. Ideally structured (a form) or clearly written. Ambiguous goals lead to ambiguous plans.
Planner. An LLM configured to decompose goals into steps and pick tools. Uses ReAct pattern, plan-and-execute, or tree-of-thought. Small models often work fine for planning if the tool set is well-designed.
Working Memory. The scratchpad. Current step, sub-goals, intermediate results, tool call history. Fits in the model's context window; grows as the task progresses.
Tool Registry. Available tools with JSON schemas for inputs and outputs, descriptions, auth requirements, scope hints. The planner picks tools by matching against descriptions.
Executor + Retry. The layer that actually invokes tools. Handles authentication, retries on transient failures, and structured error responses back to the planner.
Long-term Memory. Vector store for semantic recall of past interactions. Episodic memory for specific events. Both are optional but transformative for agents with ongoing relationships.
Reflection Loop. Periodically the agent evaluates: is progress being made? Is a tool call needed? Am I stuck? Reflection is what prevents infinite tool loops.
Guardrails. Budget caps (max tokens, max tools, max wall-clock), policy checks (allowed vs disallowed actions), and human-in-the-loop for high-impact steps.
Streaming Output. Events to the client as they happen. Progress updates, partial reasoning, streaming tokens. Users trust agents that show their work.
Trace + Cost. Per-step observability. Every LLM call and every tool call is traced. Cost attributed per session, per feature, per tenant.
Frameworks. LangGraph (graph-based), CrewAI (role-based), Autogen (conversational), Semantic Kernel (Microsoft), Google's ADK. Choose based on ecosystem fit; the architectural pattern is the same.
End-to-end agent workflow
Trace an agent workflow. User asks the agent "find last month's top-selling product and prepare a stock forecast for next quarter."
Planner decomposes: (1) query sales data for last month, (2) rank by units sold, (3) query historical trend, (4) forecast next quarter, (5) format response.
Executor invokes query_db tool with SQL for step 1. Returns 10k product-level rows. Working memory captures aggregates: top 5 products with unit counts.
Planner picks top 1 as the target. Executor invokes query_db for historical trend of that product over the last 24 months. Returns time series.
Planner invokes forecast tool with the time series. Tool returns Prophet or ARIMA forecast for the next quarter.
Reflection: is the forecast reasonable? The agent checks the forecast vs recent variance; if wildly off, it re-runs with alternative parameters. Ok, reasonable.
Planner formats the response using the assembled facts. Streams to the client with citations to each data source. Total cost logged; trace URL provided for debug.
If any tool had failed, the executor would retry with exponential backoff. If retries exhausted, the planner would either escalate to a human or produce a partial answer with clear caveats.