Choosing an AI agent architecture is a design decision, not a framework preference. This guide compares single-agent, router, sequential, parallel, planner-executor, reviewer, and human-in-the-loop workflows, then gives you a repeatable way to estimate complexity, latency, cost, reliability, and operational effort before implementation.
Overview
An AI agent workflow combines a language model with instructions, tools, state, and control logic. The architecture determines how those parts interact: whether one model handles the entire request, several workers collaborate, a router selects a path, or a person approves an action before it continues.
The simplest viable design is usually the best starting point. Additional agents and orchestration steps can improve specialization or control, but they also create more model calls, state transitions, failure points, and observability requirements. A multi-agent system is not automatically more capable than a carefully constrained single agent.
| Pattern | Control flow | Relative latency and cost | Common failure mode | Good fit |
|---|---|---|---|---|
| Single agent | One agent interprets the request and uses approved tools | Low | Unclear tool selection or broad instructions | Bounded assistants, support tasks, simple automation |
| Router | A classifier or router sends work to one specialist path | Low to medium | Misrouting or incomplete route definitions | Distinct request categories with different prompts or tools |
| Sequential | Each step passes its output to the next step | Medium to high | Error propagation between steps | Documented pipelines and predictable transformations |
| Parallel | Several workers process the same input at the same time | Potentially higher total usage, lower elapsed time | Conflicting outputs or expensive fan-out | Independent research, extraction, or scoring tasks |
| Planner-executor | A planner creates tasks and an executor performs them | Medium to high | Plans that are incomplete, excessive, or unsafe | Variable multi-step work with clear task boundaries |
| Reviewer | A primary worker produces an output and a reviewer checks it | Higher | Reviewer agrees without detecting the real defect | Quality-sensitive outputs and structured validation |
| Human-in-the-loop | A person approves, edits, or rejects a proposed action | Higher elapsed time; variable model cost | Approval queues, ambiguous responsibility, or poor escalation | Irreversible, sensitive, or high-impact actions |
These categories can be combined. For example, a router may select a sequential workflow, while a reviewer and human approval step protect the final action. Treat each added pattern as a deliberate control or capability, not as a default feature.
How to estimate
Before selecting an AI agent framework or orchestration library, estimate the workflow using a small decision model. The goal is not to predict an exact production bill. It is to compare designs consistently while the inputs are still easy to change.
- Map the task. Write the user request, required data, tools, outputs, and points where a decision must be made. Separate deterministic operations, such as validation or database lookup, from model-dependent operations.
- Count model turns. Estimate the number of model calls for a typical request and for a difficult request. Include routing, planning, tool selection, execution, review, and recovery calls.
- Identify serial depth. Count the longest chain of dependent steps. Serial depth is a practical latency indicator because later steps cannot begin until earlier outputs are available.
- Identify fan-out. Count parallel workers and estimate how often they run. Parallelism can shorten elapsed time, but it may increase total usage and require aggregation logic.
- Assign a failure boundary. For every step, specify whether failure causes a retry, a fallback, a partial result, or human review. Do not leave recovery behavior to the model alone.
- Score operational overhead. Rate tracing, state management, prompt versioning, evaluation, access control, and alerting on a simple low, medium, or high scale.
A useful comparison worksheet is:
Estimated usage = requests × average model turns × average input/output volume
For a more realistic comparison, split the estimate by path:
Total usage = normal requests × normal path usage + exceptional requests × recovery path usage
Use relative units when provider pricing, model selection, or traffic is not fixed. For example, compare a single-agent design at one unit per request with a reviewer design at two or more units per request. Replace those units with actual token or API-cost inputs only when the model, context size, and pricing assumptions are known.
Inputs and assumptions
Architecture estimates are only as useful as their assumptions. Record the following inputs in a small design document or spreadsheet:
- Request volume: expected normal and peak requests over the period being measured.
- Path distribution: the share of requests routed to each specialist, fallback, or approval path.
- Model turns: calls for routing, planning, tool use, synthesis, review, and retries.
- Context size: recurring system instructions, retrieved material, conversation history, and tool results passed into each call.
- Tool behavior: average and worst-case duration, error rate observed in testing, and whether calls are idempotent.
- Quality target: which errors are tolerable, which require a retry, and which require a person.
- State requirements: whether the workflow needs short-lived task state, conversation memory, durable records, or an audit trail.
- Observability level: whether you can trace prompts, tool calls, intermediate outputs, latency, and final outcomes without exposing sensitive data.
Keep model generation separate from deterministic checks wherever possible. Schema validation, permission checks, duplicate detection, and range validation should not depend on a persuasive-sounding response. A reviewer agent can add useful coverage, but it should supplement executable checks rather than replace them.
For prompt and workflow reliability, pair architecture testing with a representative evaluation set. The Prompt Testing and Evaluation Framework can help structure cases, expected outputs, and pass criteria. If the workflow uses retrieved documents, also review the RAG knowledge base maintenance workflow; stale or poorly retrieved context can look like an orchestration failure.
Worked examples
Example 1: Internal knowledge assistant
A knowledge assistant answers questions from a controlled document collection and does not take external actions. Start with a single agent using retrieval, citations or source references, and a structured response format. Add a router only if different question classes need materially different retrieval rules. A reviewer may be worthwhile for answer grounding, but first measure whether retrieval quality, source coverage, or prompt instructions are the actual constraint.
Estimated path: one retrieval decision, one answer generation call, and optional validation. The architecture has low control-flow complexity and moderate observability needs. Track unanswered questions, unsupported claims, retrieved sources, and user corrections before adding more agents.
Example 2: Multi-step operations request
Suppose an assistant must interpret a request, check records, prepare a change, and ask for approval. A sequential workflow is easier to reason about than a free-form agent: validate the request, retrieve records, prepare a proposed change, run deterministic checks, and pause for human approval.
Here, the human-in-the-loop step is not merely a quality feature. It is a boundary around an action that may be difficult to reverse. Store the proposed input, tool results, decision, approver, and final action as separate events. If a tool fails, retry only when the operation is safe to repeat; otherwise surface the failure for review.
Example 3: Research and synthesis
A research workflow may use a planner to define questions, parallel workers to gather independent findings, and a synthesizer to produce an answer. This pattern can reduce elapsed time when tasks are genuinely independent, but it adds fan-out, aggregation, duplicate information, and conflicting evidence.
Estimate both normal and expanded paths. If four workers run for every request, the model usage can be several times the single-worker baseline before synthesis or review is counted. Set a maximum worker count, a clear stopping rule, source requirements, and a fallback when one worker fails. A reviewer should check evidence coverage and contradictions rather than simply rewrite the synthesis.
Example 4: Classifying incoming requests
For support or operations intake, a router can assign one of a small number of well-defined categories. Start with representative examples, an explicit fallback category, and a confidence or ambiguity rule. If incorrect routing has a meaningful downstream cost, send uncertain cases to a review queue instead of forcing a specialist path.
When to recalculate
Revisit the estimate whenever the workflow, traffic, model, or business consequence changes. At minimum, recalculate after:
- Changing the model, context window, prompt, tool set, or retrieval strategy.
- Adding a planner, reviewer, parallel worker, memory layer, or human approval queue.
- Seeing repeated retries, tool failures, hallucinations, timeouts, or unexplained escalations.
- Changing request volume, peak traffic, response-time targets, or retention requirements.
- Observing that a workflow's actual path distribution differs from the original assumption.
- Releasing a major orchestration framework capability that changes available control or tracing options.
Use a short recalculation routine: export a sample of recent traces, count model turns and tool calls, group requests by path, measure serial depth and elapsed time, and compare quality by architecture path. Then update the worksheet and decide whether to simplify, add a control, or retain the current design.
Before production, prototype the smallest architecture that can satisfy the acceptance criteria. Add one pattern at a time and test the change against the same evaluation set. Use prompt regression testing to detect quality changes, review LLM cost optimization practices when context grows, and apply the LLM data privacy checklist before enabling detailed traces. The right AI agent architecture is the one whose behavior, cost, and failure handling you can explain and monitor—not the one with the most components.
