AI agent, orchestration & ops patterns, animated

How it works

1. Break the task into fixed steps, each a focused LLM call.
2. Add plain-code gates between steps to validate output.
3. The developer, not the model, decides the order.

Use it when

• The steps are known in advance (extract → draft → polish).
• You need predictable cost, latency and easy testing.

Watch out for

• Errors propagate: validate between steps.
• Breaks on inputs the chain didn't anticipate.

How it works

1. A small, fast model classifies the request with structured output.
2. Code dispatches it to a specialised prompt or agent.
3. Each route keeps its own instructions and tools.

Use it when

• Requests fall into distinct categories with different tools or policies.
• One big prompt is getting confused by unrelated instructions.

Watch out for

• Misroutes: add a confidence threshold and a fallback route.
• Measure router accuracy separately with its own eval set.

How it works

1. Split work into independent pieces (in LangGraph: Send).
2. Workers run at the same time.
3. A reducer collects results; an aggregator merges them.

Use it when

• Summarising or analysing many documents.
• Running several checks or "votes" for reliability.

Watch out for

• Provider rate limits: cap concurrency.
• Cost multiplies with the number of workers.

How it works

1. Reason: the model decides what it needs.
2. Act: it requests a tool call; your code runs it.
3. Observe: the result goes back into context. Repeat until no more tool calls.

Use it when

• The number and order of steps can't be known in advance.
• The default choice for a general tool-using agent.

Watch out for

• Loops: set a step limit (recursion_limit).
• Too many tools: split into sub-agents.

How it works

1. A strong model writes a step-by-step plan once.
2. An executor (often cheaper) runs each step with tools.
3. A re-plan check updates the plan or finishes.

Use it when

• Long, multi-step tasks (data analysis, migrations, research).
• You want the plan visible to users or reviewers.

Watch out for

• Stale plans: always allow re-planning.
• Over-planning simple tasks: ReAct may be enough.

How it works

1. A generator writes a draft.
2. An evaluator scores it against explicit criteria.
3. Specific feedback loops back until it passes or a round limit is hit.

Use it when

• Quality matters and criteria are clear: code, translation, structured reports.
• Tests or rules can act as the evaluator.

Watch out for

• Cost: cap rounds (2–3).
• A vague rubric makes the loop pointless.

How it works

1. A supervisor decides which specialist acts next.
2. Each specialist has its own prompt, tools and context.
3. Results return to the supervisor until the work is done.

Use it when

• Different skills or permissions are needed on one output.
• A single agent has too many tools or conflicting instructions.

Watch out for

• Every hop adds latency and tokens.
• The supervisor can become a bottleneck.

How it works

1. Each agent can transfer control to another agent.
2. The receiving agent now owns the conversation.
3. No central coordinator.

Use it when

• Conversations move between departments (sales → onboarding → support).
• Specialists should talk to the user directly.

Watch out for

• Lost context: pass a structured summary at each handoff.
• Ping-pong between agents: limit handoffs per conversation.

How it works

1. The agent proposes a risky action.
2. interrupt() pauses the graph; state is checkpointed.
3. A person approves or rejects; Command(resume=…) continues.

Use it when

• Money, emails, deletions or anything irreversible.
• Regulated workflows that need an audit trail.

Watch out for

• Needs a durable checkpointer (Postgres).
• The node re-runs on resume: keep side effects idempotent.

How it works

1. The agent decides whether it needs documents.
2. It retrieves, then grades the results for relevance.
3. If they miss, it rewrites the query and tries again; then answers with citations.

Use it when

• Multi-hop or ambiguous questions over your documents.
• Classic one-shot RAG returns irrelevant chunks too often.

Watch out for

• Extra LLM calls add latency: cap retries.
• Fix retrieval quality first (chunking, hybrid search, reranking).

How it works

1. An orchestrator LLM reads the task and decides the subtasks.
2. Workers run them, often in parallel (LangGraph: a list of Send).
3. A synthesizer merges results and can request follow-up work.

Use it when

• You can't predict the subtasks: code changes across files, research questions.
• Parallelization would work, but the split depends on the input.

Watch out for

• Cap the number of subtasks and follow-up rounds.
• Give workers clear, self-contained instructions.

How it works

1. A top supervisor delegates to team leads.
2. Each lead is itself a supervisor over its own specialists (a LangGraph subgraph).
3. Only summaries travel up, keeping each level's context small.

Use it when

• One supervisor would have too many workers to manage well.
• Teams need different tools, models or permissions.

Watch out for

• Each level adds latency, tokens and places to lose detail.
• Trace across levels with one run id.

How it works

1. Events (new ticket, PR opened, nightly schedule) go to a queue.
2. Workers pull jobs and run the agent graph; thread_id = job id, so crashes resume.
3. Results are stored and people notified; failures go to a dead-letter queue.

Use it when

• Work takes minutes to hours, or nobody is waiting on a chat.
• Traffic is spiky and you need back-pressure.

Watch out for

• Idempotency: jobs can be delivered twice.
• Scale workers on queue depth; alert on dead letters.

How it works

1. A shared workspace holds the goal, facts, plans and drafts.
2. Agents read it and contribute when they can help; they don't message each other.
3. A controller decides who acts next and when the goal is met.

Use it when

• Many specialists contribute in an order you can't fix in advance.
• You want every intermediate result visible and auditable.

Watch out for

• Define a schema for the board and rules for conflicting writes (reducers).
• The board grows: summarise or prune it.

How it works

1. Several agents (or models) answer independently.
2. A judge, or a simple majority vote, compares the answers.
3. Optionally, agents see each other's reasoning and revise before a final vote.

Use it when

• Hard reasoning or classification where one mistake is costly.
• You want a confidence signal (agreement rate).

Watch out for

• Cost multiplies with the number of agents and rounds.
• Agents sharing the same model can share the same blind spot.

How it works

1. Your agent discovers a remote agent from its agent card (skills, endpoint, auth).
2. It sends a task over A2A and receives streamed status and artifacts.
3. Tools and data are reached through MCP servers.

Use it when

• Agents are built by different teams, frameworks or companies.
• You want to reuse an agent without merging codebases.

Watch out for

• Treat remote agents as untrusted: authenticate, validate outputs, set timeouts.
• Version the contract (skills and schemas).

How it works

1. Every step's state is saved to a checkpointer (Postgres).
2. After a crash, a new process resumes the same thread_id with invoke(None, config).
3. Finished steps are skipped; only the interrupted step runs again.

Use it when

• Runs have real side effects (payments, emails, tickets).
• Runs are long, or wait on humans for hours or days.

Watch out for

• The interrupted step re-runs: make it idempotent.
• Use durability="sync" for critical flows.

How it works

1. Retry transient errors (429, 5xx, timeouts) with exponential backoff and jitter.
2. A circuit breaker counts failures and "opens" after too many.
3. While open, traffic goes to a fallback model or region; later it tests the primary again.

Use it when

• Any production LLM call: providers rate-limit and have outages.
• You have a second provider or region available.

Watch out for

• Never retry validation errors (400s).
• Retries from many replicas can amplify an outage: cap them.

How it works

1. Pair every side-effecting action with an undo.
2. Run the actions in order.
3. On failure, run the undos in reverse order, then report clearly.

Use it when

• One agent run changes several external systems.
• There is no single transaction across those systems.

Watch out for

• Undos can fail too: retry them and alert.
• Combine with idempotency keys so retries don't double-apply.

How it works

1. Input guard: size limits, PII redaction, injection checks.
2. Tool policy: allow, require approval, or deny each tool call.
3. Output check: grounding, PII and forbidden promises before replying.

Use it when

• Always, for anything customer-facing.
• Agents with tools that change data or spend money.

Watch out for

• Prompts aren't a security boundary: enforce permissions in code.
• Measure false blocks as well as attacks caught.

How it works

1. Every run emits traces (each LLM and tool call) and metrics.
2. Evals score sampled traces; dashboards and alerts watch metrics.
3. Fixes go through an eval gate before deployment.

Use it when

• From the first production user.
• Before changing prompts, models or tools.

Watch out for

• Tag everything with thread, tenant, prompt and model version.
• Redact PII in traces.

How it works

1. A load balancer spreads traffic over stateless replicas.
2. State lives in Postgres (pooled connections), so any replica serves any thread.
3. A rate limiter keeps LLM calls under the provider quota.

Use it when

• More than one replica, or any autoscaling.
• Traffic spikes near your provider quota.

Watch out for

• Scale on concurrency or queue depth, not CPU.
• Raise load-balancer timeouts for streaming responses.

How it works

1. Unit tests with a fake LLM, then offline evals against a baseline.
2. Ship to a small canary share of traffic.
3. Compare success, cost and latency; roll out or roll back.

Use it when

• Every prompt, model, tool or retrieval change.
• Model provider upgrades.

Watch out for

• Pin model versions; floating aliases change under you.
• Keep rollback to a single config change.