What Infrastructure an AI Agent Actually Needs at Launch
Six operational layers separate demo agents from production systems that run reliably at scale.
An AI agent can reach paying customers without its founder having to manage servers or learn software deployment and operations if managed hosting does that work. But production still needs six layers, covering job routing, memory, scheduling, policy checks, execution safety, and audit records. The choice is who operates those layers.
Why most agents never leave the demo stage
A staging agent resolves tickets perfectly, then production traffic triggers retry storms. The scheduler queue backs up, duplicate executions fire, and three customers get automated responses that conflict with each other. The model was not the root cause. Without canary gates and dead-letter handling, a problem with load became a problem for customers.
That is the gap a demo hides. A production agent runs all the time against live data, writes to real systems, and must recover at 3 AM without a founder watching. Its output can also change on the same input, making failures harder to reproduce. Better prompts cannot supply a missing decision boundary, staged rollout, or incident record. The six layers answer those gaps as a connected system, with a distinct failure mode wherever one is absent.
The six-layer stack every production agent requires
The execution bus routes work, the state store preserves context, and the scheduler controls retries and timeouts. The policy decision point checks proposed actions, execution safety constrains what runs and what leaves, and the audit trail records what happened. These are dependencies. Slow execution can trigger retries that inflate token costs and stress the scheduler. Likewise, output filtering cannot reverse a database change that bypassed the policy check.
Layer 1: The execution bus — routing work without losing it
A durable execution bus gives work somewhere reliable to go while workers are busy. NATS uses named subjects for routing, while Kafka routes events through topics. In an asynchronous setup, the user submits a task and gets an acknowledgment while the bus routes it to an available worker. Results arrive when the work is ready. The user does not have to hold a connection open throughout execution.
If developers cannot see queue depth and job status, they cannot tell whether work is waiting or has silently died. Retries then become guesswork, and retry storms can produce duplicate tickets, emails, or database records. The bus also separates routing from compute choices. AWS Lambda and Google Cloud Run functions fit stateless agents with unpredictable traffic, while ECS and Kubernetes fit agents needing consistent environments, with Kubernetes better suited to stateful workloads. Dedicated VMs serve high-volume cases where cold starts are unacceptable.
Layer 2: State and memory — what the agent remembers and where it lives
An agent that remembers a preference, previous action, or intermediate result needs an explicit storage design. Not all memory needs the same home. Redis is a good fit for temporary conversation history while sessions are active because it is fast and supports expiry. Postgres or an equivalent persistent store holds long-term memory, tool-call history, and evaluation data through restarts. Vector databases such as Pinecone or Weaviate serve retrieval across a large knowledge base.
That storage choice also affects how requests reach workers. A stateful agent needs either session affinity, which routes the same user to the same instance, or shared state that any instance can read. Without either, context can be lost halfway through a conversation. Memory must also be encrypted and isolated per user, including cached queries. Because every store adds operational work and another security boundary, the practical choice is to persist only what the agent needs.
Layer 3: The scheduler — retries, timeouts, and what happens when a run dies
The bus routes a job, but the scheduler governs its progress. A transient tool failure should not leave a workflow stranded with no record of where it stopped. Restarting it should not repeat all the actions it has already taken, either. Deterministic retry rules establish what happens next, while dead-letter routing sends jobs that exceed retry limits somewhere inspectable.
Retries are safe only when writes are idempotent. Creating a ticket, sending an email, or updating a record must produce the intended outcome without creating a duplicate when the job runs again. Timeouts belong at each stage, too. A database query, search API call, and code execution step each take time, so one slow tool must not hold up the whole run without a time limit.
Layer 4: The policy decision point — the only control that runs before the agent acts
The policy decision point evaluates a proposed action before dispatch to a tool or external system. Its check is deterministic and does not ask the model to make another open-ended judgment. If the action fails, it is blocked or queued for human approval. This is the central authorization boundary in the stack. An output filter may hide sensitive text later, but it cannot unsend an email.
The boundaries must be explicit before deployment. Goal boundaries set out which outcomes are allowed and when refusal or escalation is needed. Tool boundaries define reachable systems and allowed operations, with read-only access as the default wherever writes are unnecessary. Output boundaries set out the content filters, redaction rules, and format constraints to use. Credentials for each agent then enforce least privilege, so no agent gets blanket admin access.
Placement changes the strength of that boundary. An in-process policy check runs inside the agent’s trust boundary, which is convenient but leaves the control within the environment it governs. An out-of-process check runs as a separate service outside that boundary, separating enforcement from a compromised agent. Sandbox restrictions can also prevent harm during execution, and authorization’s distinct job as a protective control is deciding whether dispatch is allowed at all.
For actions with serious consequences, that decision should need a human to confirm it before execution. Sending emails, modifying production data, and executing code belong in that approval workflow. The record must identify who approved the action and which policy snapshot permitted it. Credential rotation evidence and versioned policy files with PR review make those permissions inspectable. A vague claim that an agent has guardrails does not answer any of those questions.
Layer 5: Output safety, sandboxing, and the security perimeter around agent execution
Agents that run code, install packages, or drive a browser need hardware-enforced isolation for each session. Containers share the host kernel, while a MicroVM gives each workload a separate kernel to contain exploits that could cross tenant boundaries. Namespace separation alone is not enough here. The sandbox should be created for one session, hold no standing credentials, reach only allow-listed external hosts, and be destroyed when the task ends.
The AWS Security Bulletin describes CVE-2026-12530 as a crafted package-name argument that bypassed an incomplete blocklist in install_packages(), allowing arbitrary commands inside the Code Interpreter sandbox. The issue was addressed in the AgentCore Python SDK version 1.6.1. The boundary of the exploit is important. Execution inside a sandbox shows why package installation needs containment.
Wiz describes a different failure in OpenClaw’s CVE-2026-25253. A malicious link sent the app’s authentication token to an attacker, who could then connect to the local gateway with operator-level privileges, change sandbox settings and tool policies, and invoke actions resulting in arbitrary code execution on the gateway host.
Output safety addresses a different part of the problem. After execution, an allow, redact, or quarantine pipeline catches personal information and secrets that got past input controls. Memory that is encrypted for each user protects the session boundary set in the storage layer. Neither control replaces the other. Safe output does not prove that execution was safe, and a sandbox does not guarantee that a response contains no private data.
Compliance requires the same separation of claims. The European Commission states that EU AI Act Article 50 applies from 2 August 2026, including obligations for systems that interact directly with people, while solely background or machine-to-machine systems fall outside that direct-interaction obligation. SOC 2 Type II confirms an independent audit of a vendor’s internal controls over a defined period. It is an entry requirement. It does not settle model-provider data handling, transcript retention and deletion, acceptable action records, or whether a particular deployment is compliant.
Layer 6: The audit trail — what makes an incident survivable
Standard application monitoring can show HTTP status codes and latency without showing what an agent did. Agent tracing needs tool calls, model calls, retries, handoffs, and finish reasons. LangSmith, LangFuse, and custom tracing solutions serve this specialized need. For every action, the audit record must also capture the approver or approving policy, the active policy snapshot, and the exact output. Those details make an incident timeline reproducible.
Append-only logs with defined retention prevent later edits from rewriting the evidence. Structured records of the agent’s reasoning process, tool calls, and decisions help debug behavior that varies on the same input. Recovery needs practice as well as records. Teams need tested rollback drills because a team that has never reversed a bad deployment cannot count on doing it well under pressure.
How staged rollout ties all six layers together
Staged rollout tests the six layers as one operational system. Before live traffic, Phase 0 replays real production requests through the policy engine without carrying out side effects. Small canary traffic then exercises the bus and scheduler, concurrent sessions test state management, and greater throughput tests policy decision latency. A full cutover hides those distinctions until failures reach every user.
Promotion requires objective gates. The measures include error budget, policy-decision latency p99, approval-workflow turnaround time, and rollback-drill success. If a gate fails, the response is immediate rollback. Run history, dead-letter handling, and audit records provide the evidence needed to understand the failure before trying again.
What this stack costs to build yourself versus what managed hosting actually handles
Self-building is expensive because of both the cloud bill and the founder’s continuing labor to configure and monitor a broker, provision and back up stores, maintain scheduler logic, version policy, patch sandboxes, and retain immutable logs. In a hand-configured B2B deployment, that work repeats for each customer. Every new instance also needs securing and recovery. The notes have no prices that can be compared, so ranking the options by dollars would hide the trade-off.
For a solo founder without an ops background, managed agent hosting shifts that operational burden to the platform. The relevant capabilities are uptime management, encryption in transit and at rest, billing controls, deep integrations, and automatic per-user agent isolation. A serverless function supplies compute. The useful comparison is therefore which layers the platform operates and which remain the founder’s responsibility. The unit of scale is one isolated, persistent agent per customer, provisioned automatically at signup.
Per-run and per-agent token budgets with hard stops belong in the deployment stack. Model-tiering routes simpler subtasks to cheaper models, while caching avoids repeatedly loading the same context. Cost attribution by team and use case makes returns measurable. These controls keep retries and growing context from adding a surprise bill when things go wrong.
A pre-launch readiness check across all six layers
For a team preparing to launch, the readiness test is concrete. For the execution bus, can the team see queue depth, backpressure, and whether each job is waiting, processing, or dead? For state and memory, is each user’s context isolated and recoverable after a crash without restarting the work? For the scheduler, are writes idempotent and exhausted retries sent to an inspectable dead-letter queue? These questions test recovery.
The remaining questions test authority, containment, and evidence. For policy, can the team identify the snapshot behind the last risky approval and show that enforcement sits outside the agent’s trust boundary? For execution safety, is each session isolated, are credentials restricted, and are outputs checked for private data and secrets? For audit, can logs alone reconstruct the trigger, action, policy, output, and approver? A recent rollback drill and a first canary gate with objective metrics then test whether those answers hold when the system runs for real.
Most teams can answer yes to most of these questions. A single no is what separates a launched agent from a liability.
