Checkpoint and Resume Architecture for Interrupted Long-Running Agent Tasks

Long-running agent tasks get interrupted. This is not a rare edge case or a sign of bad engineering. It is a statistical certainty that grows more likely with every minute a task runs. The architecture question is not whether to checkpoint, but what state to persist, when to persist it, and how to resume without replaying side effects that already landed.

Why long-running agent tasks will always be interrupted

A task with a high per-minute survival probability still has a meaningfully lower probability of completing a multi-hour run without interruption. The math compounds against you. Every minute of execution adds another point where something can fail, and the stack has plenty of candidates. Process crashes take down the worker. Deploys replace it mid-run. API rate-limit stalls freeze progress at exactly the wrong moment. Context windows exhaust themselves on long conversations. Human-in-the-loop pauses stretch for days or weeks while an approval sits in someone's inbox.

Real enterprise workflows are structurally multi-day. HR onboarding spans two weeks from offer letter to day-one schedule. Invoice disputes stall for days while vendors reply. Sales prospecting sequences stretch across a month of touchpoints. None of these fit inside a single blocking call, and none of them can survive a stateless chatbot that forgets everything the moment a container restarts.

The production evidence is already here. OpenAI's Codex team reports single runs that work on one task for more than six hours, often while the humans are sleeping. This is not a research pipeline curiosity. It is happening in production coding agents right now. Interruption is the default condition, and the only real question is what happens when it arrives.

What actually breaks when there are no checkpoints — and what breaks when checkpoints are naive

The failure is two-sided. No checkpointing wastes everything already done. Naive checkpointing creates new destructive side effects on resume. Both are problems, and solving only the first one hands you the second.

Without durable checkpoints, every interruption restarts the run from zero. The agent re-spends every token it already burned. It replays every paid API call it already made. Research loops lose their accumulated findings. Migration jobs lose their position. Multi-step approval workflows lose every signature they collected. The cost is not only monetary. A law firm's contract analysis system that processes documents nightly and fails at hour five does not just duplicate expensive processing. It misses morning deadlines, because the rerun cannot finish in time. The work was done and then thrown away, and the deadline was real.

With naive checkpoints, the failure mode inverts. Code that ran before a pause re-executes on resume. The agent sends the same email twice. It double-charges a card. LangGraph's own documentation states this directly: after an interrupt or resume, the runtime restarts the entire node from the beginning, and any code that ran before the interrupt call will execute again. The framework does not make this safe by default. Checkpointing alone is not the answer. The design question is what state to persist, when to persist it, and how to resume without replaying side effects that already landed.

The five runtime components every durable agent needs and what each one owns

Durable agent execution requires five separable runtime responsibilities. Conflating them is what makes naive implementations fragile. Anthropic's Scaling Managed Agents write-up separates session, harness, and sandbox as three distinct runtime components. Checkpoint and trace are not named in that write-up, but each has a known failure boundary and belongs in the model.

The session is the append-only log of calls, results, and approvals. It is typically an event table in Postgres, keyed by run ID. The OpenAI Agents SDK's session backends hold mutable conversation history that can be edited, cleared, and compacted, so they must not be treated as the append-only recovery log. The harness is the only component that makes decisions. It drives model and tool turns until the task ends. The sandbox isolates code, files, and network. A paused sandbox may preserve a convenient filesystem, but its continued existence must not be the only way to recover the result. The checkpoint enables resume without replaying the whole run. It is the step-boundary snapshot of graph or loop state. The trace enables post-run debugging. It holds correlated spans that let an engineer reconstruct why a run failed or overspent its budget.

These records need not be five separate products. You can hold run rows, checkpoints, and operation events in separate tables in one database. An object store can hold artifacts. A trace backend can hold timings. Keep them apart because each proves something different. After a restart, the checkpoint component must answer five questions: which run is this and may it continue, what should the agent do next, which artifact was approved, did the external operation already happen, and why did this run fail.

What to checkpoint, how often, and where to store it

Checkpoint design is about persisting the minimum state needed to resume, at the right frequency, to a store whose durability matches the cost of replaying the work. What you save is the position in the task list, which items have been processed, any intermediate results, and the accumulated context the agent needs for decision-making. You do not save the entire model context or conversation history. It grows too fast and makes recovery slow.

A key research finding from the 2026 Crab checkpoint/restore study sharpens this. Most agent turns produce no recovery-relevant state, so checkpointing all of them is mostly waste. A semantics-aware approach that checkpoints only meaningful state boundaries dramatically improves recovery correctness while cutting checkpoint traffic. The right question is not "checkpoint every step?" but "which steps carry new, non-reproducible state?"

LangGraph's three durability modes make the granularity and cost tradeoff explicit. Sync mode writes before the next step starts, the strongest guarantee. Async mode writes while the next step executes, leaving a small crash window. Exit mode writes only when the graph exits, offering no mid-run recovery at all. The right default for long-running work is async. Reserve sync for runs where replaying even one step has an external cost, like a payment or a notification.

Postgres is the durability-first default. Checkpoints survive restarts by construction, and history is queryable alongside application data. Redis is faster per write and supports native TTL management, but durability is an ops setting, not a library feature. A Redis running default in-memory settings loses every checkpoint on restart unless AOF/RDB persistence is explicitly enabled. The Redis checkpointer ships two savers. RedisSaver keeps full checkpoint history for audit and time travel. ShallowRedisSaver keeps only the latest checkpoint for high-volume ephemeral runs. Full-history checkpointing grows roughly quadratically with conversation length, because agent state typically contains an append-only message history. You need a retention job that prunes or archives completed-run history to keep the hot store bounded.

The idempotency problem: why checkpointing alone does not prevent duplicate side effects

Checkpointing makes state recoverable. It cannot make external side effects safe to replay without an explicit idempotency layer around each one. Stripe accepts an idempotency key for exactly this reason. You pass a key derived from something stable across retries, like the run ID and the step name, and Stripe ensures the charge lands exactly once. The payment API, the email server, and the Slack webhook do not know whether the agent crashed before or after they acknowledged the call. A checkpoint records what state the agent was in, not what external actions already landed.

If a persisted result exists for this run and this step, return the stored result instead of re-executing. If the ledger is keyed on the run ID and step name, the charge node can run twice but produce exactly one entry under charges. Externally visible operations, sending email, charging a card, calling a paid API, should all run inside this wrapper. The wrapper checks the store first. Only if no persisted result exists does it execute and then record the result.

Code upgrades introduce a second idempotency hazard. When the ordering of tasks or interrupts inside a node changes across a deploy, recovery may fail to correctly match already-saved results. Rehydration must preserve workflow topology, not just state values.

Human-in-the-loop pauses as a first-class checkpoint use case

Human approval gates are structurally identical to crash recovery from a checkpoint perspective. The agent checkpoints, waits, and resumes when the human responds. Treating them as the same problem eliminates a whole class of bespoke waiting logic. An interrupt raises the run's state to durable. The process can die. The pod can be rescheduled. Days can pass. The run is a row in the store, not a live process.

The new hire onboarding example from the ADK tutorial makes this concrete. The agent sends a welcome packet, pauses for days while the employee signs documents, delegates IT provisioning to a sub-agent, waits again for hardware delivery, and sends a day-one schedule. All without losing context. The agent is dormant, not blocked.

The architectural shift that makes this work is event-driven dormancy gates instead of active polling or blocked threads. The agent wakes only when an external event signals that the condition it was waiting for has been satisfied. Idle time is the dominant characteristic of real enterprise workflows. Holding a thread or socket open for days of idle time is wasteful and brittle.

Context pollution is the failure mode that durable schemas prevent. When an agent pauses for days then resumes with a massive context dump, the model hallucinates intermediate steps that never happened. It remembers approvals that were not given. It skips steps it assumes were completed. What fixes it is explicit, durable state decoupled from raw chat history, not a bigger context window.

Durable execution engines versus framework-level checkpointing

Framework checkpointers and durable execution engines both provide state persistence. They differ in who is responsible for failure detection and worker recovery, and that difference matters more than the checkpoint write itself. The sharpest practitioner critique of framework-level checkpointing is that the checkpointer saves state but there is no automatic failure detection. No supervisor watches for crashes. No watchdog checks heartbeats. The developer is the orchestrator and must build that recovery loop themselves.

LangGraph shows the framework-level approach clearly. When compiled with a checkpointer, the framework saves a StateSnapshot at every super-step, keyed by thread_id. No thread_id means no persistence and no resume, which is the first thing to verify in code review. When a parallel batch fails partway, the pending-writes ledger records outputs of steps that succeeded, so only the failed branch re-runs on resume.

Dapr offers an alternative pattern. Dapr Actors maintain state across restarts automatically. Dapr Workflows checkpoint state after each activity. Workflows suit long-running tasks with many small steps. Actors suit per-entity stateful workloads.

Durable execution engines take a different approach. Temporal, Restate, Inngest, Hatchet, DBOS, Cloudflare Workflows, and AWS Lambda Durable Functions expose failure detection, heartbeats, and worker recovery as platform primitives. The developer writes business logic and the engine manages the retry and recovery loop. Agent frameworks are also adding persistence at the agent layer. LangGraph, AutoGen, CrewAI, Dapr Agents, and Microsoft Agent Framework are all moving in this direction. The choice is increasingly which abstraction level to own.

AWS AgentCore Runtime changes one key constraint. Sessions on the default serverless microVM compute type run in isolated microVMs and can run for up to eight hours. The Instances compute type, GA August 6, 2026, supports sessions of up to 14 days. Checkpointing in that context becomes what it should be: resilience against genuine failures and a pause point for human approval, not a workaround for a short function timeout.

How infrastructure for checkpoint-resume works at the managed platform layer

Every piece of checkpoint-resume architecture must be provisioned, secured, monitored, and kept available for every customer's agent, independently. The durable store, the pending-writes ledger, the side-effect wrapper, the interrupt gate, the retention job, the trace backend. This is real and repeatable infrastructure work, and it takes founders away from building the agent itself.

The unit of scale for an agent-powered business is one isolated, persistent agent per customer. Each customer's run state must not bleed into another's. Each agent's checkpoint store must remain available even if another customer's agent is under load. Per-user sandbox isolation is the correct architecture. Managed agent platforms handle uptime, billing controls, encryption in transit and at rest, and deep integrations with Gmail, Slack, WhatsApp, GitHub, and thousands more tools via Composio. The operator delivers a working, durable agent to each customer without manually provisioning any of this.

Stripe let software companies take payments without becoming payments companies. The same separation applies to agent infrastructure. The goal is to give every customer their own durable, resumable agent without the founding team becoming an infrastructure team.

A decision map for checkpoint-resume design choices

Every checkpoint-resume system requires four explicit design decisions, and getting any one wrong reproduces a failure mode this article has already described. First, what to persist. Save minimum state only: position, intermediate results, approved artifacts, operation outcomes. Never the full conversation history in the checkpoint store. Use an application-owned run ID separate from any framework thread ID.

Second, when to persist. Match durability mode to replay cost. Async is the honest default. Sync is for when a single replayed step has an external cost, like a payment or notification. Exit-only is for cheap, stateless runs that can restart from scratch. Checkpoint after meaningful units of work, not every model turn.

Third, where to persist. Postgres for durability-first workloads. Redis only with AOF/RDB enabled and an explicit TTL policy per run class. Shallow checkpoints for high-volume ephemeral runs. Full history for audit-relevant runs, with an archival job to object storage.

Fourth, how to handle side effects. Wrap every externally visible operation in a store-check-before-execute pattern keyed on run ID and step name. Use the payment API's own idempotency key mechanism where available. Place side effects after interrupt calls, not before. Treat topology changes across deploys as a checkpoint compatibility risk.

Failure detection is orthogonal to all four. If the infrastructure layer does not detect worker crashes and re-queue the run, correct checkpoints do nothing. This is the boundary where framework-level checkpointing and durable execution engines differ, and where managed platforms remove the decision entirely.