Backpressure and Queue Depth Management for Bursty Agent Workloads

A multi-agent research tool built on a popular open-source stack once ran for 11 days without anyone noticing, because two agents were stuck talking to each other in a loop. By the time someone caught it, the bill hit $47,000. Nobody was asleep at the wheel. The system just looked busy, and busy looked fine.

Agent pipelines fail differently than the services most engineering teams have spent a decade learning to operate, and that difference is the trap. Agent pipelines fail differently than the services most engineering teams have spent a decade learning to operate. A traditional queue that fills up faster than it drains crashes hard and fast, usually with an out-of-memory kill that shows up on a dashboard within seconds. Agent pipelines don't do that. They burn money and cycles while producing nothing, and the failure often hides behind an abstraction layer built to make the system look productive. The MAST study, which looked at more than 1,600 execution traces across seven open-source agent frameworks, found failure rates between 41% and 87%, with unstructured multi-agent networks amplifying errors up to 17 times over single-agent baselines. Most of those weren't model failures. They were orchestration failures, and orchestration is the part operators are supposed to control.

The root cause traces back to one gap: the LLM generating a plan has zero visibility into queue depth, token budget, or how far behind execution has fallen. That feedback loop doesn't exist unless someone builds it.

Four properties that make agent workloads harder to queue-manage than ordinary services

Ordinary web services process requests that mostly behave. Agent workloads don't, for four reasons that stack on top of each other.

Execution time swings wildly. A basic classification task might take a few seconds; a multi-step coding task might run for minutes. That means queue drain rate can't be estimated from request count the way it can for, say, an HTTP API. Cost swings just as hard, and in the wrong direction: an overloaded agent system doesn't just slow down, it starts spending faster, because every extra iteration burns tokens, and those costs compound as sub-agent output gets fed back into the orchestrator for another pass.

Then there's the non-determinism. One run might close out in five iterations, another might take twenty for the exact same task type. Averages don't help here, because capacity planning built on mean values gets blown out at the tail, which is exactly where the expensive failures live.

Fan-out is the fourth piece, and it's the one that turns a small miscalculation into a real bill. A single request to a coordinator might spin up five sub-agents. Each of those sub-agents might trigger several tool calls of its own. Picture a research agent that decomposes into five sub-agents, each returning a few thousand tokens: that's roughly 10,000 tokens flowing back into the orchestrator per cycle. Running a few of those cycles fills up the context window, degrades response quality, and causes the agent to start hallucinating or just fall over.

Fan-out has a nastier cousin too: the rate-limit cascade. When several sub-agents fire concurrent tool calls, they can hit a provider's rate limit at the same moment. Each one retries. Those retries stack on top of each other, and a system running a handful of requests per second can suddenly be generating many times that in retry traffic alone. None of these four properties is fixable in isolation. Variable time plus fan-out means one burst request can hog the queue and drain the budget at the same time, and no single patch closes that gap.

What backpressure means when the producer is an LLM planning step

Backpressure isn't a new idea. TCP has used sliding windows for decades to make a fast sender back off for a slow receiver. Kafka throttles producers. Node.js pauses streams. The concept is always the same: a slow consumer tells a fast producer to ease up.

Mapped onto an agent pipeline, the producer is the planning or decomposition step, the part of the system deciding what work to spawn next. The consumer is the execution layer: tool calls, model inference, outbound API requests. Here's where the old pattern breaks. In TCP or Kafka, the producer is code, and code obeys a signal. In an agent pipeline, the producer is an LLM generating a plan with no built-in awareness of downstream queue depth or budget. It can't hear a backpressure signal because nobody's speaking one to it. The signal path has to be built externally, in the orchestration layer, and it has to be enforced, not negotiated with the model through a prompt.

Recent queueing-theory work (arxiv 2504.07347) models an AI-agent workload as a multi-class batch-service processing network, where each node represents a class of LLM-level requests grouped by their statistical shape, things like prefill and decode token sizes. That framing matters because it shows why a single global concurrency cap is too blunt an instrument. Workload shapes vary too: some pipelines run as sequential chains, some fork-join, some loop back on themselves in self-reflection cycles, and each of those topologies needs its own instrumentation points for backpressure to actually propagate. The implementation comes down to three pieces: bounded work queues, budget-aware planning, and execution feedback. Those are just names for now. The next section builds each one out.

The three foundational controls every agent pipeline needs before anything else

Start with bounded work queues. Every buffer sitting between the planning step and the execution layer needs a fixed maximum size. When it's full, the planner has to block, not silently drop work and pretend everything's fine. An unbounded queue is a memory leak wearing a disguise: it doesn't fail immediately, it just grows until something worse breaks downstream. The practical fix is setting a max depth at the ensemble level and wiring a backpressure signal that travels upstream, so the calling system can retry, reroute to another path, or report the failure honestly instead of stacking more work on a queue that's already drowning.

Budget-aware planning comes next. Before the orchestrator spawns a new sub-agent or greenlights a tool call, it needs to check three things: how much token budget is left, how much rate-limit headroom remains, and how deep the pending work queue already is. If any of those are tight, the right move is consolidating tasks or pushing off the low-priority ones, not spawning more work and hoping it clears in time. The structured version of this is hierarchical budget allocation, where hard ceilings at multiple levels prevent any one sub-agent from consuming the entire job's budget. The ceiling is what stops the next $47,000 bill.

Execution feedback closes the loop. Every completed or failed task needs to report its actual token spend and wall-clock time back to the orchestrator, and the planner needs to actually use that data, cutting concurrency when things run slow, batching tool calls when rate limits are tight. Skipping this step leaves the planner flying blind, making decisions with no idea what's actually happening downstream. These three controls only work as a system when all three are wired together. Two out of three still leaves a blind spot.

Five patterns that extend the foundation into production-grade capacity management

The three foundational controls stop the bleeding. These five patterns are what turn that into something that holds up under real production load.

Adaptive concurrency limits borrow directly from TCP congestion control: Additive Increase, Multiplicative Decrease. Instead of guessing a hardcoded sub-agent ceiling, the system starts conservative, ramps concurrency up gradually while tasks finish within latency targets and without rate-limit errors, then cuts back hard, multiplicatively, the moment it sees 429 errors or the context window growing past budget. The payoff is that it converges on the right throughput on its own, without an operator trying to forecast it in advance, and it re-adjusts automatically when the model, the provider, or the workload shifts underneath it.

Priority queues need an aging mechanism, or they'll starve low-priority work. A pure priority system where high-priority requests keep arriving means low-priority ones might never run. Aging fixes that: a batch request that's waited past a set threshold gets bumped to top priority and runs next, no matter what's still queued behind it. Three tiers cover most production setups: urgent and interactive, normal, and batch.

Operational profiles handle the load an operator can see coming. Rate limits and priority queues are reactive by nature; a profile is proactive, bundling per-ensemble capacity targets and shared-memory pre-load instructions into something that can be deployed ahead of a known spike, such as a recurring weekly batch job or an event triggered externally through something like an APPLY_PROFILE directive. Pre-warming capacity this way skips the cold-start lag that comes with a reactive autoscaler catching up after the fact.

Circuit breakers stop retries from compounding into a cascade. When an ensemble keeps rejecting requests with a backpressure signal, the caller should open the breaker and route elsewhere or fail upstream, rather than retrying in a tight loop and adding fuel to an already-saturated node. Concurrent retries from several sub-agents at once can multiply request volume by an order of magnitude in a matter of seconds. A circuit breaker is what keeps a temporary overload from turning into a full cascade.

Loop detection catches the failure mode that started this whole piece. Two agents with conflicting instructions, say one enforcing a formal tone and the other told to stay casual, can get stuck endlessly revising each other's output. The fix is tracking iteration count and token spend per task, and if either crosses a threshold with no terminal output produced, surfacing it as a failure instead of letting it keep running. Bounded queues stop new work from piling in; loop detection stops work that's already running from consuming resources forever. A layered timeout structure backs this up: one limit per tool call, one per task loop, one for the sandbox's total lifetime, three independent trip wires so no single failure mode can slip past all of them.

Why kernel-level isolation determines whether application-layer backpressure holds at scale

Every pattern above assumes the infrastructure underneath it is solid. At real scale, that assumption can fail on its own.

One documented hyperscale scenario involved 847 tenants queued inside a multi-tenant orchestration layer, an 18-second P99 latency, and a GPU cluster fully saturated. At that point the queue fills faster than any application-level control can drain it, no matter how well-tuned the backpressure logic is.

Three common fixes fall apart here. An unbounded thread pool, spawning a task per tenant, just grows the queue until the process is killed for running out of memory: it trades latency for staying alive and loses both. Standard Kubernetes autoscaling reacts on a 30 to 90 second lag, so a latency spike that resolves in 20 seconds has already blown the SLA before new pods even spin up, and those pods often arrive just as the system's recovering on its own, adding a second spike on top of the first. An app-layer rate limiter backed by something like Redis adds network round-trip time to the hot path and becomes a single point of failure, with its own connection pool vulnerable to a thundering herd the moment traffic spikes.

Scheduler behavior degrades once tenant count climbs into the thousands. A per-tenant process model can push TLB miss rates past 40%, visible directly in perf stat. A thread sitting on epoll_wait costs 3 to 5 microseconds just to wake up on a saturated scheduler run queue. Multiplied across 10,000 tenants, this adds up to 30 to 50 milliseconds of pure scheduler overhead, before any real work even starts. Kernel-side mitigations like IBRS and STIBP make context-switch costs worse on patched systems in 2026, and that's not a hypothetical: it's a measured cost on real hardware.

One structural alternative getting attention is a shared-nothing model built on a lightweight sandboxed runtime, where each tenant component runs in roughly 2MB of linear memory with no page table of its own, and a host runtime like wasmtime multiplexes every tenant across a fixed thread pool, no per-tenant TLB entries, no ASID allocations, no idle queue entries sitting in the scheduler. This isn't the only path forward, and it shouldn't be read as the single correct answer. Application-layer backpressure assumes the runtime beneath it isn't the bottleneck. Past a certain tenant count, the runtime is the bottleneck, unless isolation gets designed alongside the queue logic from the start rather than bolted on after.

How backpressure requirements interact with the agents operators are deploying

None of this is theoretical when it comes to the agents already running in production. Two categories carry very different backpressure obligations.

Self-hosted operator runtimes put the entire burden on the operator. OpenClaw is one example: channel-agnostic, self-hosted, built for sub-agent orchestration, and capable of acting as a meta-orchestrator that dispatches Claude Code, Codex, Gemini, and Cursor Agent as sub-tasks. That meta-orchestrator role is exactly where fan-out amplification hits hardest, which makes bounded queues non-negotiable rather than optional. OpenClaw carries 369,000 GitHub stars, and a serious security review needs to happen before any production rollout, not after.

Hermes Agent, from NousResearch, is at roughly 247,000 stars and works differently: it holds persistent memory across sessions and auto-generates its own skill documents. That self-improving loop is genuinely useful, but it also means a poorly bounded Hermes instance can quietly accumulate state and cost across sessions in a way a stateless agent simply can't.

Coding-agent surfaces split the responsibility differently. Some backpressure controls are enforced by the platform itself; others still have to be built into the harness around it. Claude Code is regarded as the strongest default for production engineering teams, but on the Max plan, routines are capped at 15 per day. This is not a bug to route around; it's a fixed constraint that has to factor into job scheduling from the start, the same way a token budget ceiling or a rate-limit headroom check does anywhere else in the pipeline.

Every one of these systems, self-hosted or platform-hosted, runs into the same underlying truth. The LLM planning the work has no idea what the queue beneath it looks like. Someone has to build that visibility in, deliberately, because it will never appear without deliberate engineering effort.