How to scale AI-agent sandboxes for high-concurrency workloads
A sandbox API can work for ten agents and fail when thousands start together. Admission, networking, storage, scheduling, or cleanup may bottleneck first.
Scaling AI-agent sandboxes requires a bounded execution architecture. This guide covers demand, bursts, tenant isolation, lifecycle, cost, failure testing, and enterprise operations.
Scale AI-agent sandboxes by separating admission from placement, keeping capacity ready for bursts, and dividing the fleet into bounded cells as demand grows.
- Define live concurrency, starts per second, time to interactive, active duration, and resource shape before setting a scale target.
- Enforce tenant quota and safe-capacity checks before placement. Queue bounded work and reject excess demand rather than oversubscribing the fleet.
- Keep hosts, common images, filesystems, networking, and credentials ready enough to meet the complete startup objective.
- Make create, pause, resume, expire, and delete operations asynchronous and retry-safe. Cleanup is part of capacity management.
- Combine runtime isolation with resource limits, network policy, scoped credentials, priority, and independent containment.
If your enterprise needs isolated execution at high concurrency, Northflank is a full-stack application platform with API-driven Sandboxes for ephemeral or persistent environments using microVM or gVisor isolation. The same platform runs APIs, workers, databases, storage, and GPU workloads.
Northflank has operated millions of microVMs monthly since 2021. In the 18 June 2026 ComputeSDK Scale Invitational, Northflank reached 100,000 concurrent live 1-vCPU sandboxes in 24 seconds. Enterprise teams can use SSO, RBAC, scoped API roles, secret injection, private networking, and audit logs to govern access across the sandbox fleet and surrounding services.
Start on Northflank Cloud, or use self-serve BYOC when workloads need to run in your cloud account with its network, regional, capacity, and billing controls. Compatible coding agents can use Northflank Skills to operate sandboxes and related infrastructure within their assigned permissions.
Get started with Northflank self-serve, or book a demo to discuss architecture, isolation, concurrency, cloud placement, or migration requirements.
High concurrency means many isolated environments are live at once, but that number alone does not describe the workload.
Track five dimensions:
- Live concurrency: environments consuming runtime capacity now.
- Start throughput: environments admitted per second during a burst.
- Time to interactive: time from request acceptance until useful work can begin.
- Active duration: how long an environment consumes compute before it pauses or terminates.
- Resource shape: CPU, memory, storage, network, process, and GPU requirements per sandbox.
Workload shape changes the decision. Short executions stress creation and cleanup; persistent workspaces retain memory and storage; browser agents use heavier images; evaluation or RL workers create large batches. A large monthly execution count may produce modest concurrency, while ten thousand starts in one minute can stress every provisioning dependency.
A scalable architecture keeps global admission separate from local placement and runtime operation.
Admission authenticates the caller, authorizes the workload, checks quota, selects a region or cell, and assigns a durable sandbox ID.
Require an idempotency key so retries do not create duplicate environments. When capacity is unavailable, use a bounded queue with a deadline and expose its state. Once full, return a retry signal. An unbounded queue converts a capacity event into unpredictable latency and cost.
Quota should cover starts per second, CPU, memory, storage, network use, process count, lifetime, and retained idle environments, not only live sandbox count.
If each workload needs defined compute resources, Northflank resource plans let you assign CPU and memory to it, while horizontal autoscaling sets minimum and maximum deployment instances.
Your sandbox control plane should enforce tenant-wide concurrency and creation-rate limits before sending work for placement. The secure AI-generated code platform comparison explains how resource controls fit alongside isolation and lifecycle requirements.
After admission selects a cell, a local scheduler should place the sandbox on a host with the required runtime, capacity, image, storage path, and topology.
A cell is a bounded group of clusters or hosts with its own capacity and failure radius. Kubernetes v1.36's large-cluster guidance describes an envelope of 5,000 nodes and 150,000 pods, while cloud quotas may constrain growth sooner.
Add cells when a cluster approaches safe tested capacity, control-plane pressure affects readiness, workloads need separate runtimes, or one failure would affect too many tenants. Keep detailed runtime state local so a global service does not process every status update.
If your enterprise needs sandbox workloads inside its own cloud account, Northflank BYOC provides the deployment model and supports sandboxed node pools, tags, and affinity rules. These controls separate pools by runtime, region, tenant tier, or hardware. For a larger cell architecture, see how to run millions of concurrent AI sandboxes.
Fast runtime boot does not help if hosts, images, filesystems, networking, or credentials still need provisioning. Keep the slowest parts outside the startup path.
This includes ready hosts, pre-pulled images, cached layers, prepared filesystems, IP space, gateways, and credential capacity. Separate pools so a large GPU image does not delay a small code interpreter.
Warm capacity should cover net concurrency growth during infrastructure provisioning plus a failure reserve. Measure cache misses and cold-host starts separately.
Sandbox creation should return an operation or ID before readiness. A controller then reconciles accepted, starting, ready, paused, terminating, and deleted states.
Make every transition retry-safe. Use leases to expire abandoned environments, and record deletion so late events cannot recreate them. Cleanup must remove compute, routes, temporary credentials, and ephemeral storage. A fleet that creates 1,000 sandboxes per second but deletes only 900 will exhaust capacity.
If agents need programmatic lifecycle control, Northflank exposes sandbox creation and lifecycle operations, including startup, status inspection, command execution, pause, resume, and deletion through its JavaScript client.
Keep orchestration state outside the sandbox and make root filesystems disposable where the workload permits. Put durable files in persistent volumes or object storage, then associate them with a stable sandbox or workspace identity.
If files must survive a restart or pause, Northflank sandboxes support persistent volumes. A Single Read/Write volume limits the attached service to one instance, while Multi Read/Write supports multiple replicas but requires coordination for concurrent writes. See when to use persistent or ephemeral AI sandboxes.
Avoid creating a public load balancer or IP for every environment. Shared regional gateways can authenticate requests and route them to private sandbox endpoints, reducing infrastructure objects and centralizing rate limits and access checks.
Plan capacity using workload-shaped load tests rather than a generic sandbox-density number. Measure live concurrency, peak creation rate, resource requirements, and the time required to add usable infrastructure. Keep enough ready capacity to absorb demand during that provisioning interval, plus the failure reserve required by your availability objective.
Capacity is bounded by whichever part of the execution path saturates first. Admission throughput, scheduler and control-plane throughput, host CPU and memory, storage, networking, image distribution, credentials, and external services can each become the limiting factor.
Autoscaling replenishes capacity, but it still depends on provider quotas, node creation, image preparation, and cluster registration. Ready capacity covers the interval before new infrastructure becomes usable.
If the execution fleet must run in your cloud account, Northflank BYOC places the workloads there and lets platform teams deploy and scale node pools while retaining workload quotas and placement rules.
Prevent cross-tenant risk by applying isolation and resource policy at the same boundary used for admission and scheduling.
Use hardware-virtualized microVMs or an appropriate user-space-kernel boundary for untrusted model-generated or user-submitted code, based on your threat model. Runtime isolation does not limit what code can reach or consume, so add:
- Per-sandbox CPU, memory, process, storage, network, and duration limits
- Tenant-level concurrent sandbox and creation-rate quotas
- Default-deny sandbox-to-sandbox networking where communication is unnecessary
- Destination and protocol controls for egress
- Short-lived credentials scoped to one task and the minimum required actions
- Priority classes and reserved capacity for critical workloads
Containment must work independently of the agent. Operators need to terminate a run, revoke credentials, block network access, quarantine state, and stop another request. Test those actions under load.
If sandbox traffic must remain private or reach only approved destinations, Northflank provides private connections between workloads. On BYOC, configure Network Policies as allow-lists or deny-all rules; without relevant rules, traffic is allowed. Combine them with secret injection and scoped API roles. See how to isolate AI agents that access company data for the wider authorization model.
A sandbox fleet is scaling when it meets readiness, isolation, cleanup, and cost objectives under expected bursts.
Measure:
- Admission: accepted, queued, rejected, deduplicated, and rate-limited requests
- Readiness: queue time, placement time, runtime boot, image preparation, network setup, and time to interactive at median and tail percentiles
- Capacity: live sandboxes, ready headroom, host utilization, unschedulable requests, and pool or cell saturation
- Runtime: forced termination, resource-limit events, command failures, and network-policy denials
- Lifecycle: pause, resume, expiry, deletion success, cleanup lag, and orphan count
- Cost: compute and storage per sandbox-minute, task, tenant, and workload class
Break metrics down by region, cell, runtime, and tenant tier. Keep sandbox or user IDs in queryable logs instead of metric labels.
Platform audit logs show who or what changed platform resources. Application telemetry should record policy-permitted inputs and outputs, tool activity, approvals, affected resources, and state changes without requiring private chain-of-thought.
If platform teams need one operational view, Northflank's built-in observability provides workload logs and metrics, infrastructure alerts, and separate audit logs for platform events.
Test the complete agent workflow at the burst shape you expect in production, then remove dependencies and capacity deliberately.
Use the real image, dependencies, network policy, credentials, storage, and readiness check. Increase creation rate and concurrency separately to distinguish provisioning bottlenecks from steady-state limits.
During the test:
- Exhaust a tenant quota and confirm other tenants retain capacity.
- Remove hosts or a cell and measure admission and recovery.
- Force image-cache misses and slow storage dependencies.
- Deny egress and revoke a credential during execution.
- Delete a large batch and confirm compute, routes, credentials, and temporary data disappear.
- Retry create and delete requests to confirm idempotency.
In the 18 June 2026 ComputeSDK Scale Invitational, Northflank reports reaching 100,000 concurrent live 1-vCPU sandboxes in 24 seconds; ComputeSDK records a matching 100,000+ 1-vCPU run. Test your images, dependencies, networking, storage, and readiness path under the concurrency and burst shape you expect.
For an implementation starting point, follow the guide to spin up a secure code sandbox and microVM, then replace its sample workload with your production image and burst profile.
Northflank brings the controls used throughout this architecture together with the surrounding stack. Teams can control isolated sandboxes through the REST API, CLI, or JavaScript client, while applying SSO, RBAC, scoped API roles, secrets, and audit logs. Compatible coding agents can use Northflank Skills to operate sandboxes and related infrastructure through the same interfaces within their assigned permissions.
With Northflank Cloud, Northflank provides managed infrastructure. With self-serve BYOC, your enterprise owns the cloud account and infrastructure while Northflank manages platform orchestration. Available sandbox runtimes depend on the cloud provider and region.
Northflank has operated millions of microVMs monthly since 2021. For enterprise security and procurement requirements, Northflank is SOC 2 Type 2 compliant and HIPAA compliant, with BAAs supported under Enterprise contracts.
Get started with Northflank self-serve, or book a demo to discuss sandbox architecture, concurrency, isolation, BYOC capacity, or migration.
These answers cover scaling decisions.
No. They measure different things: concurrency counts live environments, while starts per second measures creation throughput. Long-running workspaces can have high concurrency and a low start rate; short tasks can produce the opposite pattern.
Not immediately. Start with one tested cell, then add cells when capacity, control-plane pressure, workload separation, regional placement, or failure radius requires it. Northflank supports managed Northflank Cloud and sandbox deployment through BYOC.
Keep enough ready capacity for demand during infrastructure provisioning, plus the failure reserve your SLO requires. Load tests determine the number. For customer-cloud fleets, Northflank BYOC lets teams deploy and scale node pools.
No. MicroVM boot is only one stage. Time to interactive also includes admission, queues, placement, images, filesystems, networking, credentials, process startup, and health checks.
Not on its own. Northflank horizontal autoscaling can add deployment instances based on configured CPU, memory, RPS, or custom-metric thresholds. Admission control, quotas, ready capacity, image preparation, provider limits, and cleanup still determine whether a burst succeeds safely.
