How to run millions of concurrent AI sandboxes

Running AI sandboxes at scale means handling demand without weakening isolation or causing a fleet-wide failure.

One million live environments, daily starts, and mostly paused workspaces are different targets. This guide covers architecture, capacity, isolation, burst performance, operations, observability, and Northflank.

Run millions of concurrent AI sandboxes by splitting the fleet into bounded regional cells, then routing each accepted request to a scheduler inside one cell.

• Avoid putting the entire million-sandbox execution fleet into one Kubernetes control plane. Shard execution across independently recoverable cells.
• Separate admission from placement. The global layer enforces identity, policy, quota, and regional routing; each cell chooses a host.
• Make lifecycle changes retry-safe so controllers can finish them after clients disconnect.
• Size ready headroom for net concurrency growth and infrastructure lead time, not average concurrency. Measure time to interactive rather than hypervisor boot alone.
• Match isolation, networking, credentials, storage, and telemetry to the threat model. At this scale, cleanup and containment are core platform features.

Millions of concurrent AI sandboxes means millions of isolated execution environments are live and consuming runtime capacity at the same time. It does not mean millions of sandbox starts, paused environments, or completed executions over a day or month.

Define four service-level measures before choosing an architecture:

• Live concurrency: sandboxes consuming runtime resources now.
• Start throughput: new environments admitted per second during a burst.
• Time to interactive: time from an accepted request until the workload can execute useful work.
• Lifecycle volume: environments created, resumed, paused, and destroyed over a day or month.

A coding workspace may run for hours, while a reinforcement-learning rollout resets every few seconds. An environment can retain files without consuming compute.

As a simplified example, if each live sandbox reserves 256 MiB of memory, one million sandboxes require about 244 TiB of guest memory before runtime and host overhead. Both the management layer and compute fleet must scale.

A conventional one-pod-per-sandbox design cannot fit one million live AI sandboxes inside Kubernetes' current large-cluster design envelope of 5,000 nodes and 150,000 total pods. One million pods exceed it more than six times over.

Creates, status updates, reschedules, and deletes also pressure the control plane, the components that schedule and manage workloads. Use multiple cells instead. A cell is a bounded unit of clusters or hosts with its own scheduling capacity and failure radius. The global platform tracks which cells can accept work, while detailed sandbox state stays local.

For a build-or-buy decision, compare high-concurrency sandbox platforms by isolation, lifecycle, deployment, and production-stack support.

A cell-based architecture can support millions of live AI sandboxes by separating global request acceptance and regional routing from scheduling and execution inside each cell. Each layer holds only the state required for its decision.

The admission service decides whether to accept a request. It authenticates the caller, checks tenant policy and quota, selects an eligible region and cell, then returns a sandbox ID that remains stable across retries.

When no eligible cell has safe capacity, queue bounded work or reject it with a retry signal. Use idempotency keys to prevent duplicate environments.

Each cell chooses a host, starts the runtime, configures networking, and restores unhealthy workloads. Maintain separate resource pools and spread workloads across hosts and availability zones.

Treat a sandbox as a requested outcome rather than one long-running API connection. A lifecycle controller records what should exist and moves the sandbox through accepted, starting, ready, paused, terminating, and deleted states.

Expiry leases remove abandoned sandboxes. Cleanup controllers remove runtime, network, credential, and storage attachments, while final deletion records prevent late events from recreating an environment.

Choose isolation from the workload's trust level. Containers share the host kernel and may suit trusted internal code when your risk model accepts that boundary.

gVisor moves much of the system-call interface into a user-space kernel, reducing direct exposure to the host kernel. It is not a microVM. Hardware-virtualized options such as Firecracker or VM-backed Kata Containers give the guest a separate kernel boundary, usually at a higher resource cost.

A minimal Firecracker test VM with 1 vCPU and 128 MiB of memory takes no more than 125 ms from its start API call to guest init, with no more than 5 MiB of VMM thread overhead. Full readiness also depends on networking, storage, credentials, and process startup.

Prefer disposable root filesystems. Put durable files on volumes or object storage, and keep orchestration state outside the runtime. Immutable images and copy-on-write roots make replacement predictable.

Do not allocate a public IP or load balancer to every sandbox. Route through shared regional gateways. Default-deny east-west traffic, apply per-task egress policy, and issue short-lived credentials.

Plan capacity and burst performance by load testing one cell, calculating the active and failover capacity your target requires, and keeping enough ready infrastructure for sudden growth.

As a simplified example, if one tested cell safely supports 25,000 live sandboxes, one million live sandboxes require 40 active cells. Add separate capacity for the largest failure your architecture must absorb, such as losing a region. Size hosts independently for CPU, memory, storage IOPS, and network throughput.

Keep ready hosts, images, filesystems, and network capacity outside the startup path. Measure time to interactive from request acceptance through scheduling, image preparation, networking, credentials, process startup, and health checks. Track median and tail latency, start failures, live concurrency, available capacity, and cleanup lag by region, cell, runtime, and workload class. These are starting estimates; workload-specific load tests determine safe limits.

Secure a multi-tenant AI sandbox fleet with layered identity, runtime isolation, network policy, short-lived credentials, resource limits, and independent incident controls. Authenticate every lifecycle and command-execution request, bind it to a tenant, and authorize it before routing into a cell.

Inside the runtime, drop unnecessary privileges and devices, enforce CPU, memory, process, and disk limits, and terminate workloads that violate policy.

For untrusted multi-tenant workloads, default-deny sandbox-to-sandbox networking unless explicitly required. Restrict egress destinations, protocols, and rates to reduce exfiltration and abuse. Scope credentials to the task and expire them automatically.

Independent controls must terminate runs, revoke credentials, block egress, quarantine state, and stop further requests. Confirm that compute, routes, credentials, and temporary data are gone.

Northflank runs AI sandboxes at scale by combining isolated execution, API-driven lifecycle management, persistence, networking, governance, and the surrounding application stack on one platform.

• Sandbox lifecycle and isolation: Northflank sandbox documentation covers create, status, command execution, pause, resume, persistence, and deletion. Workloads use hardware-virtualized microVM or gVisor user-space-kernel isolation depending on the workload and infrastructure.
• Infrastructure placement: Use Northflank Cloud for managed infrastructure or Northflank BYOC to run the sandbox runtime, storage, networking, and workload data path in your cloud account. Northflank-hosted control-plane metadata remains outside that workload data plane; review configuration-dependent flows such as logs, builds, images, DNS, and backups separately.
• Identity and governance: SAML or OIDC SSO, directory sync, MFA enforcement, RBAC, scoped API access, and secret injection separate access for your teams and workloads. Audit logs record platform events and affected resources. Your application still owns tenant and task authorization.
• Agent-native operations: Northflank Skills helps supported coding agents operate sandboxes and the surrounding services, databases, GPU workloads, previews, and releases through the Northflank API and CLI. The agent's assigned credentials and permissions still govern access.
• Compliance: Northflank is SOC 2 Type 2 compliant and HIPAA compliant, with BAAs supported under Enterprise contracts. See Northflank security details.
• Demonstrated scale: Northflank has run millions of microVMs monthly since 2021. In ComputeSDK's 18 June 2026 burst test, Northflank reached 100,000 concurrent live CPU sandboxes from cold start in 24 seconds.

These FAQs answer implementation questions.

A single conventional Kubernetes cluster cannot run one million one-pod-per-sandbox environments inside its 150,000-pod design envelope. Use multiple cells.

It depends on the workload and threat model. Trusted internal workloads may accept ordinary container isolation, while untrusted user-submitted or model-generated code usually warrants a hardware-backed microVM or appropriate user-space-kernel boundary, plus network, credential, and resource controls. Northflank Sandboxes use hardware-virtualized microVM or gVisor user-space-kernel isolation depending on the workload and infrastructure.

Keep enough ready capacity to absorb the net increase in live sandboxes while new infrastructure is being provisioned, plus capacity for failures. For a worst-case burst in which no sandboxes finish, use the peak start rate; load tests determine the final reserve.

No, fast microVM boot is not the same as fast sandbox startup. Boot excludes request acceptance, queueing, placement, images, networking, storage, credentials, process startup, and health checks. In ComputeSDK's burst test on 18 June 2026, Northflank reached 100,000 concurrent live sandboxes from cold start in 24 seconds.

• What is sandbox infrastructure?
• Best platforms for high-concurrency sandbox environments
• AI sandbox pricing comparison
• Ephemeral sandbox environments
• Persistent sandbox environments
• Running reinforcement-learning agents in secure sandboxes