How to monitor and debug AI-agent sandboxes in production

An AI-agent sandbox can be running while the agent is stuck in a tool loop, using the wrong credential, sending data to an unexpected destination, or producing an unusable result. Container uptime tells you whether a process is alive, not whether the run is correct, safe, or complete.

Production monitoring must connect the task to policy, sandbox, infrastructure, and cleanup events. This article explains the signals, alerts, debugging path, and safe reproduction.

Monitor and debug AI-agent sandboxes by giving every run a stable identifier, correlating application and infrastructure telemetry, and following failures from admission through execution and cleanup.

• Monitor task outcomes, policy behavior, sandbox health, and lifecycle completion separately. A healthy container does not prove that the agent completed its task safely.
• Carry an application-owned run ID across the agent, policy engine, tool calls, sandbox, artifacts, and cleanup while retaining each system's native identifiers.
• Alert on conditions with a defined response, such as rising startup failures, runaway tool calls, unusual access, resource exhaustion, missing telemetry, and incomplete cleanup.
• Debug from the first failed transition in the run timeline instead of starting with unstructured log searches.
• Reproduce failures in a restricted non-production sandbox using the same versioned inputs and sanitized data, not live production credentials.

Production monitoring should answer four questions: is the infrastructure healthy, did the agent complete the task, did it stay within policy, and did temporary resources end as intended?

Treat the four outcomes as independent states:

• Infrastructure health: Was the sandbox admitted, created, made ready, and given enough compute, memory, storage, and network capacity?
• Task correctness: Did the agent achieve the requested result, pass validation, and produce the expected artifact or business effect?
• Policy compliance: Were tools, credentials, data, destinations, approvals, and resource use within the authority granted to this run?
• Lifecycle completion: Were child processes stopped, credentials revoked, artifacts exported, state handled, and the environment deleted or paused according to policy?

If your team needs the execution platform to distinguish startup from availability, Northflank provides startup, readiness, and liveness health checks for deployed workloads. Startup, readiness, and liveness probes test whether startup has completed, whether the container should receive traffic, and whether it should be replaced.

An agent run needs four evidence layers.

Create an application-owned identifier before admission and add it to telemetry, sandbox metadata, artifacts, and cleanup. run_id is one possible name. Preserve native trace, tool-call, container, credential, and platform-event IDs alongside it.

Propagate trace context across services to correlate traces and logs. Sanitize context received from untrusted callers, and do not add secrets or sensitive personal data to propagated fields because they may reach downstream or external services.

Useful metrics expose a failed boundary or trigger a response. Set thresholds from your workload baseline, risk, and service objective.

Measure requested, denied, queued, and started runs, queue age, creation latency, readiness failures, scheduling failures, and capacity by execution class or tenant.

Alert when startup success falls, queue age keeps rising, or one tenant consumes a disproportionate share of capacity.

If your platform service needs to automate this lifecycle, Northflank provides API-controlled sandbox creation and execution. Your application can create an isolated environment, poll its deployment status, execute commands, and collect stdout, stderr, and exit results against its own run record.

Measure duration, tool latency, exit status, retries, timeouts, validation failures, and outcomes. Segment by agent, model, tool, image, policy, and release version.

Do not make command success the task success metric. Validate the artifact or effect the user requested, such as tests, policy checks, and an exported patch.

Track CPU, memory, storage, output, network traffic, model tokens, duration, concurrency, and cost per run and successful outcome.

Watch distributions and high percentiles, then alert on sustained saturation, repeated evictions, abnormal output, or cost without task progress. Apply limits and circuit breakers for abnormal cost, latency, retries, and tool-call behavior.

If your operations team needs to test whether infrastructure pressure caused a failure, Northflank provides container metrics for CPU and memory usage, persistent-volume capacity, public and private network requests and bandwidth, and log lines generated per second. These network metrics show traffic levels rather than destination-level activity. Combine the platform signals with application-level token, tool, and outcome metrics rather than treating them as the complete cost of an agent run.

Monitor tool denials, approval bypass attempts, elevated access, unexpected destinations, sensitive-data access, policy changes, and high-risk action frequency. Compare activity with the run's purpose rather than treating every uncommon action as malicious.

Monitor decisions, tools, outcomes, approvals, policy versions, anomalies, token use, and cost in a structured format. Alerts should be able to invoke containment, not only notify a dashboard.

If agent sandboxes need enforceable network and credential boundaries, Northflank provides network policies for restricting ingress and egress on BYOC clusters. The platform also provides API tokens whose permissions derive from an associated RBAC role. Tokens can be configured with an expiry and revoked independently, while Enterprise plans provide token-specific audit-log feeds.

Measure expired environments, cleanup latency, failed deletion, retained volumes, credential revocation, orphans, and telemetry delivery.

Alert on missing run endings, unconfirmed deletion, surviving credentials, and log-stream gaps. Silence is not proof of inactivity.

If evidence must remain available after an ephemeral sandbox is deleted, Northflank provides log sinks that forward runtime logs and can optionally include CDN and network-mesh logs. These logs do not replace destination-aware network or DNS telemetry when an investigation requires it. Monitor delivery to the configured observability or storage destination so a failed export becomes an operational signal of its own.

Debug a failed run by finding its first incorrect transition, then moving outward only when the evidence requires it.

Start with the run, trace, tool-call, sandbox and container IDs, plus the image digest, agent and policy versions, timestamps, and lifecycle state. Pause automated deletion only if incident policy permits it, without leaving credentials or network access active.

Classify the symptom as admission, startup, execution, tool access, task validation, policy, persistence, cleanup, or telemetry loss. This gives the investigation an owner and hypothesis.

Order the request, policy decision, creation, readiness, tool activity, resource changes, artifact export, termination, and cleanup. Separate requested actions from executed actions and effects.

If your team needs to reconstruct runtime and infrastructure events together, Northflank provides live and historical logs for running and terminated containers and job runs. Its platform audit logs show infrastructure and configuration events. Join both to the application trace.

Diff the code, image, dependencies, agent configuration, model, tools, policy, credential scope, resources, network rules, data references, and feature flags.

Check scheduling, readiness, health probes, container state, exit code, CPU, memory, storage, traffic, and cluster capacity. If your team needs these signals in one operational view, Northflank provides container monitoring that combines container status, health results, logs, and resource and network metrics. A misconfigured startup or liveness probe can replace an otherwise healthy container, while a failing readiness probe removes it from the load balancer.

Create a new sandbox from the same immutable versions, but use sanitized inputs, test credentials, restricted networking, and no production write access. Change one variable at a time and record where the reproduction diverges from the original.

Depending on impact, terminate the run, block admissions, revoke credentials, disable a tool or policy, restrict networking, and quarantine state or artifacts. Confirm each action, add a regression test, and update the runbook. Assign incident-response ownership, rehearse the process, and use retrospectives to improve it.

A timeout can originate in the queue, sandbox, model, tool, or cleanup path. Use the earliest abnormal signal to narrow the layer.

Reproduce a production failure with the same versioned environment and less authority. Preserve the image, code, dependencies, agent configuration, model, tools, policy, resources, and sanitized inputs needed to test the hypothesis.

Use a separate environment with test credentials, restricted egress, approved data, and cost and time limits. Prefer a read-only interface, recorded response, or controlled stub for production dependencies.

If collected evidence cannot explain the failure, Northflank provides shell access to applicable running containers for last-mile diagnosis. Interactive access changes state, so use telemetry first and record who opened the shell, when, why, and what changed.

Northflank supplies execution and infrastructure observability while your application emits agent, policy, tool, and task telemetry.

• Northflank Sandboxes provide API-controlled isolated environments and command execution with stdout, stderr, and exit results.
• Logs and container monitoring cover builds, deployments, jobs, addons, container state, resource use, network traffic, and health checks.
• Infrastructure alerts cover crashes, eviction, CPU and memory pressure, volume capacity, and cluster issues, with notification integrations for routing events.
• Log sinks forward runtime logs and can optionally include CDN and network-mesh logs for configured observability or storage destinations.
• Audit logs show platform events, origin, triggering identity, affected resources, parent events, and configuration changes.

Use Northflank Cloud for managed infrastructure. Use self-serve BYOC to run workloads and storage in your own cloud account and VPC while Northflank manages Kubernetes. BYOK connects a compatible existing Kubernetes cluster your team operates.

If your enterprise needs isolated code execution, Kubernetes orchestration, infrastructure observability, GPU workloads, and deployment in your own cloud account and VPC, Northflank is our default recommendation for running AI-agent sandboxes. Startups and individual engineers can use the same platform through its self-serve developer experience.

These answers clarify common production boundaries.

No. Debugging also needs traces, metrics, versions, policy decisions, tool results, lifecycle state, and task validation. Audit events or network telemetry may be needed for infrastructure and security questions.

Only where policy permits. Prompts, outputs, arguments, and files can contain code, credentials, personal data, or hostile content. Prefer identifiers and protected references; retain redacted content only for a defined need.

It depends on impact. Candidates include expiry, sustained resource or cost limits, runaway recursion, prohibited destinations, invalid approval, credential misuse, or loss of a required control. Decide whether each condition denies an action, terminates a run, or blocks an execution class.

Export telemetry and artifacts outside the sandbox before deletion. Keep confirmation events for credential revocation, state handling, and resource removal.

Continue with these related implementation guides.

• What should an audit trail for AI-agent code execution contain? covers durable execution evidence.
• What infrastructure do AI agents need to run code safely? maps the control and execution layers.
• How to govern AI-agent code execution in enterprise environments connects execution policy to identity, data, networking, evidence, and containment.
• How to scale AI-agent sandboxes for high-concurrency workloads covers admission control, warm capacity, quotas, and lifecycle automation.
• How to deploy an AI agent from sandbox to production explains the controlled path from isolated development to a production workload.