GPU autoscaling: How to scale without wasting capacity
GPU autoscaling reduces wasted capacity when you scale application replicas using workload demand, coordinate GPU node capacity with those replicas, and release idle infrastructure while meeting your latency or job-completion targets.
Reducing replica count alone can leave GPU nodes running and your team paying for unused capacity. This guide explains how to choose scaling metrics, account for model startup, configure scale-down, and measure the results, using inference services as the main example.
GPU autoscaling should match ready replicas and GPU nodes to workload demand, then release idle capacity as demand falls while maintaining your performance targets.
- Measure one replica against your latency or job-completion target before choosing thresholds.
- Use workload signals such as queue depth and active requests, with GPU metrics for context.
- Coordinate application replicas with GPU node capacity and placement requirements.
- Allow for model startup and stable scale-down without interrupting work unnecessarily.
- Track successful work, service quality, and paid capacity together.
To scale GPU services on managed Northflank Cloud or in your own cloud account, Northflank GPU workloads support custom-metric autoscaling to match service capacity to demand. If you deploy in your own cloud, configurable BYOC node pools also let you scale the underlying infrastructure within your chosen limits.
Get started with Northflank self-serve, or book a demo to discuss GPU workload architecture and capacity.
GPU autoscaling is the automatic adjustment of application replicas or GPU infrastructure to match workload demand. It adds capacity as demand increases and removes capacity as demand falls, within configured limits.
An application replica is one instance of your service. A GPU is a device that replica uses. A node is the machine supplying GPUs, CPU, memory, and storage. One replica may need several GPUs, and one node may hold several replicas.
In Kubernetes, the workload autoscaler changes replica count, while a node autoscaler supplies or removes machines. Adding replicas only helps when they can be scheduled and become ready. Removing them releases their allocation, but the node may remain in place.
GPU capacity can remain unused when replica counts fall but nodes stay allocated, minimum capacity exceeds demand, or free GPUs cannot satisfy a workload's placement requirements. Application scaling and infrastructure scaling need to work together to release that capacity.
Use these symptoms to guide diagnosis:
| Symptom | Possible cause | What to check |
|---|---|---|
| Replicas sit idle between bursts | Minimum capacity exceeds the workload's needs | Warm-capacity requirement and scale-down policy |
| Replica count falls but nodes remain | Pool minimums or other workloads retain nodes | Node occupancy and removal rules |
| Replicas wait despite free GPUs elsewhere | Available devices do not fit placement or resource requirements | GPU count, node shape, and scheduling constraints |
| GPU activity is high but useful throughput is poor | The utilization signal does not explain application performance | Request mix, batching, and latency |
The Kubernetes autoscaling guide explains the general controllers. For GPU workloads, check the full path from requested replicas to usable devices and ready application capacity.
GPU inference autoscaling should use queue depth, active requests, or batch size as workload signals, with latency checks to validate performance and GPU utilization and memory metrics to diagnose resource pressure. Choose and test thresholds against your service's response-time target.
For inference, queue depth shows requests waiting for service. Active requests or batch size show work already being processed. Queue-based scaling can suit throughput goals, while tighter latency requirements may justify responding earlier to active work. Track response latency or time to first token to judge whether the policy works.
GPU utilization provides context, but a busy device does not reveal how much additional inference it can handle. Allocated GPU memory also needs care: servers that preallocate memory may retain it after traffic falls. A memory threshold can therefore prevent useful scale-down.
Define each metric precisely. A queue inside one replica differs from a shared backlog used by every worker. Before setting a target, establish whether the scaler receives a total, an average, or a per-replica value.
If your application needs its own scaling signal, Northflank supports custom Prometheus metrics. Gauge values are averaged across running pods; counters use a rate before averaging.
If every pod exposes the same shared queue total, averaging returns that same total, not the backlog per replica. Northflank uses the resulting value and the current replica count to calculate the desired instance count. Design the metric and threshold together so that adding workers changes the signal as intended.
Configure GPU autoscaling by measuring one replica's sustainable throughput, setting replica limits and workload-metric thresholds, and matching GPU node capacity to those limits. Account for model startup and graceful scale-down before testing the complete demand cycle.
Repeat this process when the model, hardware, or request mix changes.
Test one warmed replica with representative requests and a defined response-time target.
Record the model version, GPU configuration, input and output sizes, and batching settings. Increase load gradually while observing successful throughput, waiting requests, and latency. Identify the load the replica can sustain while meeting your target.
Save the test configuration alongside the throughput measurement. A requests-per-second figure is only useful when the requests resemble those your service will receive. Retest with longer requests and bursts before using the result to size production capacity.
Choose a minimum that meets your availability and startup requirements, then a maximum your infrastructure can accommodate.
Set the scale-up threshold early enough for new replicas to start. Check for repeated additions and removals, and adjust stabilization accordingly.
Treat the maximum as a capacity limit. It does not determine the full bill: how long replicas run, the nodes retained underneath them, and other infrastructure still affect cost. Decide what acceptable service behavior looks like when demand exceeds that limit, and include that condition in testing.
Ensure the node configuration can satisfy each replica's complete resource request.
Check GPU type and count, CPU, RAM, storage, and placement constraints. Free GPUs spread across nodes may not satisfy a replica that needs several devices together. Check provider quotas and available capacity before relying on a higher replica maximum.
If workloads must run in your own cloud, Northflank provides BYOC node-pool autoscaling with minimum and maximum node counts and scheduling controls. Align those bounds with the service's requirements so replica growth has somewhere to run.
If you need a specific GPU type and quantity for a planned workload, request GPU capacity from Northflank to discuss your requirements and timeframe.
The guide to running CPU and GPU workloads in your own VPC covers the wider infrastructure choice.
Count a new replica as serving capacity only after the application is ready to accept work.
Measure the interval between requesting capacity and serving a successful request. Separate scheduling delays from container initialization and model loading. Otherwise, increasing the replica count can make a dashboard look healthy while the original instances still carry all the traffic.
If repeated model downloads slow restarts, Northflank supports persistent model storage. Retaining files can reduce repeated downloads, but the application must still initialize and load its model. Check the storage arrangement against your intended replica placement.
Provision enough CPU, memory, and temporary storage for startup as well as steady operation. An available GPU is not enough if the application runs out of memory or temporary storage while starting.
Reduce capacity only after checking both sustained demand and the work still running.
Use stabilization to avoid removing capacity during a brief dip. Then test application shutdown: stop taking new work and allow active requests to finish within the configured termination period.
Kubernetes graceful termination has a deadline; remaining processes can be killed when it expires. A long-running request therefore needs explicit shutdown handling. Include interrupted requests and retries in your test results rather than counting every removed replica as a successful optimization.
After demand falls and replicas shut down, check whether their nodes can also be removed. Find out why idle nodes remain before making scale-down more aggressive.
GPU workloads should scale to zero when idle periods justify releasing the last replica, requests can tolerate cold-start delays, and an external mechanism can detect new demand and restart capacity.
Intermittent work with flexible completion times is a better candidate than an interactive endpoint with a strict first-response target. Compare the expected idle interval with measured startup behavior before choosing between zero replicas and a warm minimum.
Before enabling scale-to-zero on Northflank or another platform, confirm that the platform or an external controller can detect demand and restart the service from zero. Enabling metric-based autoscaling alone is not enough to assume that recovery will happen.
Decide where requests wait, how long they can wait, and what callers receive if startup fails or takes too long. Test the recovery path with no replicas running: send new work, check that a replica starts and serves it, then check that idle GPU nodes are released when demand ends.
A GPU scaling cycle detects increased demand, adds replicas and any required nodes, waits for the application to become ready, and removes excess capacity after demand falls.
Use your load-test results to estimate how many ready replicas can handle the expected traffic within your latency target. Allow additional capacity for bursts and failures, and account for waiting requests and the time new replicas need to start.
Follow the scaling cycle through four steps:
- Demand rises. The measured workload signal crosses its target and the scaler requests more replicas.
- Capacity starts. Replicas wait for suitable nodes if necessary, then initialize. They do not yet contribute their measured throughput.
- Replicas become ready. Check that waiting work declines and response times meet the target.
- Demand falls. After stabilization and graceful shutdown, check whether freed nodes leave the allocated fleet.
If the final step retains the same machines, you have reduced application allocation without necessarily reducing infrastructure spend.
Check GPU autoscaling efficiency by comparing paid GPU capacity and cost per successful request or completed job for equivalent workloads. Capacity use should fall without exceeding latency targets or increasing failures.
Track successful requests or completed jobs, waiting time, latency, ready replicas, allocated nodes, and cost. Useful measures include cost per completed job or per successful request for a consistent request mix. Choose a unit that reflects your application's output.
Test a steady load, a sharp burst, and a quiet interval. Include the time spent starting and shutting down capacity. Compare configurations against equivalent work so a lower bill caused by fewer requests is not mistaken for better efficiency.
Define success before changing the policy: less paid capacity for the same useful output, while meeting the performance target. A policy that reduces GPU-hours by increasing failures or leaving work unfinished has not met that goal.
Northflank supports GPU autoscaling through application metrics that control service replica counts and BYOC node-pool controls that adjust infrastructure capacity in your own cloud account. Your team configures the thresholds and limits around measured workload requirements.
Start with the GPU workload requirements to choose a compatible application image and deployment target.
For managed infrastructure, deploy GPUs on Northflank Cloud by creating a project in a GPU-enabled region and selecting an available GPU model and count for your service.
In the service's resource settings, enable horizontal autoscaling, set minimum and maximum instances, and configure the Prometheus endpoint and port, metric name, type, and threshold for a custom signal. When multiple metrics are enabled, Northflank uses the highest required instance count calculated from any of them.
To deploy GPUs in your own cloud account, use a project on your GPU-enabled cluster and configure suitable node pools and their scaling bounds. Northflank manages the configured pools; you control their resource choices and limits, and your cloud provider bills the infrastructure.
Run the full traffic cycle before increasing the maximum. Your application team remains responsible for choosing meaningful metrics, measuring model performance, and handling startup and shutdown correctly.
Get started with Northflank self-serve, or book a demo to discuss GPU scaling, workload placement, and capacity planning.
These questions help diagnose ineffective scaling policies.
GPU utilization alone is not a reliable scaling signal for every inference workload because device activity does not directly measure spare serving capacity. Use it only after testing its relationship with latency and successful throughput; queue depth and active work can provide more useful demand signals.
Removing GPU pods reduces costs only when it reduces billable usage under your provider's pricing model. If the underlying nodes remain allocated, removing pods may leave the infrastructure charge unchanged. Check allocation and billing rather than inferring savings from replica count.
GPU replicas can be running without serving requests because their applications are still initializing or loading model data. Configure readiness checks around the ability to accept useful work, and distinguish ready replicas from those that have merely started.
Use these guides to plan GPU infrastructure and compare capacity options.
