Find idle Azure Machine Learning inference capacity
On this page
Flag capacity for owner review when managed online deployments retain a configured minimum or compute clusters scale back to a nonzero minimum during sustained low demand. Do not label it idle from CPU alone or change resources without checking schedules and operational requirements.
Why this is worth a look
Managed online endpoints use Azure Monitor autoscale profiles with minimum, maximum, and default capacity. Scale-in can release instances only down to the configured minimum. AmlCompute clusters release nodes after jobs finish and scale to their configured minimum. Review those floors alongside CPU utilization, request latency, schedules, job activity, quotas, and the endpoint's tagged cost.
Run this check
CHECKLISTA read-only Azure portal and Azure Machine Learning studio procedure for reviewing endpoint autoscale floors, cluster minimum nodes, supported metrics, and tagged endpoint costs.
Scope and assumptions
[ ] Select the subscription, workspaces, managed online endpoints, and AmlCompute clusters to review.
[ ] Choose a time window that includes normal demand and any scheduled workload periods. Record the window and use instance counts, node counts, milliseconds for RequestLatency, and instance-hours for any estimate.
[ ] Confirm read access to the workspaces, deployments, compute clusters, autoscale settings, and metrics being reviewed. For subscription Cost Analysis, use at least Billing Reader access.
Managed online endpoints
[ ] Review each deployment's autoscale configuration and record its deployment identity, configured minimum, maximum, and default scale set capacity.
[ ] Record the applicable metric rules and schedule rules. Review CpuUtilizationPercentage and RequestLatency where available, then select other available endpoint or deployment metrics appropriate to the model and hardware.
[ ] Compare the configured minimum with demand across the selected window. Flag a sustained low-demand deployment that retains this minimum for owner review, not for automatic deletion.
[ ] In subscription Cost Analysis, filter Resource to the Azure Machine Learning workspace. Add Tag filters azuremlendpoint:<endpoint name> and azuremldeployment:<deployment name>.
AmlCompute clusters
[ ] Review each managed Azure Machine Learning compute cluster's configured minimum and maximum node counts and recent job-driven scaling behavior.
[ ] Flag a cluster whose configured minimum is above zero when the owner confirms that no workload requires that floor. Ask the owner about queued or scheduled work rather than asserting that a particular queue view is available.
Decision record
[ ] Estimate observed retained capacity in instance-hours or node-hours from the selected window. Use the configured minimum as the floor, not the maximum, and label the arithmetic as an estimate.
[ ] Record the owner's availability, failover, cold-start, rollback, schedule, and quota considerations. Azure Machine Learning quotas can apply by region, subscription, workspace, and VM family.
[ ] Send any setting recommendation for approval. Make no resource changes during this review.How to confirm it
- 01
Set scope and review window
Choose the subscription, workspaces, endpoints, deployments, and AmlCompute clusters. Use a period that includes normal demand and scheduled workload periods, and record the time window and units used.
- 02
Review endpoint floors and signals
For each managed online deployment, record configured minimum, maximum, and default scale set capacity, plus applicable metric and schedule rules. Review CpuUtilizationPercentage and RequestLatency where available, then select other available metrics appropriate to the model and hardware.
- 03
Review cluster minimums
For each AmlCompute cluster, record configured minimum and maximum node counts and recent job-driven scaling behavior. Ask the owner whether queued or scheduled work requires a nonzero floor rather than assuming a specific queue view is available.
- 04
Match endpoint usage to cost
In subscription Cost Analysis, filter Resource to the Azure Machine Learning workspace, then add the azuremlendpoint and azuremldeployment tag filters. This documented tag workflow applies to managed online endpoints.
- 05
Document and seek approval
Estimate retained capacity in instance-hours or node-hours using the selected window and configured floor. Record availability, failover, cold-start, rollback, schedule, and quota considerations, then recommend a setting review without making changes.
Before making changes
Assume the review window covers normal peaks and scheduled workloads, and label all instance-hour or node-hour arithmetic as an estimate. Low CPU or low latency demand alone does not prove that a configured floor is unnecessary because autoscale can also use schedules and other available metrics. Ask the owner about availability, failover, cold-start, rollback, and quota requirements. The documented Cost Analysis tag workflow applies specifically to managed online endpoints, so verify the selected subscription, workspace, tags, and period.