HDInsight worker nodes without autoscale
A Spark, Hadoop or Interactive Query cluster whose worker role has no autoscale configured, so it pays for peak worker capacity every hour of every day.
Why it happens
HDInsight bills every node for as long as the cluster exists, and worker nodes are the bulk of that. Autoscale is free, and Microsoft describes it as scaling worker nodes within preset limits to minimise running cost, either against load or against a schedule. A cluster without it holds its full worker count overnight and at weekends whether or not anything is running. It is only available on Spark, Hadoop and Interactive Query clusters, so HBase and Kafka clusters are excluded rather than given advice they cannot follow. No saving is estimated: what autoscale returns depends on how variable the load is, and HDInsight publishes no utilisation metric to measure that from.
Paste this into Resource Graph Explorer in the Azure portal.
resources
| where type =~ 'microsoft.hdinsight/clusters'
// Scope below matches CloudFinOpsPro policy DEFAULTS: exclusions.protectTag, environment.tagKeys,
// environment.allowedTagValues, and allowUntagged false so an untagged resource counts as production.
// If your policy.yaml differs, edit the three lines below or this will disagree with your own agent.
| where tolower(tostring(coalesce(tags['cfop:protect'], ''))) !in ('true', 'yes', '1')
| extend cfopEnv = tolower(trim(' ', tostring(coalesce(tags['Environment'], tags['environment'], tags['env'], ''))))
| where cfopEnv in ('dev', 'development', 'test', 'qa', 'sandbox', 'nonprod', 'non-prod', 'staging')
| mv-expand worker = properties.computeProfile.roles
| where tostring(worker.name) =~ 'workernode'
| where isnull(worker.autoscale)
| project cluster = name, resourceGroup, location,
workerCount = toint(worker.targetInstanceCount)The property is autoscale, not autoscaleConfiguration. Head and ZooKeeper nodes cannot autoscale, so only the workernode role is examined. The cluster kind is not projected here because Resource Graph rejects a query that reads properties alongside an mv-expand of properties; read it separately with tostring(properties.clusterDefinition.kind), and check it before acting, since not every kind supports autoscale.
Applies to microsoft.hdinsight/clustersWhat the agent does about it
A Spark, Hadoop or Interactive Query cluster whose worker role has no autoscale configured, so it pays for peak worker capacity every hour of every day.
It will not fix this one
It finds it and tells you what to change. The change itself is yours to make, because it is not one the agent should make on your behalf.
Related checks
SQL Managed Instance without Azure Hybrid Benefit
A SQL Managed Instance paying the licence-included rate, which a customer holding SQL Server licences with Software Assurance can lower without downtime.
Serverless SQL database with auto-pause switched off and nothing using it
A General Purpose serverless SQL database with auto-pause disabled that had no sessions and no CPU at any point in the window, so it billed its minimum compute floor around the clock for doing nothing.
SQL Server VM paying the Azure licence rate
A SQL Server virtual machine on a Standard or Enterprise edition with its licence set to pay-as-you-go, which Azure Hybrid Benefit would replace if the customer holds Software Assurance.