Triage Dataflow and Spark job patterns

Why this is worth a look

Dataflow monitoring exposes job status, stage progress, data freshness, resource utilization, I/O, logs, errors, recommendations, and estimated cost. Its estimate uses Cloud Monitoring resource-use metrics and prices for the job region, so it can differ from actual cost because of discounts or billing adjustments. It excludes Dataflow Prime jobs and extra GPU accelerator cost. For streaming jobs, autoscaling views show worker counts, CPU utilization, backlog timing when Streaming Engine is used, logs, history, and scaling reasons. Cloud Monitoring collects Managed Service for Apache Spark metrics for Cloud Dataproc cluster, job, batch, and session resources. Spark autoscaling concerns worker nodes, uses Cloud Logging for autoscaler logs, and does not support Spark Structured Streaming.

A runnable, read-only console checklist for examining Dataflow and Managed Service for Apache Spark signals over one matching investigation window, without changing jobs or clusters.

DATA PROCESSING OPERATIONS TRIAGE

Execution surface: Google Cloud console, using Dataflow monitoring, the Dataflow Cost and Autoscaling tabs, Cloud Monitoring Metric Explorer, and Cloud Logging.
Permissions: read access to the relevant Dataflow project and region, Cloud Monitoring views, and Cloud Logging views available to your role.
Scope: one Google Cloud project, the relevant Dataflow region, and the Managed Service for Apache Spark resources under review.
Window and units: use one matching investigation period in each console. Record units exactly as displayed, including worker counts, CPU utilization, backlog time, resource usage, and estimated cost.

[ ] Dataflow monitoring: record running, completed, and failed jobs, elapsed time, stage progress, data freshness, resource utilization, I/O, logs, errors, recommendations, and estimated cost.
[ ] Dataflow Cost tab: record processing vCPU, memory, Dataflow Shuffle or Streaming Engine data, and SSD or HDD usage. Mark the result as an estimate. Use Cloud Billing reports separately for actual Dataflow cost when available.
[ ] Dataflow streaming autoscaling: compare current, target, minimum, and maximum workers with CPU utilization. For Streaming Engine jobs, review backlog timing, autoscaling logs, history, and scaling logic.
[ ] Spark resource metrics: in Metric Explorer, enter "dataproc" in Filter by resource or metric name, select a Cloud Dataproc resource, and review available cluster, job, batch, and session metrics.
[ ] Spark custom CPU metrics: when custom metric collection is enabled, select VM Instance > Custom metrics and review agent.googleapis.com/cpu/utilization. For autoscaled clusters, read Managed Service for Apache Spark autoscaler logs in Cloud Logging. This check concerns worker-node scaling and does not apply to Spark Structured Streaming.
[ ] Rank operational follow-up: flag repeated errors, lag or data freshness issues, resource pressure, backlog, or workers reaching configured boundaries. Record the next diagnostic step. Do not change the job, cluster, autoscaling policy, or billing configuration.

How to confirm it

1. 01

   Set one review window

   Choose one Google Cloud project, the relevant Dataflow region, and a matching investigation period. Record units exactly as shown. Use read access to Dataflow, Cloud Monitoring, and Cloud Logging views available to your role.

2. 02

   Review Dataflow operations

   In Dataflow monitoring, record running, completed, and failed jobs, elapsed time, stage progress, data freshness, resource utilization, I/O, pipeline logs, errors, recommendations, and estimated cost.

3. 03

   Separate estimated from actual cost

   In the Dataflow Cost tab, record processing vCPU, memory, Dataflow Shuffle or Streaming Engine data, and SSD or HDD usage. Mark it as an estimate. Use Cloud Billing reports separately for actual Dataflow cost when those reports are available.

4. 04

   Inspect autoscaling and Spark metrics

   For streaming Dataflow jobs, compare current, target, minimum, and maximum workers with CPU utilization. For Streaming Engine jobs, review backlog timing, autoscaling logs, history, and scaling logic. In Metric Explorer, enter "dataproc" and select a Cloud Dataproc resource for Spark resource metrics. Separately, when custom collection is enabled, select VM Instance > Custom metrics for agent.googleapis.com/cpu/utilization. Read autoscaler logs in Cloud Logging. Spark autoscaling scales worker nodes and does not support Spark Structured Streaming.

5. 05

   Record operational follow-up

   Flag repeated errors, lag or data freshness issues, resource pressure, backlog, or workers reaching configured boundaries. Treat these as investigation signals, not a cost ranking. Record the next diagnostic step without changing the job, cluster, autoscaling policy, or billing configuration.

Primary sources